Just to pass on a few additional papers that might be helpful for reducing your scale:

Smith, G. T., McCarthy, D. M., & Anderson, K. G. (2000). On the sins of short-form development. Psychological Assessment, 12, 102-111.

Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167-194.

Widaman, K. F., Little, T. D., Preacher, K. J., & Sawalani, G. M. (2011). On creating and using short forms of scales in secondary research. In K. H. Trzesniewski, M. B. Donnellan, & R. E. Lucas (Eds.), Secondary data analysis: An introduction for psychologists (pp. 39-61). Washington, DC US: American Psychological Association.

I would recommend to do a Factor Analysis using Maximum Likelihood method with direct oblim rotarion. The is a very good paper from Osborne & Costello (2005) Best practices in FA (http://pareonline.net/pdf/v10n7.pdf) and an other paper on sample size (http://pareonline.net/getvn.asp?v=9&n=11).

For a quite different approach, take a look at this article: Planned missing data designs in psychological research. Graham, John W.; Taylor, Bonnie J.; Olchowski, Allison E.; Cumsille, Patricio E.

Psychological Methods, Vol 11(4), Dec 2006, 323-343. I'm not suggesting this instead of the factor analysis, but as a complement to it.

I try to factor analysis, combined with the analysis of the reliability of the items. And if any item can assume the role of dependent variable, also prove the linear regression.

I have often wondered why so many researchers apply factor analysis at all, because the outcome would often seem to be to add together redundant information (a scale of items that partly measure the same thing). Now, if you want to distinguish whether there are different dimensions in a material, that would seem to be OK, but if you have a single dimension, what would be the use? Now, I believe there are cases when scales should not be formed from items that correlate, or when the correlation is irrelevant. Your material could be such a case.

Consider that you want to measure how dissatisfied people are with their medical treatment. In such a case, I would say that all possible negative events should need to be canvassed, and the responses summed, regardless of whether they correlated, because the items would not tap a single dimension, but the satisfaction of the client would be equal to the degree of absence of negative events.

So, in that case, we would not be measuring a psychological dimension (which I believe is what factor analysis is intended for), but events and a reaction to them.

So, I would say that whether factor analysis should be applied or not is a matter of the goal of the research, and the theoretical basis of it.

Going back to the original question, I would suggest testing each item against some sort of external criterion, like change of clinic, or refusal to comply with recommendations, or wish for a second opinion. This would of course be dependent upon the goals of this research, but something which would indicate whether the item was tapping something which was important for whatever the intended use of the scale is. Having excluded items that do not predict anything useful, you might also be able to amalgate other items, if they appear to measure similar things (instead of an item for each possible situation, you could construct more general ones). For a somewhat different example of what I am saying, you could have a suicide scale of items that asks whether the respondent has tried to commit suicide by each of a number of different ways (they would probably not correlate), or just a single question on whether they have tried at all.

But of course, whether this advice is relevant is dependent upon a number of factors in your research that I have been guessing about.

Mixing up two scales in the same analysis could be a bit tricky for interpretation. Here are two simple ways to verify if this can be done and how:

1) If you assume that the increase on one scale should rise the same way it would for questions from the other scale, then you should be able to use factorial analysis combining all questions together. I would however check both linearity and association between ranks of questions from each scale to support this assumption.

2) If on the other hand you believe that both measure two different aspects of a single phenomenon, then it can be appropriate to analyze both sets of questions from each scale separately and then repeat the factorial analysis using two sub-scores.

Dear william you can try categorical principle component analysis and do not worry about your sample size

For those data that you can treat as quantitative, you can try the dimensionality reduction by PCA (principal component analysis). For demographic data this was done in the classic book by Hofstede on the "cultural dimensions". Then you need to assign the meaning to the top dimensions - but this should be fun for the domain expert :)

Alternatively I would try some data mining for association rules, most likely after encoding attribute-value together - the size and description of your data set seem suitable. The academic tool for doing it may be eg. Weka package. http://www.cs.waikato.ac.nz/ml/weka/

I agree you should try an FA

Ideally you would use a method to accomodate the ordinal nature of the indicators, such as that implemented in Mplus

Also, in Mplus you can run a monte carlo simulation to estimate power post hoc- some loadings may be diffciult to estimate where information is lacking (e.g. lack of variation in responses etc).

I would agree with Anders Wåhlberg, my use of factor analysis used at more than baseline just rendered me headache as the data differed at 2 different times of measurements e.g baseline and after 12 weeks . If one has data from just one measuring point perhaps factor analysis can be used properly ?

First, I would definitely try with FA to determine the scale dimensionality and to see which items have small saturations or have saturations on more than one dimension (these are your candidates for item reduction). In doing item analysis you can also try with discrimination power index. Here are some basic guidelines.

http://mailer.fsu.edu/~akamata/edf5432/3_19.pdf

http://www.spsstools.net/Syntax/ItemAnalysis/UsingSPSSforItemAnalysis.pdf

I would examine first of all the internal consistency of the scale, e.g. Cronbach's alpha. Perform also item-total analysis to check which items have contributed more or least to the underlying construct. If a given item is a poor contributor to the scale's internal consistency, then it should be dropped out, Otherwise, you will need a really huge sample size to run FA, which is dependent on the inter-item correlation.

The SPSS used to run, before the FA, an inter-item correlation matrix and 2 tests: Bartlett's test of esphericity and Kayser-Olkin-Meyer (KMO) sample adequacy to test the strength of correlation. For each pair of items, the correlation should be higher than 0.35 in the item matrix. This will guarantee that you don't need too much dimensions to explain the total data variance. The Bartlett's test should be significant and the KMO > 0.65. Otherwise, you will violate the assumptions for FA. Sometimes, you also can request the anti-image matrix to see how well the pairs of items correlate (should be > 0.90).

For FA of scale building, most psychometrician use principal component analysis (PCA), as they only consider inter-item correlation and relax the error assumption of a more formal factor analysis. Also, do inspect the commonality for each item. This analysis also can be helpful to define which items to be used as dependent variables in the factorial model. In each step, the clinical wisdom will guide you to delete redundant items. The good sense should help you to achieve a valid scale, which underlying construct you intended to measure.

In summary: first reduce redundant items and then perform factor analysis, if your data allow such kind of analysis.

Levente Kriston outlines a number of important strategies, many of which are not constrained by sample size issues. I'll just quickly add or emphasize a few finer points:

1. Are you seeking to make clinical decisions (where appropriate cutoffs are more important to estimate) or to understand relationships between variables/constructs (where appropriate correlations are important to estimate)? Answering this question would help guide which items to select -- for example, an item with low variance might not inform a correlation coefficient, but the item might be a powerful indicator for some type of low-base-rate clinical decision.

2. Is some item content very similar? Sometimes similar content is good (to make sure respondents understand a question, ask them a few times in a few different ways)...but sometimes it is bad (ask the same questions a few times to raise alpha to nominal .70 levels, at the cost of frustrating the respondent for asking repeatedly). You might examine the items to see whether the content is parallel (or even identical) and winnow accordingly.

3. Are frequency and impact highly correlated? Consider one or the other if respondents are not distinguishing well between them, or if one of them turns out to be much more important than the other in light of your goals (e.g., see #1 for consideration of goals).

I'll end with a shameless self-citation for shortening a personality measure, where the goal was of the correlational type (overall reliability and validity concerns):

Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The Mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18, 192-203.

Best of luck!

Fred

I agree the most with Fred's points. Not that you will not use FA at some point and in some way, but you have no degrees of freedom with that many items and that sample size. Particularly because mainly you have double the items by having two scales (frequency of impact). Once you have reduced your item pool, probably a well crafted CFA (Mplus, categorical items, depending on the distribution of responses) would do the trick.

First, go through Fred's points.

Just to pass on a few additional papers that might be helpful for reducing your scale:

Smith, G. T., McCarthy, D. M., & Anderson, K. G. (2000). On the sins of short-form development. Psychological Assessment, 12, 102-111.

Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167-194.

Widaman, K. F., Little, T. D., Preacher, K. J., & Sawalani, G. M. (2011). On creating and using short forms of scales in secondary research. In K. H. Trzesniewski, M. B. Donnellan, & R. E. Lucas (Eds.), Secondary data analysis: An introduction for psychologists (pp. 39-61). Washington, DC US: American Psychological Association.

Thank you all for your in-depth, creative, helpful suggestions--this gives me a lot to work with!

Here's a bit more information about the project: I am creating a scale to measure racial & ethnic microaggressions in clinical settings (subtle, often unintentional slights or offenses based on perceived racial/ethnic characteristics of the patient). The long term goal is to use the scale to examine the relationship between microaggressions and therapeutic outcomes (e.g., early termination of treatment, satisfaction, improvement, etc). This is just one small piece of the mental health disparities research--in this case, the question of why treatment is reported as less satisfying & effective by racial and ethnic minorities than White clients. There is a similar scale already in existence, but it is 10 items long and has been used in only 1 paper since its 2007 publication. I'm trying to improve on that existing scale by creating a slightly longer, multidimensional research tool with enough reliability & validity that it will be useful down the road.

Why am I reducing the number of items? The main reason is a practical one. Scales over 30 items tend not to get used in psychotherapy process & outcome research.

How did I create the items? Last summer a team of psychology & sociology Ph.D. students wrote 400 items based on 22 separate themes I identified from the literature. Themes include "Invalidation of interethnic differences" (Assumption that all members of one large racial group look/think/act alike); "Accused hypersensitivity regarding racial or cultural issues"; "Colorblindness"; "Myths of Meritocracy"; etc. Out of the 400 items, we used a group consensus process to eliminate redundant and poorly worded items, resulting in a set of 128 that we felt good about. I tested those in an online survey over the past three months, and now I need to stop and write up the results for a paper that's due by the end of the summer. While I could continue testing all 128 items, I would rather use the data I've collected to get rid of items that are not performing well.

Why ask participants for both frequency and impact ratings? A) The literature indicates that these interact in interesting ways, and do not always perfectly correlate. B) In my case the difference in the two ratings has been very telling so far--some items have high frequencies with very low negative impact, suggesting that participants didn't mind when their clinician did or said these things. This is probably the difference between the academic sensitivity to these issues vs. how they are experienced in day-to-day life. For example, the item "My clinician avoided criticizing my family’s culture or values" was written with the idea that this could be offensive to someone; i.e. the clinician is using kid gloves because she/he assumes the client will take offense if she/he is more direct. However, the data show that participants seem to appreciate a somewhat hands-off approach re: family culture & values. So here is an item that can almost certainly be removed from the pool--this one is unlikely to predict negative outcomes.

I'll stop there, but please let me know if you have more questions, and thanks so much for the suggestions thus far!! I will begin experimenting with some of the techniques you've recommended.

--Billy

First of all, as you said, your sample is too small for that number of items. You need almost 15 subject for each item, ir order to guarantee variability (despite being all items in a Likert Scale).

Second, exploratory factor analysis, especially in the first step is very useful, because you can verify how items correlate. But I think this is the last step. Is very very useful when you want to check construct validity.

Third, are you using SPSS? SPSS do some great reliability analysis, SPSS compute reliability index when you delete one item. If an item when affects reliability idex, you must be careful with it. If reliability is better, this item must be revised, if not, this item probably is important for the whole test reliability.

Also check means and variance for each item. If an item has poor variability, you must revise or remove that item.

And check all items skewness, positive and negative skewness items may have psycometric consequences.

To summarize: first of all, you have to do an exhaustive descriptive analysis of all items, and next, continue with reliability and finally compute a factorial analysis.

I hope this helps!

Good luck with your test and your research!

I messaged Peter Smetaniuk about this directly, but would be interested to hear what others think...

Regarding this comment: "As suggested by others in this forum, do a correlation matrix and discard variables (items) with extremely low (< .1) and extremely high (> .9) coefficients." Does that mean ANY instance of a correlation of < .1 or > .9 makes both variables good candidates for discarding? Or does it mean that ALL correlations for the variable in question should be < .1 or > .9 in order to discard it?

What I'm seeing in my inter-item correlation matrix is that variables correlate very differently with each other. All the variables have at least one correlation below .1, while some have 60 or more such correlations. Only two variables correlate > .9.

Would some sort of correlation mean be a good criterion for which variables to discard? Right now the correlation means range from .47 to -.03. Would it be a good idea to take all those with, say, mean correlations below .3 and discard them? Just to give you an idea, that represents 49 of my 128 original items.

What about Clark & Watson's (1995) recommendation that interitem correlations be between .15 and .50? This is particular relevant for a multi-dimensional construct, which mine is. Maybe I should be weeding out semi-redundant, but not totally collinear items as well?

My concern about throwing out items with low correlations is that many of them have high correlations with OTHER generally low-correlating items. I'm taking this to mean they are tapping some other aspect of a multi-dimensional construct. This is, of course, what a factor analysis would show, and I do plan on doing that after I do some initial weeding out of poorly-performing items. Just not sure of the best way to do that first weeding-out step. Many thanks for any help you can provide.

--Billy

Reference

Clark, L. A. & Watson, D. (1995). Constructing validity: Basis issues in objective scale development. Psychological Assessment, 7(3), 309-319.

Another quick question-- I Googled Horn's parallel analysis engine and came up with the following: http://ires.ku.edu/~smishra/parallelengine.htm

Can anyone tell me how to interpret the results? What do Root, Means, and Percentile refer to and how should I use these results in SPSS?

You can find FACTOR for download, free of charge, here:

http://psico.fcep.urv.es/utilitats/factor/Download.html

http://psico.fcep.urv.es/utilitats/factor/Download.html

William Somerville

Compared to the number of items of measurement, your sample size is really very small as you have no degrees of freedom and an FA would certainly result in spurious outcome. Statistically speaking, there is no method to reduce number of items if all are deemed necessary. However, on theoretical level, being an expert in the field, you may develop certain theoretical constructs (categories) and using available statistical method add certain items, after relevant and warranted transformation, into these meaning ful categories (constructs), items which to you on theoretical grounds belong to these constructs. After reduction of items into a manageable number, you may perform FA or if items are reduced in a very small number of constructs you may use them directly for further analysis. It is not always necessary that FA is used.

@Jan Boehnke

May be I am wrong. It is not the question of small sample size, rather that of sample size in relation to number of items (variables). Naturally, in order to use the variance-covariance matrix, one has also to calculate correlation matrix. In order to find out whether coreelation between two variables is to be significant or not, we need degrees of freedom.

@ Dear Jan Boehnke

My reference to degrees of freedom was based on the fact that factor analysis is based on the correlation matrix of the variables involved, and correlations usually need a large sample size before they stabilize. In this context I share the view expressed by Child (2006:90) “Decisions about sample size are clearly needed before a research begins. The smaller the number of individuals, the greater is the effect on sampling error and the reliability of correlations. If you can read a ruler to an accuracy of one millimetre, the influence of this is proportionally greater for a length of one centimetre than for one metre.”

Further, several researchers have commented on the minimum number of individuals/observations usually required in relation to the number of variables in the analysis (Gorsuch, 1983; Baggaley, 1982 and Comrey and Lee, 1992). Gorsuch suggests an absolute minimum of 5 individuals/observations per variable/item, but with minimum individuals/observations of 100 to give 20 variables. Comrey and Lee's (1992) regarding sample size have advised: 50 cases are very poor, 100 are poor, 200 are fair, 300 are good, 500 are very good and 1000 or more are excellent. As a rule of thumb, they suggest a minimum of 10 observations per variable is necessary to avoid computational difficulties.

However, I am aware of the fact that in psychometric analyses researchers sometimes use more variables/items than individuals/observations.

If yourinstrument is based on a theory, then you may identify smaller subfactors a priori, thus you may contact factor analysis for each subfactor indepentently solving the problem of the small sample

William,

You may have to reduce the effective number of parameters to be estimated by using penalized factor analysis. This is similar to ridge regression where one can have more regressors than subjects.

See:

http://takane.brinkster.net/Yoshio/c042.pdf

for a paper, and

http://cran.r-project.org/web/packages/fanc/

for a program.

Best,

Henk

There are 3 ways to reduce the dimension of observations:

1. Principal Component Analysis, which tries to reduce the dimension of data by linearly independent linear combinations of original variables; This leads to Y_i=e_i'X where e_i is the i-th eigen vector of the matrix of correlation coefficients. The resulting pc's would be linearly independent since the eigenvectors of a matrix are linearly independent.

2. Factor Analysis, which considers a model like X=LF+e where X is the vector of observations, F is the vector of hidden factors, L is the loading matrix and e is the vector of errors.

3. Classification of Variables, which tries to classify the variables into some similar groups.

For more details, please refer to SPSS or other statistical softwares.

Two other thoughts... It is possible to use item packets or "mini scales" to reduce the sample size needed; group similar items together in item packets and experiment with including/excluding different packets from the analysis. Another non-traditional method would depend on whether your instrument is formative or reflective. If reflective, then classical measurement theory is best. But if formative, you could model your variables (e.g. with SEM) to determine which items to include.

In EFA/PCA, the sample size can be smaller if the homogeneity of the subsets of items is higher. PCA and PAF are nonparametric, so you may want to consider those.

I respectfully recommend taking a course in Testtheory.

There is an excellent book on this issues: Scaling procedures http://www.amazon.com/Scaling-Procedures-Applications-Richard-Netemeyer/dp/0761920277

Nevertheless, and regardless of this issues pertaining sample size or sample size/nº of factors ratio, with PCA you can and you will easily reduce the number of items. Be a little conservative concerning item elimination, and probably you can easily cut most of the extra weight your item pool is carrying.

I too recommend PCA or PAF, but would also recommend Varimax rotation. I feel very unconformable with the theoretical justification for Oblimin in a PCA.

Most of the options for reducing items of a scale has already been recommended and relevant references have also been shared. In this situation I am not in a position to recommend any thing new. However, I think that all the suggestions (given earlier) and some more may be organized in step-wise fashion to reduce the items with even a small sample.

step-1: First you may give all the items to a panel of experts to rate them on a 5-point or 3-point scale how far they reflect the intended construct. Please brief the experts about your conceptual frame of the construct and the strategy of item writing selection. (if this step already done then may be skipped).

Step-2: Select those items that have received highest ratings from most of the raters. This will result in a smaller pool of items, Any index of Inter-rater agreement may be used for this purpose.

Step-3 (if you consider the scale taps different domains of the construct): At this step you may classify the items according to the preconceived domains. Again use of different experts is recommended. Select only those items for each domain where the experts show strong agreement. Drop those items which have been classified in more than one domain.It will result in still smaller pool.

Step-4: Now use classical test theory and do item analysis for each domain separately and select only those items which were found adequate as per the multiple criteria such as item-total correlation, alpha-if-item deleted, squared multiple correlation etc.

Step-5: (if items have not been reduced significantly). If up to step 4 you have not arrived at a smaller item pool suitable for doing further statistical analysis for item reduction then you may try this step. Otherwise go to the next step. At this step, you short the items according to some indices of item reliability/validity, say for instance item-total correlation. And then select the first n items (n predefined to meet your criteria of variable to participant ratio for carrying advance statistics) with the best psychometric properties.

Step-6: Now conduct the widely used methods such as factor analysis/principal component analysis to reduce the number of items further and classify them into the empirically derived latent factors.

Step-7: Now again, you may conduct the item analysis as per the classical test theory separately for each of the empirically derived categories and see whether there are some items with poor psychometric properties. If yes, then you may drop them.

If you have also collected information on some external criterion measure, then you may use it also for selecting the best items (that correlate strongly with the criterion measure). This may be done near step-4 (preferable before it)

Hope, if one follows the said approach then he/she may arrive with a smaller but psychometrically sound pool of items to measure the intended construct.

I would suggest Factor analysis and run the KMO for sample size adequacy to see if your sample is strong/large enough.

William,

In several cases researchers use a larger number of item than subjects. Theoretically it is not recommended as the correlations are not stablised in this situation. Most experts suggest five to 10 subjects (observation) per item or variable. If from your theory it is possible to let go or cluster certain number of items, do it confidently and bring the number of items appropriate for a reliable factor analysis. Otherwise, work out correlation matrices for state and impact responses using Spearman's correlation. The variables which are observed to have the highest correlations may be clustered together to reduce the number of items.

William,

I also think you have too much itemsin your scale.. An easy way to delete some items is to estimate their indice of discrimination with ETS technique for exemple. If the indice of discrimination is not acceptable, well delete the item of your scale.

Jaouad,

what is the ETS technique?

Hello Gian,

ETS provide a Difficulty index (pi) and (d) Discrimination index (Di) were evaluated by assessing the criteria based on the Educational Testing Service (ETS) method. This system is based on the sum of 1) the number among the ten participants who obtained the highest marks on the test (10++) and met the criterion and 2) the number among the ten participants who obtained the lowest marks (10--) but met the criterion. The pi index is the sum of these two statistics whereas the Di index is the difference between these two statistics. Success in meeting a criterion is defined by the researcher (for example: 60%). Therefore, in the case of my research (see ref), a criterion is considered as being met if the participant obtains the following minimum marks: 18/30 (from the university professor); 36/60 (from the professional supervisor); 6/10 (student self-assessment). According to the ETS method, an index of difficulty greater than 17 indicates that the criterion is too simple whereas an index lower than 7 indicates that the criterion is too difficult. A criterion for which the degree of discrimination is lower than three indicates that the criterion cannot be used to differentiate between the subjects.

Ref: http://ir.lib.uwo.ca/cjsotl_rcacea/vol3/iss1/5/

Hi Jaouad. I believe that the ETS method you describe would fit for analyzing items with true-false coding (aptitude types). However, it seems to me that William is using the attitudinal/likert type of questions. Nonetheless, I am also interested in statistics on difficulty or discrimination index for such attitudinal items. Can you describe an alternative for attitudinal items?

Best regards,

Imam

I think that what Jaouad have described are CTT (Classical Test Therory) procedures for calculating difficulty and discrimination of dichotomous items (item analysis). I have never seen them associated with the EST acronym before.

Imam, CTT procedures for calculating difficulty (affectivity) and discrimination of items are available for both dichotomous and and likert-scale items.

I am completly agree with Gian:

Even though with attitudinal/likert type of questions (let's say a 5 lickert point scale), you can use CTT procedures for calculating difficulty and discrimination index.

The only thing to do is to dichotomise the scale with your own standard : for exemple : 'more than 3 on 5' means 'succes'.

It seems that you would like to take a first pass at reviewing the quality of your items before collecting additional data. Since sample size is your concern, I am not sure that the EFA being suggested by most folks is going to be your solution. You should try and first perform a detailed item analysis. I would recommend evaluating the internal consistency of the measure (assuming it is a single scale), and review the large item-intercorrelation matrix. Items with extremely low correlations with other items, items with very low item-to-total correlations, etc., might be flagged. Another criteria to use would be alpha if item deleted. You can also use the classical item discrimination analysis (not the IRT based analysis), wherein you identify participants with the highest scores, and those with the lowest scores, and then run an item analysis to identify any items with negative discrimination values. These are all just some basic suggestions. Some of the citations provided by Samuel should help you further. If there is someone in your organization who has experience with item analysis, I would recommend a collaboration to make sure you are headed in the right direction. Good luck!

A easy way to reduce items is to do factor analysis, main components...and you can decide how many components you can add to the list of items. greats!

I would recommend using an exploratory factor analytical approach (EFA) in the first instance. This may also include principal components analyses (PCA) , fundamental difference statistically being the latter model produces an orthogonal component solution where factors are not correlated. There are large amounts of literature on these classic test theory (CTT) based methods. Even in EFA its important to have some a priori theoretical understanding of how postulated factors might load on items (essentially the items should have been developed with a fairly clear understanding of the latent construct that you'd like to be able to observe). The EFA process is also important in providing a diagnostic understanding, if you like, for poor performing items and this may help you reduce your item pool - hopefully not at the expense of the measures content validity.

We have just published an article about scale purification, i.e. the process of eliminating items from multi-item scales. We have used the example of SCM, but our framework can be applied to any other discipline. Download: https://doi.org/10.1108/SCM-07-2016-0230 (or request via my ResearchGate page).

I would suggest using either Principal Component Analysis (PCA) and/or Exploratory Factor Analysis (EFA).  In either case, I would recommend retaining the variables/parameters that explain 80% of the variance/effect, and dropping the rest.