May use dimension reduction/factor analysis to see whether you can reduce the number of items in a construct.

Thank You, Sir. However, I would be glad to know if I can conduct factor analysis on the sample size i.e. 50 since this number is assumed to be quite low for conducting factor analysis. Phuong Tran Huy

Oshin Gulsia, I can appreciate your frustration and disappointment if your PI is insisting on some things that produce unsatisfactory results.

One of the fundamental questions is whether your PI demands obedience. If so, I think you're jammed.

If some flexibility (from your PI) is possible, I would begin by looking carefully at the set of items within each construct to check whether there is any obvious redundancy among them. If so, I'd remove all but one of the redundant items for each "set" of redundant items (it might be only pairs you need to consider).

Then I'd obtain an interitem correlation matrix and see whether any items have high correlations and whether, on close inspection, they are redundant. If so, I'd judiciously remove any item(s) to avoid that redundancy.

Then, I'd prepare to run exploratory factor analyses (EFA), employing appropriate procedures such as "genuine" factor analysis, not principal components analysis - and first checking whether your data have sufficient factorability (according to the KMO index and Bartlett's test).

Then I'd check how many factors are probably in your data by using a scree plot or parallel analysis (not the Kaiser criterion).

Then I'd run a "proper" EFA, probably with principal axis factoring and promax rotation and look at the pattern matrix to see what the structure of your construct is likely to be.

Then, based on the items in each factor, for each factor I'd run a coefficient alpha (which has the common misnomer of Cronbach's alpha) and include item-total correlations and the table of alphas if each item is removed to see whether removal of an item results in a better alpha. That should give you more insights about any unsatisfactory items. If any are unsatisfactory, I'd remove them.

Low factor loadings would help you identify unsatisfactory items, also

That would need to be done with each of your four constructs.

If you need more help but don't want to do it publicly, please feel free to contact me privately through RG and I'll see whether I can help you further.

Thank You so much Sir @Robert Trevethan for your help. I am grateful and shall certainly discuss these intricate details with you. As of now can I proceed as you have stated with the sample size of 50 respondents or do I need to collect more responses for running these tests?

Once again, I am grateful for explaining this so beautifully!!

Thank You once again!!

Oshin Gulsia, I'm glad I could help. With as few as 50 respondents, your analyses (particularly EFAs) are likely to be misleading. There are various recommendations regarding sample size for EFA, and choosing the most appropriate size can be complicated.

My rule of thumb, at least for a start, is to have 10 x the number of people as there are in a construct with the highest number of items, which, in your case would be a little more than 300 people.

If that looks difficult, I'd put a lot of effort into checking for redundancy among your items as a first step.

One problem is that you seem to be at the pilot stage of your investigation, so having 300 or so participants might be too difficult. Again, I'd look carefully at the prospect of redundancy.

Redundancy might not exist in your items, however. In some of our current research, we have 40 items and have tried very hard, and I think successfully, to avoid redundancy. Maybe the same would apply to your items - in which case getting the 300 or so participants seems to best strategy.

Robert Trevethan You are right, Sir! I shall check for the redundancy first.

I shall be glad to know one more thing from you! All the values in the correlation matrix for one construct are in positive. Further, deleting any one of the twenty items wouldn't significantly impact the value of CA (increase or decrease by decimal points only). Can I still check for the CA value against the deleted item statistics for CA, delete the one which is slightly raising the value of CA upon deletion, one at a time, and then run the analysis to delete other items successively?

Is it possible, Sir or I would have to collect data on the sample size recommended by you, as the only best alternative?

Oshin Gulsia, first, all of the correlations in any of your interitem correlation matrices should be positive. Ideally, they should be between .15 and .50, but it's not necessarily problematic if they are above .50.

Second, if you have as many as 20 items, removing one of those items is almost inevitably not going to alter the alpha value much. Alpha is sensitive to the number of items in a scale, and by the time there are 20 items, the alpha value is highly likely to be pretty high and constant - and essentially meaningless.

Third, if you are "playing with" coefficient alpha without knowing the factor structure of your items, you are essentially wasting your time and energy. Alpha is a bit tricky at the best of times (something that many researchers don't seem to appreciate), but, if it is used with items that have disparate content, it is likely to be way too fuzzy to be useful.

In essence, I emphasise that you need to clarify the factor structure of each of your constructs before going much further.

Why are you calculating Cronbach's alpha? What do you expect it to show with a scale where you believe there are four dimensions. I assumed if you had a larger sample you'd be using some kind of confirmatory model. What analytic procedures did you assume when you conducted your power analysis?

Daniel Wright I agree with you. My PI actually told me to compute the alphas for each of these constructs. However, this is just the waste of time, as apparent from the results for 88 items. Since, this is the pilot study only, I would be grateful to know if I can go for factor analysis first for item reduction, after increasing the sample size required for the same?

Robert Trevethan I agree with you, Sir. Does it imply that I shall have to increase the number of respondents to more than 300 first and then conduct an exploratory factor analysis for item reduction?

Oshin Gulsia, first, I now realise (mainly because of the post by Daniel Wright) that there could be some confusion between the meaning of construct and factor.

I had assumed, when you wrote "latent constructs", that you were referring to four quite separate things, e.g., self-esteem, ambition, cooperativeness, and conscientiousness - and that each of those things might have factors within them, particularly given that three of those so-called constructs have more than 20 items.

Daniel seems to have assumed you have a single scale that comprises four factors - or dimensions. (Daniel, pls feel free to correct me if I am wrong.)

Can you let us know which is the case, please - perhaps by telling us what your constructs are?

That should clarify things a bit more.

Robert Trevethan I have developed a conceptual model which needs to be evaluated through SEM. I have used four constructs in this model wherein two of these should play/are assumed to play a mediating role in between the Institutional Support and Business Performance (the other two constructs). As explained previously there are a number of questions describing these individual constructs. e.g. factors like financial and other kinds of support for Institutional Factors totaling to 22 items for this particular construct only. My motive is to examine the conceptual model and mediating role of two other constructs in order to support my hypotheses.

Oshin Gulsia, thanks for getting back. First up, and forgive me if I appear to be lecturing you, but it's important to use words such as construct, item, question, and factor such that they have quite specific meanings. From your answer above, I suspect that you mean

latent construct (which might have several components, or factors within it) when you use the word construct,

the words question and item to mean the same thing, and

factor to mean an influence.

Are those things correct?

Furthermore, are you familiar with exploratory factor analysis - and is your PI familiar with it?

@Robert Trevethan You are absolutely right Sir, my PI is not well versed with this statistical technique, hence the problem. By constructs I mean that these cannot be measured directly unless some questions related to these constructs are used to describe them. For example I have one such construct entrepreneurial competencies, with scale adopted from Man (2001). It comprises of a set of competencies such as technological competencies etc.. my PI has suggested me to add 5 questions each for these many competencies which are describing a particular construct. I have to measure interrelationships between four such constructs and that's why it has become so bulky with so many items.

Please,correct me if I have gone wrong somewhere. I've lot to gain from your wisdom!

Oshin Gulsia, the more I find out about your study, the more complex it is becoming, and I therefore wonder at what level you are studying. Is this an undergraduate, master's, or doctoral project - or something else? I ask because that might determine how "big" and "polished" you want it to become.

Apart from that, I have just looked at some information about the Man (2001) scale, and apparently it has eight dimensions across its 53 items. I realise you've adopted something related to this scale, but that makes me think that each of your latent constructs could well have at least two factors within them and therefore it might be best if you were to conduct either exploratory or confirmatory factor analysis, depending on how confident you are that each of those constructs might comprise particular factors and which items would belong to those factors.

So, are you at a level where it's worth putting a lot of effort into your project - and, also, are you confident doing EFA and CFA, anyway?

Certainly, diving in and obtaining some coefficient alphas initially is likely to be quite unproductive. You need, first, to identify which items group together.

Robert Trevethan Thank You Sir, for giving this query your precious time! As suggested by you, I shall go for an EFA first in order to reduce the number of items from my questionnaire.

First, if you have 88 items in your set of measures, the usual minimum number of observations is 5 per parameter in the model (i.e., you also need to include inter-factor correlations and any structural effects). So that implies an N of around 500.

Second, Cronbach's alpha does not address the issue of how many factors are represented within a set of items -- i.e., it assumes a one factor solution.

Third as a "quick and dirty" way to assess the situation, you could look at the item-to-total correlations for each scale and select the 4 or 5 highest items on each, and then re-run alphas with just those high scoring items.

David L Morgan Thank You, Sir, for this valuable information!

I am indebted indeed!

I agree with Robert Trevethan , and David L Morgan. It seems that you have selected more items as you could. I think a few but right items is better than using only many available items.

Thank You so much Omid Mahdieh

Well, I do not understand how can you conduct a pilot study of 88 on 50! Anyways reducing items on the basis of statistics is not a good idea! If questionnaires of your pilot study and thesis are different, you cannot claim the pilot study!

One possible solution might be to use multicollinearity diagnosis! Merge the items that have similar concept and high multicollinearity! Though it is a longitudinal parameter but I think it can form the basis of reducing your items under constructs!

High collinearity is exactly what you do want in constructing a scale, because the goal is to bring together items that measure the same underlying construct.

Also, there is no statistical rule agains testing 88 parameters with an N of 50, but, the results will be relatively unstable, so you should consider it a pretest at most.

Although some degree of correlation is desirable among items in a scale, my sense is that collinearity, let alone high collinearity, is usually undesirable because it is likely to indicate redundancy among the items. Furthermore I think that, if collinearity is found to exist, it requires careful consideration of item content to see whether or not there really is a problem with redundancy.

Although there is, indeed, no rule about analyzing as many as 88 parameters with a sample of only 50 participants, there are good reasons why there are recommendations against such a situation. In my experience, far less extreme combinations of item numbers and participant numbers can easily be demonstrated to result in extremely unstable outcomes.

In any case, given that there are four constructs in the current context, and also that the largest number of items on any of those constructs is 31, I think the number of items to be "worried" about is 31, not 88 - but maybe I'm misunderstanding something there.

Robert Trevethan unless you have a high average correlation, you are going to need a large number of items to produce a decent coefficient alpha. So, if the content of the items is truly different, then then a high correlation between is desirable.

Hello Oshin,

It looks as if you've already gotten a lot of potentially helpful feedback to your query. My reaction is a wee bit different, perhaps because I might not fully understand what's going unsaid in this thread.

So, without any formal verification of items "belonging to" (affiliating with) four latent variables, you found internal consistency reliability estimates in excess of .85 from your pilot sample of 50 cases. Most people would find that good news, not bad news!

So, why are you complaining? What's the source of your urgency to jettison items? Admittedly, what you found with 50 persons may or may not replicate with a larger sample, but why (all of a sudden) the press to reduce the number of items/variables per factor?

If you want to know, what subset of items (for some target number of items, k) if retained, would maximize any internal consistency reliability estimate, this is a pretty straightforward exercise in selecting that subset of k items having the highest average covariance. Doing so, however, may have important and unpredictable effects on how well or how poorly estimates of the associated factor/latent variable correlate with other factors/latent variables. In other words, it could impact the validity. So, whatever path you take, I suggest doing it thoughtfully and carefully, justifying your decisions along the way.

Good luck with your work!

David L Morgan, it seems we disagree about whether collinearity is always good or not. I was accepting Clark and Watson (1995) having recommended that interitem correlations on scales should lie between 0.15 and 0.50, and Briggs and Cheek (1986) having recommended that the mean of those correlations should ideally lie between 0.20 and 0.40.

Briggs and Cheek pointed out that higher values than they recommend could indicate redundancy in item content.

Might you disagree with the above - or are we misunderstanding each other?

In my experience, redundancy certainly can exist and is likely to produce collinearity; however, I've seen collinearity where careful inspection of items indicates those items are not redundant in terms of their overt content. In such cases, I guess individual participants simply have similar responses across a number of items.

Therefore, under some circumstances I believe collinearity indicates a problem with items; in other circumstances collinearity might be an interesting phenomenon that might not need to be avoided.

I wonder whether we define collinearity differently. For me, it refers to correlations that are up around .80 and higher.

Briggs, S. R., & Cheek, J. M. (1986). The role of factor analysis in the development and evaluation of personality scales. Journal of Personality, 54, 106–148. https://doi.org/10.1111/j.1467-6494.1986.tb00391.x

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309–319. https://doi.org/10.1037/1040-3590.7.3.309

Thank You all for your valuable inputs!

As I said, the value of alpha depends on only two things:: the average inter-item correlation and the number of items. With an average inter-item correlation as low as .3, you would at least 10 items to reach an alpha of .8. Increasing the number of items would also increase the size of the sample, according to traditional recommendations to get an N of 10 per item. Of course, there is also a connection between the number of items in a scale and the time it takes to complete it.

In addition, I do not understand the concern about "redundancy" because the assumptions for alpha are all based on true score theory, where the "non-redundancy" in any item is assumed to be nothing but random error.

David L Morgan, I agree with you about coefficient alpha being influenced by interitem correlations (though I think it's not, as you say, the average interitem correlation, but all interitem correlations, isn't it?) and the number of items.

However the notion that alpha needs to be at least .80 is not one that's supported by good methodologists / statisticians whom I'm familiar with, and I think that the notion that there must be 10 participants per item is contentious also. There is a fair amount of evidence that such a rule of thumb isn't supportable (I can provide references concerning that if you'd like) - though I confess it's a rule of thumb I like to start with until I get a sense of what the data might look like.

In my experience, high alphas are usually associated with a large number of items (many of which might not have much association between them) or a small number of items with overt redundancy. As a result, I believe that the pervasive "worshiping" of high alphas is misguided and often demonstrates a lack of understanding about what alpha does and doesn't indicate.

1. The calculating formula for alpha that I used involves only the average correlation and the number of items.

2. I used the value .8 for illustrative purposes; some people consider alphas as low as .6 to be acceptable, but there is no statically defined cut-off.

3. There is indeed a lot of debate about whether it makes sense to work with a standard such as 10 observations per parameter, but it does point to the basic idea that the more items you have in an analysis, the more observations you need in order to obtain reasonably stable standard errors.

4. Redundency in content is in the eye of the beholder. As I said earlier, "if the content of the items is truly different, then then a high correlation between [them] is desirable."

5. I agree that one should not "worship" alpha, and in general I would recommend Structural Equation Modeling as an alternative. But I also think that the advantages of SEM can be overstated, because they it replaces alpha's assumption of equal weighting for items, so it only really matters when there is substantial variation in the loadings.

Overall, for researchers who do not possess the knowledge or experience to use complex statistics, a recommendation to use alpha as a way to evaluate the creation of simple scales is often an entirely adequate approach.

David L Morgan, thanks for getting back. I'll respond to your points according to the numbers you've given them.

1. First, I hope I'm not misunderstood by being taken as saying above that coefficient alpha is calculated by using interitem correlations. I'm aware that (at least usually) it's more complicated than that. But I'm interested in your saying that you calculate alpha by using the mean interitem correlation. That's interesting. If it's easy to do so, would you mind sharing your method, please?

2. I think we are in agreement that there's no definite cutoff for alpha. The way you wrote about .8 seemed to indicate that it was a level of alpha to aim for. And I believe, as you now indicate, that under some circumstances, an alpha of .6 would be quite acceptable. I would say that's particularly the case when the number of items is small.

3. I'm glad you agree with me about debate concerning number of participants in relation to parameters. Certainly, it seems that, in general, the more the merrier. However, one issue concerns how few participants can be enough to still provide valid outcomes. There is now evidence that such things as communalities provide information concerning that. I think it's important to be aware of that when researchers might have difficulty satisfying criteria such as the "10x" rule-of-thumb but have data that don't raise small sample size as a problem.

4. You are right: Redundancy is in the eye of the beholder. However, I have seen a number of scales in which alphas are high and researchers are proud of that despite not realizing (or acknowledging) that at least some of their items are patently redundant. Apart from that, I agree with you that it's possible to have items that are nonredundant and highly correlated. In fact, I'm wrapping up research at the moment in which I believe that has occurred - at least with our particular samples. Many interitem correlations in that research are between .50 and .80 despite the items being quite different from each other in overt content. Perhaps you and I were applying different meanings to the word collinearity. For me, collinearity refers to very high correlation, say, > .80

5. I'm glad we agree about not worshiping alpha. I constantly see it trotted out like some kind of mantra when I'm sure the researchers have little (or any accurate) idea about what it signifies. I'm not sure that SEM is an alternative, but it seems we both have our reservations concerning that.

Dear Oshin, if your goal is to reduce the number of items to a smaller number of uncorrelated components, I would suggest to employ Principal Component Analysis and then apply Cronbach's Alpha to each component.

Thank You all for your overwhelming response! I am grateful indeed!!

Andreas Stylianou, may I ask why you recommend principal components analysis rather than exploratory factor analysis, please?

Apart from that, would you combine items from all four constructs initially, or would you analyse each construct separately?

Dear Robert, in my opinion PCA is considered more appropriate when the goal is to reduce the number of items/variables into a smaller number of components that explain most of the variability of the initial dataset. However, it always depends from the research goal. In social sciences EFA is used more often, but sometimes mistakenly.

Regarding the second part of your question, I would run PCA with all items simultaneously. But again, it depends from the research objective.

This time, I agree with Robert Trevethan that Principal Components is not the favored alternative. the reason is due to technical issues about how it handles commonalities. Instead, I like many others prefer either Principal Factors Analysis or Maximum Likelihood.

Also, I personally would prefer to run the factor analyses separately, because the existing factors were already assessed for reliability and validity in the original publications. So, the question here is whether those earlier results also apply to the current data set.

Overall, the best option would probably be Structural Equation Modeling with correlated factors, but I'm not sure that would be possible (or worth the effort) for a pre-test sample.

Andreas Stylianou and David L Morgan, thanks for your contributions. They have certainly given me food for thought.

My experience could be too limited, but from memory I've found PCA to produce very similar results to EFA, with the exception that EFA has sometimes produced more interpretable results. In addition, the methodologists / statisticians whom I most respect seem to recommend EFA over PCA in general.

Like you, David, I'd prefer to run the factor analyses separately for each construct - at least if the four constructs are regarded as independent of each other. Otherwise, all kinds of "statistical interferences" could be playing into the picture.

However, if the four constructs were prospectively overlapping, I suspect that a factor analysis with all items thrown into the pot might be enlightening - as long as the sample size was sufficiently large to avoid compromising the validity of the results.

Robert Trevethan and David L Morgan thank you for this fruitful discussion. I agree that PCA and EFA most of the times produce similar factors/components. As I have mentioned, I believe that it depends from the research objective. It would also be useful to run both methods to check which one provides the best solution. It is also important to have a relatively large sample to employ such procedures. A ratio of 10 or more :1 (observations to variables) is, in my opinion, the most appropriate to provide stable results.