Alessio Tesi, first, I would tap into the query / issue raised by Viswa C. above about having only two items per dimension. No fewer than four items have been recommended by Fabrigar et al.:

Fabrigar, L. R., Wegener, D. T., MacCallum, R., C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272—299. https://doi.org/10.1037/1082-989X.4.3.272

Second, I confess that I can't really understand the rest of your question(s) - not because, I suspect, I don't know enough about psychometrics (I know a passable amount), but rather because the procedures you mention don't seem to be appropriate for what you might want to do.

For example, running CFA and coefficient alpha with only two items per factor in the each case is highly likely to produce nonsense. And I can't work out what you might achieve by running correlation coefficients.

I suggest you find a tame, authoritative statistician and sit down and have a solid chat. :-)

Start with univariate (e.g., tables, do any of the variables have mostly 5s or 1s) and bivariate (tables or scatter plot matrices jittering the points, and looking at these to decide appropriate bivariate statistics). Multivariate procedures will depend on the answer to these. ONce you get these done, you might start a new question with this information.

Is it recommended to have more than two questions in each dimension? Does inclusion of more questions contribute towards increasing the content validity of the dimension measured? Is there a rationale for the confinement to two items in each of the dimensions?

Alessio Tesi, first, I would tap into the query / issue raised by Viswa C. above about having only two items per dimension. No fewer than four items have been recommended by Fabrigar et al.:

Fabrigar, L. R., Wegener, D. T., MacCallum, R., C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272—299. https://doi.org/10.1037/1082-989X.4.3.272

Second, I confess that I can't really understand the rest of your question(s) - not because, I suspect, I don't know enough about psychometrics (I know a passable amount), but rather because the procedures you mention don't seem to be appropriate for what you might want to do.

For example, running CFA and coefficient alpha with only two items per factor in the each case is highly likely to produce nonsense. And I can't work out what you might achieve by running correlation coefficients.

I suggest you find a tame, authoritative statistician and sit down and have a solid chat. :-)

Thanks to everyone. I will check univariate and bivariate first.

I did not choose the number of item x dimensions by myself, but instead I used an already validated measure. Despite the measure has been already demonstrated to have adequate psychometric properties, I would also check if it works properly in the sample I collected.

Alessio Tesi, it seems you're stuck with what you've been "handed". I wonder how well the measure you're using was validated originally, but that's another matter.

In your case, I suspect that the best you could do would be to hope that each pair of items correlates highly with themselves to suggest they're pretty much measuring the same thing. It might be good to correlate all of the items with each other to see whether the "pairs" are more highly correlated with each other than with other items. I suspect that Spearman's correlations would be good enough for that.

All the best with your research.

CFA-SEM to highlight the proposed model in case it does not fit, it would be advisable to use EFA to confirm it later, unless you can group those factors in subscales to integrate more items according to the theoretical framework

Hello Alessio,

I've seen a number of examples of 2-item per factor scales, most frequently associated with the "Big 5" personality characteristics. The motivations for such brief indicators appears to be: (a) the desire for quick (and a bit "dirty") measures for well-established constructs to speed data collection; and (b) wanting to minimize time/effort/attention load on the participants.

I concur with the suggestions given above, particularly the very good advice by Robert Trevethan . Here are a few, additional points to consider for your specific project:

1. CFA would appear to be a good idea, to affirm that the structure claimed for the measure in fact holds with your sample/target population. Given the Likert-type response scale, you may wish to consider the use of polychoric correlations instead of ordinary Spearman or Pearson correlations.

2. Traditional, internal consistency score reliability estimates (such as Cronbach's alpha), assuming that the two-item per factor structure holds) will not be especially meaningful. McDonald's omega might be more useful. Do recognize that one of the most common ways to improve score reliability estimates is by adding items (so as to get more information about the examinee's level on the characteristic of interest)--two is generally considered too few.

Good luck with your work.

And remember that a scale is only valid, reliable, and/or fair, for a particular data population. If the original psychometric properties were based on data from a different population, then the psychometric characteristics reported there may not apply. This happens fairly often when a scale for a clinical population is used with university students.

David Morse, thank you for adding some thoughts - and thoughtful thoughts at that. I'd like to follow up with some questions if I may:

First, I'm not sure how many response points/options Allesio had on his Likert scale, but I've come across the view that polychoric correlations might be advisable if there are five or fewer options, but that they're not likely to provide better outcomes if there are at least seven response options. Do you have any response(s) to those notions, please?

Second, I'm aware of the recommendation that McDonald's omega be used in place of coefficient alpha, but when I have seen both reported in research, they seem to be almost identical. Do you have the same impression, or is omega often / sometimes quite different? If it is different, by how much?

Daniel Wright, thank you, also, for your thoughts. Exactly! I suspect that what you have written about occurs a lot more often than is investigated or acknowledged.

For some time, colleagues and I have been investigating teacher sense of efficacy (TSE), and many researchers simply adopt TSE scales without checking on the psychometric characteristics of those scales with their own sample(s) despite considerable evidence that those psychometric characteristics can vary substantially from one sample to another. The samples in that field of research are both preservice and inservice teachers. The factor structure for both groups is quite "fragile".

@David typically I would also use and suggest the two Omega measures, but I remember that you need at least 3 items to be able to separate the general and total reliable variance (correct me if I am wrong).

@Robert often Omega_tot and Cronbachs Alpha agree, but the latter should only be similar to Omega_general if there is only one underlying general factor. This additional information together with the lesser strict assumptions for the data structure, makes it much more appealing imho.

Daniel Wright Daniel Wright I agree with your thoughts. That's why I would check the psychometric properties in the sample I collected, 'cause is little bit different from the original target population used for validation.

Thank you Robert Trevethan and David Morse for yours insights. The scale I used has a response format on 5 likert point (from totally disagree to totally agree). According with your comments polychoric correlations could be adequate.

Dear All,

This is an interesting thread, to which I'll add just two more points (can't promise the same level of interest, though!).

1. McDonald's omega can be different with multidimensional structure, which is what Alessio is working with, so that prompted the suggestion. Conceptually, it relaxes the assumptions from those of Cronbach's alpha, which makes it more attractive in many applications.

2. Benjamin Wright made a few pithy observations about the utility of two-item scales in this brief explanation: https://www.rasch.org/rmt/rmt122b.htm

Hope all are staying safe.

Alessio Tesi ,

Please see examples in the two articles below.

Article Exploring Personal Experience of Wartime Crisis Effects on J...

Article Political Advertising Effectiveness in Wartime Syria

Best,

Ali