Hi community,
I am trying to determine what is the best way to calculate the internal consistency of a newly developed self-esteem scale. I am working together with two other researchers on the evaluation of this scale, that is planned to be used in an international NGO's impact surveys. This self-esteem scale has been filled out by two samples of Ghanaian youth, and we are currenly analyzing the data. We are experiencing some puzzling findings while doing this.
The scale exists of 3 questions in the short version and 5 questions in the long version. Question answers are rank ordered (from low to high self-esteem), so the questions are of the ordinal measurement level. We used the McDonald's omega to compute the internal consistency of the questions, but I am starting to wonder if this is the best procedure. Do you maybe know what is the best suitable manner of calculating internal consistency with our particular set of questions, especially since there is a small number of questions and they have an ordinal measurement level? Ideally, this is a method we could use in SPSS, Stata or JASP.
We also have low levels of the McDonald's omega, about .310 (minimum should be .6 out of 1), which is lower than we hoped for. This could have something to do with a number of things, but what is interesting to note is that both the 3-item and the 5-item scales correlate significantly (p=
Coefficient alpha (Cronbach's alpha) is widely used with ordinal scaled items, even though it its based on correlations.
You may want to use Cronbach alpha for it gives you the coefficient of the items or see the performance of each item using inter-item correlation from Factor Analysis.
You may also want to use Kappa coefficient
Cronbach's Alpha is used to estimate the Reliability Coefficient for Likert-Type Scales. The attachment gives you insights on how to calculate, interpret, and report Cronbach’s Alpha.
Good luck,
Sharon Schroen, the above advice, namely that coefficient alpha is often used to identify internal consistency, is correct. Furthermore, McDonald's omega is often recommended as better than alpha, so I can see why you used the latter.
However, alpha is highly dependent on the number of items entered into the analysis, so with as few as three or five items, I suspect you'd struggle to get a "decent" level of alpha even if there is quite a good level of association between them. I suspect you'd struggle to get a decent omega, too, because I have found it to closely resemble alpha.
Many people seem not to be aware that high levels of alpha (and also omega?) can be achieved merely by having a reasonably large number of items (e.g., around 25) even if some of those items carry very little association with each other. We have certainly found it in our research - and in fact I have refused to calculate alpha when a large number of items are involved because I believe that alpha is likely to be meaningless under those circumstances.
In support of what I've just written, it's interesting that subscales that are touted to be independent of each other very often have lower alphas than the alpha that's produced when all of the (different) subscales are combined into a single analysis.
The following two articles might be useful if you're not familiar with them already:
Cho, E., & Kim, S. (2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207–230. https://doi.org/10.1177/ 1094428114555994
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98
Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353. https://doi.org/10.1037//1040-3590.8.4.350
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
I hope the above is helpful - and all the best with your research.
Dear Sharon Schroen , absolutely agree with Prof. Robert Trevethan however don't you think 3 and even 5 items are really very few to develop a new scale? How did you decide on a "Short-Scale" of 3 items from a newly developed non-validated only 5-items scale?
According to my understanding, developing a new scale requires some trial and error with a set of fairly large pool of items (developed thoughtfully based on the definition of that construct) and then with EFA can reduce the set of items to have a practical number of reliable/valid items.
Regards
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS