Hi everyone, I'm dealing with a one-dimensional 10-items psychological scale that is widely used in the literature to measure a certain psychological construct. The fact is, however, that there are no studies in the literature that analyze its internal validity (using factor analysis or other methods), but only results regarding reliability (Cronbach's alpha or omega) are always presented.
In our data (N > 500) this scale has good reliability, as reported in other studies, but the CFA I conducted (Mplus, estimator = MLR) does not fit [SBχ2(35, N = 557) = 553.958, p < .001; SCF = 1.42; CFI = .710; TLI = .627; RMSEA = .163, 90% CI [.151, .175], p < .01]. I believe this is an item-wording and item content overlap problem, because several items are similar to each other (or even synonymous). In fact, the modification indices provided by Mplus suggest that several items' residuals are strongly correlated (M.I. over 100.0).
Therefore, I would like to present these results and critique the validity and usefulness of the scale. Could you please point me to any references/studies I can cite that show that these types of situations are very problematic for the validity of the scale and can therefore invalidate the research results?
I have consulted Podsakoff et al. 2003, 2012, but do you have any other useful references in mind for my purposes, specific on this issue?
Thank you very much
Lorenzo
Dear Lorenzo Filosa
I suggest you go through the following articles first:
1. https://www.researchgate.net/publication/12032567_The_Effect_of_Item_Content_Overlap_on_Organizational_Commitment_Questionnaire-Turnover_Cognitions_Relationships
2. https://psycnet.apa.org/record/2001-16970-015
3. Article Best Practices for Developing and Validating Scales for Heal...
4. https://books.google.com/books?id=wC6UDwAAQBAJ&pg=PA96&lpg=PA96&dq=study+on+content+overlap+in+item-wording+of+a+psychological+scale&source=bl&ots=wIWwSduxSp&sig=ACfU3U0gAMfR2cgURrsIojUNSeIjBFaw8w&hl=en&sa=X&ved=2ahUKEwjokJ3y24jzAhXrFlkFHX5xBmkQ6AF6BAgIEAM#v=onepage&q=study%20on%20content%20overlap%20in%20item-wording%20of%20a%20psychological%20scale&f=false
Kind Regards
Qamar Ul Islam
Steven Reise has raised this point several times when arguing for essentially unidimensional models (as opposed to strictly unidimensional ones, which could lead to "cheating by repeating"). The following excerpt is from Reise et al., (2018, p. 681):
It is clear from the heat map that some of the highest correlated item pairs simply are the same question asked twice (and there are several in the DES). We have dubbed this common scale construction practice as the cheating (inflating internal consistency reliability) by repeating (asking the same item twice –a content doublet) phenomenon, which in turn leads to what Cattell (1978) called “bloated specifics”(nuisance dimensions that arise due to shared content when data are factor analyzed).
Content doublets are not an unusual occurrence in personality or psychopathology measures, and the practice has its roots in the demand for obtaining high alpha values. Simply put, when one attempts to write multiple items to represent a conceptually narrow construct (so that item inter-correlations are large, and, thus, alpha is high, and scoring subscales can be justified as “reliable”), it becomes clear that the diversity of trait manifestations is limited and, thus, it is extremely challenging to write items that are not merely repeats of the same stimuli over and over.
Chapter Bifactor Modelling and the Evaluation of Scale Scores
Thank you very much Pablo D Valencia! Your answer is really helpful!
Also, have a look at recent experimental psychometric studies such as the following:
Bandalos, D. L. (2021). Item Meaning and Order as Causes of Correlated Residuals in Confirmatory Factor Analysis. Structural Equation Modeling: A Multidisciplinary Journal, 28(6), 1–11. https://doi.org/10.1080/10705511.2021.1916395
Spratto, E. M., & Bandalos, D. L. (2020). Attitudinal Survey Characteristics Impacting Participant Responses. The Journal of Experimental Education, 88(4), 620–642. https://doi.org/10.1080/00220973.2019.1566200
Hope this helps!
- Badges
- Science topic
- Similar topics
- Psychology