Hi, there.
I am working on my Ph.D dissertation. I have run a factor analysis of 56 items with maximum likelihood and direct oblimin method. It turns out that only one factor can be extracted (scree plot and Egenvalue>1) from a pool of 56 items, and one item has excessively large loadings (about 70%) on the single factor.
Examining respondents' responses to the items, I found many cases of identical answers to consectutive questions (at least 5). Does this suggest for the revision of the survey?
Thank you very much.
Not required. Please merge similar questions while analyzing. Look for consistency in responses.
Hi, this will probably be the cause of your one-factor solution.
How large is the sample? I am puzzled that you said you found 5...5 what? Individuals?
Generally, an avenue would to do some sort of latent class or mixture modeling approach to separate the sample into the groups of people who had given valid answers and those with severe response styles. But this requires a substantial N.
Van Rosmalen, J., Van Herk, H., & Groenen, P. J. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157-172.
McIntyre, H. H. (2011). Investigating response styles in self-report personality data via a joint structural equation mixture modeling of item responses and response times. Personality and Individual Differences, 50(5), 597-602.
Best,
Holger
If the items all reflect the same or similar construct and are written in the same direction, you would expect people to respond the same to them. If the items are supposed to reflect different constructs, but collapse to a single factor, maybe the constructs you have in mind are not so different. I have had that happen.
You say there is only one factor, but not how you reached that conclusion. Typically the first factor accounts for a lot more variance than the second. I would look at the scree plot to see if there might be more than one. I would at a minimum rotate two factors to see if several items form a clear second factor and whether it makes sense.
Manzu He, I suspect that in the construction of the questionnaire you did not start from a pre-existing theoretical model, because in this case the factors were better defined in the factor analysis.
From the data presented by you, it results that you are in the primary analysis phase of the questionnaire development. I recommend that you complete this phase by selecting the interrelated items and including them in an intermediate questionnaire.
Best regard,
Intermediate analysis phase which involves:
1. pretesting the questionnaire on a large number of subjects, at least double the number of items, but not less than 120 subjects;
2. analysis of the means and standard deviations for each item. In this stage, the items that register deviations of more than one point from the theoretical average will be marked. These items approach a truism and are therefore a factor in biasing the answers. Regarding the standard deviation, the items that register a flattened, bimodal curve will be marked. This type of curve indicates that the question was understood differently by the respondents to the questionnaire which can lead to errors;
3. analysis of the internal consistency of the questionnaire. Depending on the coefficient you expect, you can proceed to eliminate the items marked in point two.
After completing this phase, move on to the final phase which involves calculating the fidelity and validity coefficients.
If after going through these phases the results are not what you expected, I think it is advisable to proceed to build a questionnaire based on a pre-existing theoretical model.
Best regards,
Dear Manzu He,
I would suggest another approach, complementary to my colleagues. I mean, if the number of subjects by item is ok, at least 300 subjects, theory clarity, etc as posted before. If all this is ok, to help you to choose which items are interesting to your instrument, use IRT analysis (I will offer some classic references above, but with R you can easily run it, or if you are in a university with more resource, you can use Parscale or other software). Keep the items with a better discriminant parameter. Another possibility is to conduct the Smallest Space Analysis and you can see if the items are so close that they are measuring more or less the same (another classic approach, with some critics, but interesting). My last and more trend suggestion is to apply network analysis.
References suggested:
du Toit (Ed.), M. (2003). IRT from SSI. SSI Scientific Software International.
Childs, R. a., & Chen, W.-H. (1999). Software Note: Obtaining Comparable Item Parameter Estimates in MULTILOG and PARSCALE for Two Polytomous IRT Models. Applied Psychological Measurement, 23(4), 371–379. https://doi.org/10.1177/01466219922031482
Guttman, L. (Jerusalem/Israel). (1982). Addendum: Facet theory, smallest space analysis, and factor analysis. Perceptual and Motor Skills, 54(2), 491–493.
Schwartz, S. H., & Bilsky, W. (1987). Toward A Universal Psychological Structure of Human Values. Journal of Personality and Social Psychology, 53(3), 550–562. https://doi.org/10.1037/0022-3514.53.3.550
Epskamp, S. (2017). Network Psychometrics. Academisch Proefschrift. http://www.worldscientific.com/doi/abs/10.1142/S0218539313500216
Bringmann, L. F., Lemmens, L. H. J. M., Huibers, M. J. H., Borsboom, D., & Tuerlinckx, F. (2015). Revealing the dynamic network structure of the Beck Depression Inventory-II. Psychological Medicine, 45(4), 747–757. https://doi.org/10.1017/S0033291714001809
I suggest you to have an intermediary questionnaire since there could be an issue in your model.
What you are describing presents the same as what psychologists (and perhaps marketers) call the "foot-in-the-door" response. Try reverse coding some of the questions to prevent and/or spot this. Here's a layman's definition of that phenomenon: https://www.alleydog.com/glossary/definition.php?term=Foot-In-The-Door+Phenomenon
You could also try mixing up questions that have naturally different expected responses.
Finally, who are your subjects? Do you have respondents that will tell you anything just to get out the door (and therefore do not read the questions)? Consider the environment, respondents and incentives, and whether they might play into your situation.
A way to reverse survey items with identical information is to validate the items using Factor Analysis to drop or merger the items with low coefficient. This is because the key concept of Factor analysis is that multiple observed variables have similar patterns of response as the items are associated with a latent.
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running