While developing a questionnaire to measure several personality traits in a somewhat unconventional way, I now seem to be facing a dilemma due to the size of my item pool. The questionnaire contains 240 items, theoretically deduced from 24 scales. Although 240 items isn't a "large item pool" per se, the processing time for each item is averages on ~25 seconds. This yields an overall processing time of over >1.5 hours - way to much, even for the bravest participants!
In short, this results in a presumably common dilemma: What aspects of the data from my item analysis sample to I have to jeopardize?
- Splitting the questionnaire into parallel tests will reduce processing time, but hinder factor analyses.
- Splitting the questionnaire into within-subject parallel tests over time will require unfeasible sample sizes due to a) drop-out rates and b) eventual noise generated by possibly low stability over time.
- An average processing time over 30 minutes will tire participants, jeopardize data quality in general.
- Randomizing the item order and tolerating the >1.5 hours of processing time will again require an unfeasible sample size, due to lower item-intercorrelations.
I'm aware that this probably has to be tackled by conducting multiple studies, but that doesn't solve most of the described problems.
This must be a very common practical obstacle and I am curious to know how other social scientists tackle it. Maybe there even is some best practice advise?
Many thanks!
Hello Björn,
Yes, there's been discussion of methods to overcome this challenge.
One, that was mostly applied to classical test theory models, was multiple matrix sampling. David Shoemaker's expositions (see below) are a very good introduction.
The second, which arose from equating in classical test theory models, and calibration of large pools in item response theory models, is that of anchor tests or items (common item sets and unique item sets administered to subsets of the total sample).
Good luck with your work!
MMS references:
Shoemaker, D. M. (1973). Principles and procedures of multiple matrix sampling. Ballinger.
Shoemaker, D. M. (2006). Multiple matrix sampling. In Encyclopedia of Statistical Sciences. New York: Wiley. (Online DOI: 10.1002/9781118445112.stat02622)
Gonzalez, J. M., & Eltinge, J. L. (2007). Multiple matrix sampling: A review. Washington, DC: U. S. Bureau of Labor Statistics, Office of Survey Methods Research. https://www.bls.gov/osmr/pdf/st070070.pdf
Equating/IRT item linking for calibration references:
Kolen, M. J., & Brennan, R. L. (2013). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York, NY: Springer.
Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: MESA Press. (Available for free download: https://research.acer.edu.au/measurement/1/) (see ch 5)
David Morse raised some good suggestions for obtaining a smaller set of items that work as well as the complete/total set. Anchor testing can be done easily if you have a considerable number of respondents who completed the full survey, that is, you have existing data on your hand. If not, I would strongly advise you to re-examine your items and re-consider their face validity and how you proposed the number of components/factors. 24 is a large number and based on factor analysis or PCA, it will be quite a feat if you can obtain factors about half that number. This is because of inter-correlations between factors/components that are conceptually or semantically similar to each other. Based on personal experiences and those of my colleagues, this always happens. Emprically derived factor structure/pattern matrices, by default, normally do not conform with theoretical expectations.
More importantly, if you intend to apply orthogonal matrix rotation, chances are high that you will not get 24 orthogonal factors. Matrix rotation/inversion normally run into complications with too high a number of factors. A detailed explanation of this issue can be found in most text books on multivariate stats that includes factor analysis.
Sounds like you've created an instrument which is sound as it is based on theory - but this version should only be the pilot version, rather than the final instrument. As you see - there are too many items, which will affect the quality of the data collected. Have you conducted any sort of pilot study - factor analysis - and tested to see how those items relate? You'll probably find some items are redundant, and possibly even some scales ... use EFA to explore how the items load - then as you delete items - the number of items in each scale will decrease - you can then delete scales on the basis of insufficient items.
I find David De Vaus' work on survey design and validation very useful:
https://www.amazon.co.uk/Surveys-Social-Research-Today/dp/0415530180/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=ER3KY0K00XZC84NB7GHW
It is also worth considering is an instrument should be trying to accurately measure 24 constructs. For many uses it users of the scale might be interested in only a subset. Are they meaningful ways to divide this instrument by construct type?
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire