Researchers do shorten questionnaires but very little is known (?) about good practice in this field. There are probably some methods, both rooted in IRT and CTT, but do someone empirically tested what metod is more useful in what circumstances?
Hi Emilia
Do you mean shorten a questionnaire or shorten a scale within a questionnaire - the two are different. For the former, just get rid of items that are either superfluous, poorly answered, or suffer from floor or ceiling effects. For the latter, the usual procedure is via a factor analytic approach, trying to reduce the number of items, but still maintain the factor structure and psychometric properties.
Adrian
You can start with exploratory factor analysis (not principal component analysis) to determine the number of factors and which items under each factor. This process will normally remove some items of the original scale. If you need to trim the scale further, examine each factor one at a time by removing item(s) that will lower the Cronbach’s alpha of the factor.
I agree with Eddie. But primarily, you could have done some form of pilot testing to increase the validity of your questionnaire at the very beginning. This could have been done with quantitatively/qualitatively or in combination. I prefer a combination of semi-structured interview and a questionnaire since it helps elicit unarticulated thoughts in the participants.
After performing the above, you can reconstitute your questionnaire, eliminating themes that are not important and including more valid ones. This will definitely help limit the number of questions per construct.
Alternatively, you could run the questionnaire anyways and run factor analysis to identify the number of factors related to each items. I suggest you run a parallel analysis to get a clearer number of constructs that can be generated from your data.
After eliminating items based on the rotated matrix in factor analysis, you could run Cronbach alpha or Kruder Richardson analysis to determine which items can be deleted if the scales seem not statistically significant.
However it is my suggestion that you design the questionnaire and run a pilot first. This will save you a lot of trouble later and you can even generate supplementary questions to slot into each construct in case Cronbach is low later.
Rabin
Hi Emilia,
probably a bit late for you, but maybe interesting for other researchers struggling with the issue of shortening scales:
Scott Tonidandel and colleagues presented a pareto optimization procedure to shorten scales according to a set of criteria (e.g. predictive validity of a chosen set of items) at SIOP2019. There is a beta version of the tool to determine the best items for the purposes of a specific study:
https://orgscience.uncc.edu/about-us/resources
However, pilot data are required.
Best,
Oliver
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire