Hello,
I was wondering if anyone would be willing to share any literature/knowledge on determining item order in scale development.
For context, we have developed and tested a scale. Question order was randomized in Qualtrics and the scale has since been validated. I have read several papers on scale development and validation, but they don't discuss item order within individual questionnaires at depth or at all. As the order of the items in a questionnaire should be consistent when used in the future, I was wondering at what stage this order is decided and how it can be tested (e.g., in terms of validity and reliability).
Thank you kindly in advance!
Kat Schneider
Hello Jekaterina,
In surveys, one common consideration associated with item/question/stimulus order is the advice to separate sensitive or controversial probes from others with more neutral items, and to begin the survey with more neutral items.
In fixed-forms achievement testing (same test given to all), item order is commonly recommended to follow an "easy to hard" sequence, based on the challenge or demand of the questions. The rationale for this was not to stun an examinee at the outset of a test with overly hard items. However, a number of studies comparing randomized to difficulty-sequenced items have shown negligible differences.
Response-contingent testing (computer-administered, usually following an IRT model) is, in fact, a method that purposively selects items based on their level of challenge. In the absence of other information, one would start administering a "moderate" difficulty item/question/task to the examinee, then adjust the subsequent item based on whether the given answer was correct (harder items follow correct responses; easier items follow incorrect responses). So, this ad hoc sequencing method (adapting to the proficiency of the respondent) is pretty common. This tactic leads to a very efficient process for estimation of proficiency.
Since your initial analysis was based on a randomized order of items and appears to have functioned satisfactorily, there's probably little need to consider changing that going forward.
Good luck with your work.
Items are grouped based on the dimensions of the scale, regarding the order there is no provision, unless there is a suggestion for item modification when doing Confirmatory factor analysis (CFA).
Dear Scheider,
I completely agree with David Morse. However, it is not common to conduct and organise attitudinal items based on IRT. Pasquali (text in Portuguese) already affirm that we should present attitudinal items using IRT.
We published a couple of papers doing it. We suggested that the arrangement of the items should be done applying IRT analysis (above). However, all are in Portuguese (abstracts in English) and I don´t know if you can read them.
Pérez-Nebra, A. R., & Torres, C. V. (2010). Medindo a imagem do destino turístico: uma pesquisa baseada na teoria de resposta ao item. Revista de Administração Contemporânea, 14(1), 80–99. https://doi.org/10.1590/s1415-65552010000100006
Torres, C. V., & Pérez-Nebra, A. R. (2015). Evaluación del individualismo-colectivismo vertical-horizontal en Brasil: una propuesta de medida. Perspectivas En Psicología, 12(2), 9–21.
Pasquali, L. (2004). Psicometria: teoria dos testes na psicologia e na educação. Petrópolis: Vozes.
- Badges
- Science topic
- Similar topics
- Papers