Dear Gaters,
I have read articles on experimental designs and field surveys wherein researchers conclude that the (survey) questions order had no effect on the study's main results. Typically research indicates ordering of questions might bias result (e.g. Gray, 2003)
I don't understand why researchers reach such conclusion especially when we consider the effects of recall bias, confounding/confounders [Pannucci and Wilkins, 2010], order context, education of participants, and other types of bias.
I do not suggest that these researchers' assessment is wrong or should be qualified. I am intellectually curious and interested in understanding the accepted procedures for testing whether my study's main effects are affected by the ordering of my questions?
How and why can I reasonably scientifically/empirically test and theoretically support that the ordering of my questions does/did not affect my main results.
Thanks for also providing references.
Tobi.
Reference
Gray, K. E. (2004). The effect of question order on student responses to multiple choice physics questions (Doctoral dissertation, KANSAS STATE UNIVERSITY).
Pannucci, C. J., & Wilkins, E. G. (2010). Identifying and avoiding bias in research. Plastic and reconstructive surgery, 126(2), 619.
The basic method is to prepare alternate versions of the survey (or test) with the questions re-arranged. Ideally, every permutation is tested; a Latin Square design is a pretty good next-best-thing. (In this design, each item is equally likely to be in each ordinal position on a participant's protocol, but the order/sequence does not change.) In either of these cases, the ordinal position of the item becomes one of the variables. You can run a one-way ANOVA or include this variable in a factorial design.
My impression is that most of the time this is either done for a subset of the sample (maybe even a pilot group), or, if it's done for everyone, the effect isn't necessarily reported or even looked at, on the theory that the re-ordering controlled for any such effect anyhow. Which is a reasonable argument.
It's also likely to be the case that some lazy, ignorant, or less scrupulous researchers don't bother with any of this, but just write wording like what you mention ("The order of the questions did not appear to affect the pattern of responses" or some such twaddle). Unless an astute peer reviewer (or editor) calls them out on it, this may slip through and appear in the printed article.
Stephen is correct that only way to produce solid research on this issue is to vary the ordering of the question in what amounts to an experimental design.
Thanks Dr. Joy and Dr. Morgan.
A. Latin square design is used in experiments. Does it then imply that only researchers who conduct experiments should draw this conclusion because their participants and/or treatments are randomly assigned? This position might raise review questions of the external validity, generalizability and contextualization of items.
B. If, "It's also likely to be the case that some... just write wording like what you mention" how can reviewers of survey based manuscripts verify such statement. The cost of experiments is in some cases prohibitive. Hence, an existing limitation is that findings from experiments might lack generalizability.
C. Alternatively, Can I interpret the statement, "prepare alternate versions of the survey (or test) with the questions re-arranged" to mean: "1) conduct multiple (pilot) tests; 2) vary the order of the questions for each test; 3) run one-way ANOVA to assess whether results are not only statistically significant but also, where or not they vary significantly?" If yes, this approach might likely raise additional survey and experimental design issues. In addition to conducting 1-way ANOVAs which other test should be performed - for instance, assess the mean and standardization of the results?
D. Can you please provide references?
Thanks!
Tobi
Here is a review chapter that includes a discussion of question ordering effects.
- Badges
- Science topic