You have asked a very good question.i would not follow that advice attached are my suggestions for outliers.. Best wishes, David Booth

Hello Jenny,

Often, when online survey platforms are used, the researcher has the option of looking at elapsed time for the total survey, as well as sometimes for individual pages/portions of the survey. The general concern when considering elapsed time is whether it might signal a less-than-forthright response style.

Why think about jettisoning cases in which the elapsed time was too short? The concern is that the respondent didn't fully read the items/questions/stimuli and therefore couldn't possibly be giving useful infomation in their answers.

Why jettison cases in which elapsed time was too long? There are several possible reasons, one being the respondent simply left the application, or was distracted by other events/stimuli, and so on. This one is obviously a harder one to pin down, so normative approaches are sometimes used (e.g., when response time is so high as to be in the upper 1% of cases), and can obviously be fallible.

Here's a link that looks at the issue in a different setting, online testing. Some of the observations may be helpful: https://largescaleassessmentsineducation.springeropen.com/articles/10.1186/s40536-021-00100-w

Good luck with your work.

@David Morse why not jettison patient data because it doesn't fit with the researcher's preconceived notions about what the results should look like. Eg. The patient just didn't get better fast enough so let's jettison the observation. This appears to me to be a very slippery slope to walk upon. Researcher's don't get to pick and choose among their data. At least I suggested conditions for setting aside for FURTHER study rather than throwing it away just because I didn't like it. But then why believe most online surveys anyway. David Booth

Hello Professor Booth,

I'm not advocating for the proposition; I was trying to explain some of the usual justfications for applying time criteria to response acceptability. However, in the case of too-quick a response time, at least one can empirically determine a distribution of expected response times, given that the respondent fully engaged in the desired task. This can inform whether elapsed times are so short that the associated responses could justifiably be viewed skeptically. Response patterns are another sentinel that can sometimes be helpful, as is the inclusion of so-called validity or "lie" items (such as, "My birthday is on February 30").

Aside from poorly constructed measures, the largest threats to on-line surveys are: (a) the typical convenience rather than probability sample; (b) the typical low response rate; which leads to (c) the typically atypical sample that emerges due to the high degree of self-selection for choosing to participate. The literature is replete with examples of how volunteers (for research participation) differ in numerous important ways from non-volunteers.

I do not and have not endorsed cherry-picking of data. However, inspecting data for what Wilcoxon called "outright liars" is certainly fair game.

Best regards to all on this thread.

thanks for all your answers and references. I know there are several criteria we can use in combination to improve the quality of data collected from online questionnaires. on the other hand, I am also concerned about the subjectiveness involved in such a selection, which might consequently influence our results. Hence, only when quite confident that some data are suspicious (say, 42 seconds for a scale with 30 items), I will not delete data collected.