Hello there, I am conducting listening experiments for the perception of various of noises. The test persons can listen to a reference noise and compare it with 20 other noises using a unipolar likert scale (more annoying (-5) - similarly annoying (0) - less annoying (+5), in total of 11 scales). For the analysis I used box plots in order to find the outliers. The persons who had the highest numbers of outliers where removed from the analysis, raising the coefficient of determination from 0,9692 to 0,9734, which is a good sign. How can I furthermore decide what data are to be removed in order to improve my model? I also checked if the test persons used the whole range of the scales or how long they needed for the test. How can I check if those criteria matter on the consistensy of their answer? Should I remove all the answers of a test person which is overall suspicious or can I remove just the supsicious single ones and replace them?
Thank you in advance, and I am glad to hear your answers!
You should always decide on which method you're going to use to detect and/or remove outliers prior to analysis and there should always be theoretical justification for removing participants/data. Inconsistent responses is not sufficient to remove participant data unless you included some kind of manipulation check in the experiment that a participant failed.
On another note, the coefficient of determination seems remarkably high. I wonder if the model is overfit to the data?
Hello Ifigenia,
There's no assurance that people who elect extreme option responses are wrong, biased, malicious, or inattentive to the task: only that they appear--relative to others in your sample--to be different. As well, the tendency to avoid extreme responses is a bias that has been long recognized in both survey and scaling literature.
I agree with Blaine Tomkins that criteria for declaring a case as unsuitable for inclusion in analysis should have been declared a priori. Doing so and subsequently checking to see that model fit is improved sounds like "cherry picking," and in all likelihood would yield an overly optimistic estimate of model-data fit or S-N ratio (in other words, overfitting).
Good luck with your work.
Thank you very much for both answers! David Morse Blaine Tomkins
I forgot to mention that the rsquare of 0,9734 corresponds to the normalized mean one (using mean evaluation of each sound as input). The normalized rsquare was found to be 0,7107.
First, to repeat what Blaine Tomkins said, because it is very important: Do never remove outliers to "improve" your model! This can be considered fraud.
If you must identify outliers, never use boxplots! They only work for symmetric distribution. Rating scales normally produce asymmetric distribution (such as all other measures, strictly).
Your outliers could be extreme responses of extreme responders. If you have repeated measures, you can estimate a multi-level. The participant-level intercepts correct for response style (to some extent). The residuals of the model can be used for outlier detection (with care!).
Something else you can do, is estimate the same model with and without outliers, to determine how much influence these points have. That's called influential points or leverage analysis and is fully legit if you report it. It is a way to show that the overall trend is not just an artifact, created by a few extreme observations.
Martin Schmettow Thanks for taking the time to reply!
I didn't quite understand the last paragraph ''If you have.. repsonse style.''. Could you clarify that for me?
I describe this in my book:
https://schmettow.github.io/New_Stats/glm.html#rating-scales
- Badges
- Science topic