In perceptual listening tests, subjects have to listen to sound examples and rate their sound quality or other characteristics.
As these tests can be quite long, a serious and practically relevant question is if participants change their rating behaviour over time, maybe because the prolonged concentration while listening and rating leads to fatigue, or subjects adapt to the stimuli in some way.
Do you know of any studies or publications that treat this question, whether subjects rate stimuli differently depending on whether they're presented at the beginning of the test or at the end?
Cheers!
Not exactly due to listening fatigue, but my recent taxonomic study of restaurant sound sources found a clear and quite intriguing (I think) order effect in evaluations of "liking" of sound sources. The discussion goes along the lines of an initial social desirability bias being countered by a task-induced change in listening mode (increased cognitive load) causing an effect over time (on the order of a couple of minutes). The manuscript is not yet accepted, hopefully coming soon :)
There are many aspects about ratings by listeners during the session. Listener's familiarity and/or interest about the audio samples played, their state of mind, active or passive listening mode and so on... These factors can also impact the ratings.
We have observed this during few experiments on listener's feedback on some Hindustani classical audio clips of duration about 3 minutes.
@Permagnus and @Makarand, thanks for your remarks! How did you analyse these effects and their significance? I would be interested in having a look at your data, and the related publications, if and when available.
In the 2 data sets I have that spawned this question, there were small but significant order effects according to a linear regression analysis (article awaiting acceptation).
@Andriy, thanks for sharing this resource, but all the keywords and search phrases that I can think of are not specific enough, so that there are millions(!) of matching articles. The same goes for web search. I think we need a more semantic-aware search system here, e.g. human brains. =-)
Listener's familiarity and/or interest about the audio samples played -can be taken as a feedback in the response sheet. their state of mind- very difficult to note but we can have it in our questionnaire before session, active or passive listening mode - their response as tapping for rhythm can be an indicator for active listening.
@Diemo, as per separate email - I spotted a suspected 'order effect' while doing other work on the data (!) Post-hoc testing with ANOVA and Tukey HSD showed significnat differences between evaluations at 3 successive time points. A plot indicated a clear trend, and I calculated effect size between first and last evaluations.
Hi Demo,
Try
The Impact of Test Duration on User Fatigue and Reliability of Subjective Quality Ratings
Authors: Schatz, Raimund; Egger, Sebastian; Masuch, Kathrin
Affiliation: Forschungszentrum Telekommunikation Wien (FTW), Vienna, Austria
Journal of the audio engineering society, Volume 60 Issue 1/2 pp. 63-73; January 2012
Dear Josh, thanks, that was exactly the kind of reference I was looking for and couldn't find (I would have never dared to search for "user" =-)
I'll post a link to the ICMC article that I just finished analysing such phenomena in 2 data sets as soon as it is published in September.
As promised, here is the article that I'll present Friday 16.9. at the ICMC in Utrecht that presents an effect of test duration on ratings in perceptual listening tests found in two different datasets.
Conference Paper Effects of Test Duration in Subjective Listening Tests
- Badges
- Science topic
- Similar topics
- Biological Science
- Molecular Biology
- Biomolecules
- Nucleic Acids
- DNA