I am reviewing a research work done by a commercial group. I have a few questions in interpreting the speech perception data. They conducted speech sound (e.g. Chinese tones) discrimination and identification test to investigate their predictability for learner's FL listening and speaking performance. Learner level was novice (e.g. first a few week of the FL instruction).
Question #1: The accuracy of the performance is closer to a chance (e.g. 50% for discrimination - same or different). Can we still use this data to answer any research question?)
Question #2: The discrimination test result and the identification test result are patterned differently in relation to the listening and speaking performance. For instance, only discrimination result demonstrated the meaningful degree of predictability toward listening and speaking performance while identification test didn't. What will be the possible explanations?
Question #3: In the discrimination test, the accuracy, which was in the level of chance, showed significant relation to the speaking/listening test but the reaction time didn't demonstrate the same relation to those language test. What are the possible explanations?
What should be the fair conclusion when only one of the four measures shows the statistical significancy?
Any comments will be appreciated.
Question 1: If your research question is to what degree you can use discrimination performance as a predictor for speaking/listening performance, then it seems like you might not be able to. If subjects are at chance level, this can't act as a predictor. However, individuals might vary here and for certain individuals, it might be predictive. How much inter-subject variability was observed?
Question 2: Identification typically requires long-term memory representations (though, this can depend on the task). How were naive listeners/speakers trained in the Chinese tone categories prior to the experiment? By contrast, discrimination relies on both psycho-acoustic and phonetic cues. It's perfectly possible for a person to be able to distinguish two tones very well but not know the categories.
Question 3: If the discrimination results were at chance level and you are saying that they showed a significant relation to the speaking/listening test, then one can only conclude that subjects did extremely poorly in speaking/listening. It's not clear to me that you can conclude anything from this.
As far as your last question "What should be the fair conclusion when only one of the four measures shows the statistical significancy?", I am a bit confused. You only mentioned three possible measures (identification accuracy, discrimination accuracy, and RT for discrimination). What was the statistical test? What was the relation that you examined?
TO: C T DiCanio, Thank you so much for your answers. I will look into the inter-subject variability right away. The subjects are a student who just started their lesson in Chinese program and they have no previous lesson before. The goal of the research was to see if students' aptitude in tone can be measured or predicted by tone discrimination or identification test. So the researcher intended to avoid the classroom lesson effect by testing students at the earliest possible stage. By "four measures", I meant accuracy and reaction time of discrimination test and identification test. Hmmm, where should I look to find anything that I can help Chinese teachers or learners (American)? I was thinking about looking at the test results by different pair or tone. Appreciate any thought you might have.
How many items were included in each tasks ? also how many participants could you have? Could you have piloting before the real conduct of the test? after piloting , things migh have been clear on whether you can follow the tests or skip some of the items..i also think that items were pretty easy to discriminate for the learners...
I could annswer your questions in general as i am also dealing withe perception in my dissertation.. ın brief, what i could suggest is to focus on your items in each tasks and the effect of additional language classes as well as individual differences...