I would like to find one measure for average response time while accounting for accuracy. The test consisted of reading sentences (reading time) which the subjects were then asked to interpret by choosing one of three different interpretations. The time it takes to select one interpretation is considered to be a measure of reaction time and accuracy is whether the subject selected the interpretation which is the most adapted to the sentence they just read.
In order to do correlational studies, I would like to find a way to combine these two variables into one score. The inverse efficiency score is not a good option as accuracy is quite good.
If it is at all possible, I would also like to know if there is a way to combine RT and the type of response? Where the correct answer is +1, and the two others gets -1 and -2?
Thank you for your understanding!
Good and important question. You already mention the inverse efficiency score (Townsend & Ashby 1978). It is a heuristic and thus lives or dies with the usefulness. And here is a problem, it never really caught on.
Have a look at https://www.psychologicabelgica.com/articles/abstract/10.5334/pb-51-1-5/ They also give a good discussion of focussing on one measure, when the other points in the same direction, excluding a speed accuracy tradeoff.
You may also use an adaptive procedure to match accuracy across subjects or conditions, when possible.
Sometimes it is possible to fit models that make predictions on RT and ACC. But this is typically the case when you have good baseline studies available and try parametric variations.
I guess this is all not fully satisfactory. Specifically, when you come across a nice method, let me know, please. Best, Peter
Hello Catrine Bang Nilsen,
The following links can hopefully provide you with the information you are looking for.
Best of luck,
R. Biria
https://www.mdpi.com/2079-3200/4/4/14/pdf
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.462.6501&rep=rep1...pdf
https://www.sfs.uni-tuebingen.de/~hbaayen/publications/BaayenMilin2010.pdf
https://star.psy.ohio-state.edu/html/wp/pdf/PNAS11.pdf
Thank you so much, Peter and Reza! I hope to find a good method for my study. Have a great day!
With computer based testing, using response times/latencies with accuracy has become a big issue. Much of the cognitive literature involves short RTs with the assumption that single cognitive processes are in operation. With multiple choice tests the RTs are much longer and the assumptions (and models) don't fit as well. So while models like Ratcliff's is great for describing RTs and accuracy in some situations (with very similar items that are "sequential-ish"), your task sounds different. The attached is what I looked at for ACT test data (and it uses a VERY simple model). I think the first step will be understanding the relationships between RT and accuracy for your task, and then thinking about an optimal metric.
Article Treating All Rapid Responses as Errors (TARRE) Improves Esti...
This is what I would do:
Assign value 1 to correct answer and 0 to wrong answer (seems like you only have right and wrong choices). Then, normalize the RT so it also fits the scale of 0 to 1. Add these two together for each question and now you have something that ranges from 0 to 2. You can average these scores and correlate them with other measurements.
Although, you can argue that this loses information, or that it is inaccurate representation. But you probably won't find much better solutions.
Also, do you have baseline measures? I would suggest to substract baseline RT from your scores for each participant, so that you are not dealing with individual drawbacks.
I like this straighforward solution idea, Deividas Stankevicius, but wouldn't you need to reverse one of them along the way, since "correct" would be adding to the score and fast would be ...sort of subtracting, if you get my meaning there.
(I can see its a 2 year old thread but as I am trying to figure out a similar issue myself, I thought i'd chime in- anybody still out there?).
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics