Dear Sabine.

At first, you are using a not very- well-defined concept: 'people who weren't thinking in the task'. You can't know what participants are thinking about. Your problem is called 'outliers'. A value of accuracy or reaction time, also the response pattern, maybe is biasing results. To face outliers there isn't a unique rule. Is a decision. Relevant is you share/ publish your decision, reviewers will give you feedback. One rule is to normalise data, calculate z punctuations. Look the skewness and kurtosis of the adjustment, only if you consider your variable distribute X-axis cumulative Y-axis values in a gaussian distribution histogram. Once you reach this step, calculate the percentage of X-cases in +/1 z=1,63. Is a decision to delete that cases (outliers), not a straight rule related to thresholds. Walkways consider the Soli and Ceil Effects. It can be plotted using an F curve. Never forget that all cases, included outliers, are information. Best!

This is a tricky situation. As you say, reaction times much under 200ms are more or less impossible for the human nervous system, so instances of responding in 126ms are suspect, especially when it is a 6-option choice reaction time task. But you don't want to be arbitrary, and you certainly don't want to exclude 'outliers' whose responses could potentially be valid. I can offer just a couple of thoughts here; I hope someone with greater expertise in this sort of experiment can help more.

1. Presumably you measure errors as well as reaction times. People who are not engaging with the task (just pushing any button as fast as they can) will make many errors. Even if they only do the random button-push thing a quarter of the time, they'll hit the wrong button 5 out of 6 of those trials. So a simple step is to eliminate all data for anybody with what you deem to be an unacceptable number of errors. (It would also be interesting to study the characteristics of the people who do so, but that's another study!)

2. Many people will make an occasional impulsive jab at a button, so you may get a few ultra-fast reactions even from people who are taking the task seriously. But most of these reaction time experiments include lots and lots of trials, so you could just leave them in. They add noise to the data, sure, but no measurement is perfect. (I work more with psychological tests than reaction times; in my area, we study reliability so as to understand how much 'noise' there is. But running hundreds of trials gets past reliability issues by brute force, so to speak.)

3. A few of the super-fast reactions may even be valid! If a person happens to have their finger poised over the correct button and be looking at the exact spot where the stimulus appears, they 'got lucky' that time. Won't happen often, but again, a few outlier trials won't affect the overall pattern of results much at all.

Dears Sabine and Stephen,

that is a very interesting topic! I agree with Stephen that a task design that includes six options increase in an exponential mode what Stephen (I interpret) is saying: Engaging with the task, the previous mapping between executive functions and instructions, working memory, random movements are included. The main guideline is the process you are measuring, and the conceptual model that builds the bridge between the process and the task. As a beautiful example, in the Attentional Network Task, Dr Michael Posner only included the Accuracy= YES responses. The Reaction Time measure, of course, combined with Accuracy= YES, was obtained averaging RT. But, and now we are opening the gates of the Castle, Michael Posner did a previous pilot study to know the RT and ACC relation ( the functional model), the RT global distribution (ALL RT responses), the RT ACC= YES responses distribution. He excluded participants performed

My guess would be that using cut-offs is okay. You might lose some genuine responses, but will mostly exclude guesses.

https://link.springer.com/content/pdf/10.1007/BF03395630.pdf

Thank you so much for your replies and your thoughts on the topic. Maybe to clarify: the aim is not a detailed examination of reaction times, but rather to use the reaction time measure to detect arbitrary key presses among healthy participants who still took the task seriously. However, we also don’t want to exclude fast performers. We have already tried out nearly all the things you suggested. As Julian so beautifully wrote “the task is your guide to find an answer”. However, our task does not yet speak to us :-) In most cases we would exclude more answers than we feel are really necessary to exclude.

So far, our favorite option is Stephen’s suggestion #1. However, we are not satisfied yet. We still think, that having a cutoff like the Hick-Hyman law would suit our needs best. Does anyone have any experience with it? The formula is RT = A + log2(n)*B; A = time needed to press the button, n = number of possible answers (in our case 6) and B = time needed to process the information (or in other words the time needed to make a decision). We have calculated the following: The intercept of the regression is 230.321, from literature we assume A to be 200ms, thus the formula would be RT = 200 + log2(6)*230.321 = 795.37. Does this seem correct? Is it right to take the intercept? From literature we are not sure if we have to take the slope of the regression line instead of the intercept. However, the resulting cutoff would not make any sense for our data. The number of RTs that fall below the cutoff calculated above using the intercept seems reasonable.

Thanks again for your valuable help!

Dear Sabine,

Thanks to appreciating the metaphor I used. To say the same without metaphoric language, an RT who correspond to an arbitrary key press depends on your task. You need to have an expected range of RT, in order of your task. I can give you a thousand examples, I work with Alan Baddeley here in Psychology of York, UK. It is so easy to detect, that the preferred answer you choose looks as complex as a student like, but as unnecessary as experience teaching. Then, you can calculate all your logarithms, but you can see the 'Neuropsychologia' journal new edition. We have published (I am the first author, Elizabeth Jefferies did the supervision) an article using RT's like a dependent measure. If you send an article using the preferred answer, as a probable reviewer, I can easily do what's Latin's call 'reductio an absurdum' what means: If I continue the reasoning, to apply that is enough to get a direct rejection. I understand that there are steps in a nice career where 'increasing mathematical operations is equal to increase how to look at the article'. To remove RT's coming for arbitraries key presses is so easy, but need to apply some proceedings,

Best, Julián

Wow, you're getting me involved in a challenging problem right when I need to move all my classes online! But okay, it's fun, so:

I was not familiar with the Hicks-Hyman Law, though obviously it has roots all the way back to Donders. Reviewing it quickly, I have two possible corrections to offer. First, "time needed to press the button" would NOT include any processing time, just the actual motor action - which would be much less than 200ms (which includes basic cognitive processing, as in "did I see anything just now?"). Second, the formula I find online reads (n+1) rather than n (it also doesn't always include the "a" term). So anyway, log(2) of 6 = 2.585 and log(2) of 7 = 2.807, which aren't that different. I'm not sure how best to incorporate the "intercept" into this, but maybe as you say it represents the necessary processing time? In which case you'd have 230.321 times 2.807 = 646.511ms or possibly 230.321 times 2.585 = 595.380.

The trouble here, I think, is that the regression equation you calculated must be based on the sample as a whole - but not everybody has the same processing speed. So you would expect a distribution of values for the intercept based on each individual's speed. Which kind of brings you back to the problem of people just jabbing away without processing! I think the best solution here would be to use a simple reaction time test as a control (again, kind of going back to Donders!). Assuming your people don't make lots of errors when it's just "hit the key when you see the stimulus," you could get a baseline speed measure for each.

Wow, you sure lead a complicated life!

Dear all,

I am part of Sabines research group and I would like to thank you for your replies and suggestions and also add a few comments.

As both of you have mentioned in different ways, the best would have been to determine a task specific expected range of RTs, which could have been obtained with a baseline measure of simple reaction time as Stephen suggested. However, unfortunately we did not take such a baseline measure, which is certainly a limitation of the study and good to keep in mind for future studies.

I have looked into the Hick’s law again and I might be wrong, but I understand that the Hick-Hyman equation only works when there is a changing number of response alternatives (e.g. 4 figures, 5 figures, 6 figures to choose from). But we only have a constant number of 6 figures to choose from, which change their number of features (5,6,7). I think the Hick-Hyman equation reflects the change in RT depending on the increase of binary decisions depending due to increasing response alternatives. The model then predicts that RT should be linearly related to the number of bits of information (i.e. zero bits for simple RT, 1 bit for 2-choice RT, 2 bits for 4-choice RT as it doubles the possibilities and so on).

So, we can only use the intercept and slope as a straight-line function of RT with the number of bits (choices) as the independent variable. But in our study, there are always 6 figures, with either 5,6 or 7 features and using the number of features as the response alternatives (6 figures x 5 features = 30) seems odd, as I doubt that binary decisions are being made among features themselves. Do you have any thoughts on this?

Seeing that we: 1) don’t have a baseline simple RT measure 2) are not convinced about applying any kind of rule that is dependent on the RT data itself to find a cutoff (2SD, 2.5percentile ect.) as we are then excluding data we’re actually interested in (correct, incorrect, missing) and instead would like to use RTs only to detect arbitrary key presses among healthy participants who still took the task seriously,

I wonder whether we could use a conservative cut-off based on a previous study of average human reaction time to six visual stimuli. But so far I have not been successful in my research.

Have you come across any other studies or formulas on how long humans need to visually look at 6 stimuli and react on average?

Thank you very much for your help and suggestions.

Best,

Marlen

Here is a link to a chapter in Matthews et al. (2000) on the Hick Hyman equation that made me think about its applicability :

https://books.google.ch/books?id=1XK3AQAAQBAJ&pg=PA48&lpg=PA48&dq=how+to+find+the+intercept+of+the+hick+function&source=bl&ots=DrZ4LxcqxZ&sig=ACfU3U1yjGRjtcT0rjixp1PqPktdPRMBIg&hl=de&sa=X&ved=2ahUKEwjIlJn23qHoAhWszMQBHaSwArkQ6AEwEHoECA0QAQ#v=onepage&q=how%20to%20find%20the%20intercept%20of%20the%20hick%20function&f=false