I'd be interested to know of these myself - I can't think of one that examines that specifically offhand.  There are some papers using think aloud to look at other aspects, certainly.  Schuwirth et al (2001) looked at context-free vs case-based MCQs,   Vorstenbosch et al (2013) looked at altering response/answer formats (images vs text).

For cueing effects in MCQs, Schuwirth's 1996 paper is often cited, but it didn't use a think-aloud protocol. 

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.1996.tb00716.x/abstract

http://onlinelibrary.wiley.com/doi/10.1002/ase.1382/abstract

Article Do short cases elicit different thinking processes than fact...

I'd be interested to know of these myself - I can't think of one that examines that specifically offhand.  There are some papers using think aloud to look at other aspects, certainly.  Schuwirth et al (2001) looked at context-free vs case-based MCQs,   Vorstenbosch et al (2013) looked at altering response/answer formats (images vs text).

For cueing effects in MCQs, Schuwirth's 1996 paper is often cited, but it didn't use a think-aloud protocol. 

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.1996.tb00716.x/abstract

http://onlinelibrary.wiley.com/doi/10.1002/ase.1382/abstract

Article Do short cases elicit different thinking processes than fact...

It's hard to find anything specific. 

A few references, mostly tangentially relevant. Still waiting for a knight in shining armour to turn up on RG…

Beullens, J., Struyf, E., & Van Damme, B. (2005). Do extended matching multiple-choice questions measure clinical reasoning? Medical Education, 39(4), 410–417. doi:10.1111/j.1365-2929.2005.02089.x

Coderre, S. P., Harasym, P., Mandin, H., & Fick, G. (2004). The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts. BMC Medical Education, 4, 23. doi:10.1186/1472-6920-4-23

Drennan, J. (2003). Cognitive interviewing: verbal data in the design and pretesting of questionnaires. Journal of Advanced Nursing, 42(1), 57–63.

Durning, S. J., Artino, A. R., Jr., Beckman, T. J., Graner, J., van der Vleuten, C., Holmboe, E., & Schuwirth, L. (2013). Does the think-aloud protocol reflect thinking? Exploring functional neuroimaging differences with thinking (answering multiple choice questions) versus thinking aloud. Medical Teacher, 35(9), 720–726. doi:10.3109/0142159X.2013.801938

Knafl, K., Deatrick, J., Gallo, A., Holcombe, G., Bakitas, M., Dixon, J., & Grey, M. (2007). The analysis and interpretation of cognitive interviews for instrument development. Research in Nursing and Health, 30(2), 224–234. doi:10.1002/nur.20195

Paxton, M. (2000). A Linguistic Perspective on Multiple Choice Questioning. Assessment & Evaluation in Higher Education, 25(2), 109–119. doi:10.1080/713611429

Pottier, P., Hardouin, J.-B., Hodges, B. D., Pistorius, M.-A., Connault, J., Durant, C., et al. (2010). teaching with cognition: Exploring how students think: a new method combining think-aloud and concept mapping protocols. Medical Education, 44(9), 926–935. doi:10.1111/j.1365-2923.2010.03748.x

Dear Ronán, I have used some software to extract the keyphrases from your last list. Send these to Google Scholar: "medical education" "multiple choice" cognitive thinking

You can also substitute "think-aloud" for "thinking". 

It seems to me that you will have to write the paper yourself, as it seems that no paper with your envisaged title exists!

Ian: drat. I'm coming to the same conclusion.

since think aloud distracts the cognitive processing of problem solving activity,  I suggest you use retrospective review . 

A very complete description of the thinking aloud method can be found in Ginsburg, H. P., Kossan, N. E., Schwarts, R., & Swanson, D. (1983). Protocol methods in research on mathematical thinking. In H. P. Ginsburg (Ed.), The development of mathematical thinking (pp.7-47). New York, NY: Academic Press.

Ericsson and Simon (1980) argued that the thinking aloud method provides the best description of the cognitive processes. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87 (1), 215-251.  

I have just written a paper (under review- I'll post it here if it is accepted by the journal) on anxiety as a consequence of taking online MCQs.

If you are deploying the test in an online environment, it might well be worth considering the anxiety levels of your students, especially if they are new to using ICT and web-based platforms.  Also, look into the facilitation of the dominant response as this may have an effect on student performance.  Hope you find the solution you are looking for.

Gareth.

Please do post it, Gareth - it sounds interesting.

My own interest is in how people actually answer the questions.

Some questions will be uninformative: the person just knows the answer. I'm interested in questions involving reasoning. How do test takers sieve the answers? What heuristics do high scorers use that discriminate them from low scorers? Are good test takers just ones who can access large knowledge bases, or do they also differ in strategy?

There's no doubt that MCQ-taking is a skill that clusters in some students (I have data on that, and so has everyone else). But why?

Found one!  Just published in 2014, Heist et al used a think aloud protocol & thematic analysis in a small (n=13) postgraduate study.  Their conclusion was that "different patterns of clinical reasoning strategies and test-taking behaviors may be used by high and low performers during high-stakes clinical vignette MCQ examinations".  

http://www.jgme.org/doi/pdf/10.4300/JGME-D-14-00176.1

Jane – you're a little star, so you are.

@Jane: How did you find Heist, and we didn't. We want to know for next time, please.

@Ian - references lists on related papers!   I was reading Dory 2010, and noticed the Heemskerk 2008 reference on p57 (which I hadn't read before), so looked it up through google scholar to get the paper.  I then clicked on "cited by 16"...  Heist et al are one of 16 papers that have cited Heemskerk.

http://link.springer.com/article/10.1007/s10459-009-9169-z/fulltext.html

http://link.springer.com/article/10.1007/s10459-006-9057-8/fulltext.html

Hi Ronán, 

Just thought I'd answer from my own experience. I am using a think aloud protocol with collaborators to look at mathematical problem-solving in students from different backgrounds, to see are there variances in metacognitive skills. The data that have come from that study are fascinating (although analysis is still very much underway!). 

At the same time as I was completing that study a number of students who had repeatedly failed the calculations component of the pharmacy licence exam came to me to seek assistance with their study. This is a multiple choice exam. Having the equipment etc. to hand, I thought I would be useful to try and understand what they were (or weren't!) thinking as they were completing the calculations, the general advice to practice, complete sample papers etc. had not worked! The outcome was very interesting, and not what I might have expected. So, I think your question is very interesting!

Interestingly, I think that it is consistently identified that expert/novices (as in Ericsson's work already mentioned in an answer to this question, that would also cover some of the heuristics/biases information) in terms of think aloud, in addition to this some really interesting work in the maths education field by a mathematician Matthew Ingils at Loughborough University (e.g. http://homepages.lboro.ac.uk/~mamji/files/JRME_EyeMovements.pdf)  that looks at how experts/undegraduates actually read and look at problems, identifying that individuals appear to actually read the same problem differently. I hope to expand into using that plus think aloud to look further into mathematical problem solving again. That might also add a further dimension into the MCQ-reading/completion strategies.  

So based on my limited experience and some reading of maths literature, I think your study sounds like it will helpful to try and understand what test-takers are thinking. From my experience of using think aloud as a method, I think it will be apparent from the verbal reports what the students are doing in terms of using heuristics and other problem-solving skills/strategies, and it is also likely (but difficult to capture) that they actually 'see' the problem differently, especially if you are using vignette-based questions.

Really interesting conversation!

Michelle

Thank you for the discussion.  Just writing a masters assignment about a clinical reasoning teaching session I did with some students who had failed an MCQ.  As you  and many of these papers indicate there definately seems to be a pattern of using faulty heuristics to answer questions amongst struggling students.  I wonder whether clinical vignettes in MCQs actually promote system 1 thinking and if so do we want to encourage this in our students?

Perhaps with the amount of knowledge needed in medicine we actually do need to encourage heuristic use.  However if this is the case then we should encourage students to be mindful of this so they don't get into tricky situations in the future.

It is an interesting topic.  Thanks for the usful references for the assignment - due in tomorrow so will need to not get too distracted.

Geraldine:I think that you have an important point there. If you think of arithmetic, solving problems entails both bringing up the right knowledge (your tables) but also in selecting appropriate heuristics to apply these. 

What did you find when you worked with the students who had failed the MCQ? And have you any literature that we've missed so far??

Yes I often found students answered incorrectly because they took one piece of information such as weight loss and made a connection to particular diagnosis such as malignancy.  This would occur even if there was clearly other reasons for weight loss such as HIV or thyrotoxicosis.  We all use such heuristics, perhaps what distinguishes the good students (and doctors) is just knowing when to turn them on and off. 

Unfortunately no more literature.  It wasn't the main focus of this essay but I do have one on assessment coming up so I'll let you know if I find anything then.