And if in doubt….. guess ‘D’

July 13, 2013 in Talking point

By now, hopefully you’re enjoying a well-deserved summer break (at least those in the Northern Hemisphere…..) In the summer spirit, here’s an interesting question that we were asked this week.

This week, we gave a virtual workshop on PeerWise, as part of the Western Conference on Science Education, held in London, Ontario. (Slides here, if you’re interested) One of the participants asked a seemingly innocent question that got us thinking.

What is the most common choice of correct answer chosen by authors of an MCQ?

Whilst not knowing the answer there and then, we realized that we were sitting on a goldmine of data! The PeerWise system now contains over 600,000 student-authored questions. Granted, not all of these are 5 item MCQs, but a substantial fraction are. Could we mine this data to see if there really was a preferred answer choice and, if so, which option was it?

It turns out that there are nearly a quarter of a million 5-item ‘active’ MCQs in the PeerWise system, and that the most commonly chosen answer by authors of questions is ‘D’. The percentage of ‘D’ correct answers (24.98%) may not look all that much more than the 20% that might be expected if the choice of answers was totally random, but the sheer number of questions analyzed here (223,435) makes this a highly significant result (in the sense of ‘statistical significance’, possibly less so in the sense of ‘change-the-world-significant’…..)

It’s interesting to note that the extremities of the answer range are both below the ‘random choice’ value of 20%. There’s a certain logic in thinking that authors may want to conceal the right answer somewhere other than the last, or perhaps even more so, the first answer choice.

Is this just a peculiarity of student-generated questions, and what about questions with fewer than 5 answer choices, you might be thinking? The collective wisdom of the internet is not a great deal of help here. Various sites (Yahoo answers included) include commentary that lacks a definitive answer, but is not short of ‘definite’ proclamations that it is answer B. Or C. Or, if you’re not sure, pick the longest answer. Clearly, some people would be better served by adopting a different strategy when preparing to answer MCQs, such as actually learning the material. Learning solutions magazine claims the most common answer on a 4-item question is C or D. When I got down the Google search page to a link to ‘Wild guessing strategies on tests’, I stopped reading. Feel free to supplement this with your own research….

(Just for the record, from our analysis of another 220,000 4-item MCQs, the most popular answer chosen by authors is C, by a nose from B, both of which are well above the 25% expected value if truly random.)

4 responses to And if in doubt….. guess ‘D’

  1. The if in doubt pick C does not surprise me as I often hear students say this is their strategy when they have “no idea”. I know when I write or QA multiple choice exams this is something I consider and ensure that there is a reasonably even spread across the exam. An interesting research project I think!

  2. When I create my own multiple choice questions for exams, I try to have more A answers. It’s indeed quite rare for me to have A’s, especially when I try to arrange the answers in either numerical or alphabetical order. I wonder what I can do to include/create more A answers.

  3. When we create exams, we almost (but not exactly) balance the letter grade distribution – probably with a few less E’s. It would be interesting to see if common D responses in PeerWise would influence students on tests. Would students be more likely to guess “D” after using the database as a study tool?

  4. We always put our answers in alphabetical or ascending numerical order for exams and I instruct the students to do this when authoring questions. Not sure how many do though…

Leave a reply

You must be logged in to post a comment.