by Jan Kuks
Many neurologists are most enthusiastic about educating colleagues, medical students, paramedical staff etc. This may be because education and explanation are crucial for good patient-care. Education goes closely together with testing. Exams should be fair, reliable, valid and objective. To meet the last criterion closed questions, as there are multiple choice questions, single best answer questions, true-false-questions are used. To meet the other criteria, closed questions should be well-constructed.
Even after several decades of experience it remains a challenge to produce sound examination questions. Several guidelines exist for this purpose. As a teacher and continuous medical educator, producing some hundreds of questions a year, and reviewing a couple of thousands, I made my own – still incomplete – list, derived from other guidelines and good advices from colleagues. I hope this short list may be helpful for the reader who is faced with the task of producing closed questions, for example for the CME process as currently in the EFNS and future EAN.
Terminology
A question with a couple of answers is referred to as an Item. The question may be introduced by a scenario (e.g. with a clinical vignette), which may be named as ‘Stem’. The question itself is also referred to by some authors as ‘Stem’, others name it ‘Lead-in’. The possible answers are called ‘Options’.
Closed questions are questions with a couple of answers with one correct answer and other incorrect answers (distractors or distracters). There are true/false questions with only 2 options, Single best Answer questions with 2 or more options, Extended Matching Items with pairs of options to be combined in the right way.
General recommendations
1. Make a blue print containing numbers of questions to be assigned to the various objects.
2. Let a question be valid as to relation with the learning contents. Check whether a question is really relevant to be asked to a student of a certain level.
3. Never write trick questions deliberately misleading a student.
4. Avoid the use of abbreviations; be most precise in using capitals and punctuation.
5. Always ask a second opinion on your MCQ. It is (nearly) impossible to write a perfect MCQ without intervision.
Scenario (Stem)
6. A Scenario exists only to get the student into the right context of the question. Putting information in a scenario may enable reduction of text in the question and the options. Abbreviations can be introduced and explained in the scenario. Using clinical vignettes may enhance testing clinical reasoning and higher order thinking. A scenario is useful to make a focused (which of the following statements applies for a situation as described?) instead of an unfocused question (which of the following statements is true?)
7. Keep the scenario simple and ensure that the scenario is meaningful. A clear problem should be presented. It often occurs that a question can be asked without necessity to read the stem anyway. Realize that a medical exam is not intended for testing reading abilities.
8. It is discouraged to use denials in the scenario to avoid confusion of the reader.
9. The scenario should only contain information that is true or to be accepted as being true. The question should be clearly separated from the stem, e.g. by using another font type.
10. One scenario can be used for more than one question but beware of dependency.
Question (Lead-in)
11. Try to ask a complete question which can be answered without reading the options instead of writing half a statement that should be combined with one of the options.
12. Negative words in questions should be avoided, but this may not always be possible in cases where true answers are abundant and false are exceptional (‘which of the following can NOT be used for ….’). In these cases negative words should be underlined or written in capitals.
13. Beware of double denials in the question and the various options.
14. Be sure the connection between stem and each option is grammatically correct. Do read the question with each option aloud to check whether they fit with each other.
15. Avoid inaccurate terms like mostly, generally, frequently, rarely, often etc. They are too qualitatively. Define terms as ‘chronic’, ‘long-lasting’ etc.
16. It is recommended to write MCQs in an comparative or superlative form (the best option is ….., A) is better than B).
17. If the student is requested to indicate the best answer it may be that there is an even better possible answer not mentioned in the list provided. Therefore always ask for the best option ‘… of the following ….’.
18. Avoid terms like ‘always’ and ‘never’. Statements with these qualifications seldomly can be held true in our discipline.
19. Be cautious with statements containing the word ‘can’ or ‘is possible’. Such statements are mostly correct.
20. Only ask one single question that you really want to ask. Avoid phrases like “the patient should be treated with 500mg paracetamol and 25mg phenergan”. One of both may be correct while the other is not.
21. Normative questions: “A patient should be treated with betablockers”. Who decides whether this ‘should be’? Mention a reference source: “According to the WHO guideline ……”
22. Never ask what a person should do: “Would the GP decide to prescribe aspirin?” in case the question really is: “is it opportune to prescribe aspirin?”. The student cannot know what the GP is doing and each answer is defendable because we do not know whether the actor is adequate.
23. If a text is repeated in each option, this text could be added to the lead-in to reduce the text of the separate options. (“Which of the following is the best description of perioperative complications? A perioperative complication is …”).
24. Avoid the use of the same (or a related) word in the correct option and the question (e.g. which of the following agents is most likely to cause a community acquired pneumonia? a. Klebsiella, b. Pseudomonas, c. Pneumococcus, d. Escherichia).
25. Avoid Convergence, i.e. repeating part of the correct answer in the distractors (e.g. which colours are shown in a traffic light? A. Red, green, blue B. Red, green, yellow C. Purple, red, green D. Yellow, green, orange. As blue, purple and orange only are used once B apparently must be the right answer.
Options
26. One and only one option should be unequivocally correct. Start with writing this correct option before constructing the distractors.
27. Distractors should be plausible. The more options are included, the more difficult it will be to find plausible distractors. There is evidence for an optimum of not more than two distractors in an MCQ. There is no need to keep the number of options constant in a test, so mixing up true/false with 3- or 4-options etc. within a test is definitely possible.
28. Try to find options that are equivalent. E.g. avoid 4 options with 2 diagnostic proposals and 2 therapeutic ones. Try to describe the options with an equal length. The longest options often are true.
29. Try to mix up options that are mutually exclusive with options that are not (e.g. hypothyroidism and hyperthyroidism together with diabetes and hypercortisolism).
30. Avoid constructions with ‘all options above are incorrect’. If this is the right option you really can’t know whether the student is aware of the right answer. With other words: always include a (most) correct answer in a list of option.
31. Avoid constructions with two statements to be validated by a. 1=true, 2=true, b.1=true, 2=not true, c. 1=not true, 2=true, d. 1=not true, 2=not true. This may lead to unwanted errors.
32. Do not ask for concrete numbers like a. 35%, b. 55%, c. 75% etc. Such a figure is nearly always incorrect. Rather use intervals that do not overlap: a. 30-39%, b. 40-49%, c. 50-59%.
33. Order the options in an alphabetical or a logical way (not: a. mesencephalon, b. spinal cord, c. pons, d. medulla oblongata but rather a. mesencephalon, b. pons, c. medulla oblongata, d. spinal cord).
Prof Jan Kuks is chair of the EFNS sub-committee on CME and Professor of Neurology at the Department of Clinical Neurology and the Department of Medical Education, at the University Medical Centre Groningen, The Netherlands
