Swipe om te navigeren naar een ander artikel
The online version of this article (doi:10.1007/s40037-015-0212-x) contains supplementary material, which is available to authorized users.
This study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education.
Multiple-choice questions with too high or too low difficulty (difficulty index < 0.4 or > 0.8) and insufficient discriminatory ability (point-biserial correlation < 0.2) on previous administration were identified. Items in Experimental Subgroup A underwent removal of item writing flaws along with enhancement of tested cognitive level (21 multiple-choice questions), while Experimental Subgroup B underwent replacement or removal of non-functioning distractors (11 multiple-choice questions). A control group of items (Group C) did not undergo any intervention (23 multiple-choice questions).
Post-intervention, the average number of functioning distractors (≥ 5 % selection frequency) per multiple-choice question increased from 0.67 to 0.81 in Subgroup A and from 0.91 to 1.09 in Subgroup B; a statistically significant increase in the number of multiple-choice questions with sufficient point-biserial correlation was also noted. No significant changes were noted in psychometric characteristics of the control group of items.
Correction of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions. This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions.
(DOCX 29 kb)40037_2015_212_MOESM1_ESM.docx
Mehrens WA, Lehmann IJ. Measurement and evaluation in education and psychology. Fort Worth: Holt, Rinehart and Winston; 1991.
Masters JC, Hulsmeyer BS, Pike ME, Leichty K, Miller MT, Verst AL. Assessment of multiple-choice questions in selected test banks accompanying textbooks used in nursing education. J Nurs Educ. 2001;40(1):25–32.
Jozefowicz RF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew H. The quality of in-house medical school examinations. Acad Med. 2002;77:156–61. CrossRef
Case SM, Swanson DB. Constructing written test questions for the basic and clinical sciences. Philadelphia: National Board of Medical Examiners; 2011. http://www.nbme.org/publications/item-writing-manual-download.html.
Downing SM. The effects of violating standard item-writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ Theory Pract. 2005;10:133–43. CrossRef
Tarrant M, Ware J. Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Med Educ. 2008;42:198–206. CrossRef
Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ. 2002;15(3):309–34. CrossRef
Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurs Educ Today. 26(8):662–71.
Newble D. A comparison of multiple-choice and free-response tests in examination of clinical competence. Med Educ. 1979;13:263–8. CrossRef
Maguire T, Shakun E, Harley C. Setting standards for multiple-choice items in clinical reasoning. Eval Health Prof. 1992;15:434–52. CrossRef
Elstein A. Beyond multiple-choice questions and essays: the need for a new way to assess clinical competence. Acad Med. 1993;68:244–9. CrossRef
Boshuizen H, Vleuten C van der, Schmidt H, Machiels-Bongaerts M. Measuring knowledge and clinical reasoning skills in a problem-based curriculum. Med Educ. 1997;31:115–21. CrossRef
Shakun E, Maguire T, Cook D. Strategy choices in multiple- choice items. Acad Med. 1994;69(10 suppl):S7–9. CrossRef
Rodriguez, MC. Three options are optimal for multiple-choice items: a meta analysis of 80 years of research. Educ Measure Issues Prac. 2005;24(2):3–13. CrossRef
Downing SM, Haladyna TM. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ. 2004;38:327–33. CrossRef
Tarrant M, Ware J. A comparison of the psychometric properties of three- and four-option multiple-choice questions in nursing assessments. Nurs Educ Today. 2010;30(6):539–43. CrossRef
Messick S. Standards of validity and the validity of standards in performance assessment. Educ Measure Issues Prac. 1995;14:5–8. CrossRef
Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830–7. CrossRef
Kern DE, Thomas PA, Hughes MT. Curriculum development for medical education: a six-step approach. 2nd ed. Baltimore: Johns Hopkins University Press; 2009.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65:563–7. CrossRef
De Champlain A. A primer on classical test theory and item response theory for assessment in medical education. Med Educ. 2010;44:109–17. CrossRef
Tavakol M, Dennick R. Post-examination analysis of objective tests. Med Teach. 2011;33:447–58. CrossRef
Oosterhof AC. Similarity of various item discrimination indices. J Edu Meas. 1976;13:145–50. CrossRef
Attali Y, Fraenkel T. The point-biserial as a discrimination index for distractors in multiple-choice items: deficiencies in usage and an alternative. J Edu Meas. 2000;37:77–86. CrossRef
- The Impact of item flaws, testing at low cognitive level, and low distractor functioning on multiple-choice question quality
Syed Haris Ali
Kenneth G. Ruit
- Bohn Stafleu van Loghum