A comparison of the psychometric properties of three- and four-option multiple-choice questions in nursing assessments

Summary

In multiple-choice tests, four-option items are the standard in nursing education. There are few evidence-based reasons, however, for MCQs to have four or more options as studies have shown that three-option items perform equally as well and the additional options most often do not improve test reliability and validity. The aim of this study was to examine and compare the psychometric properties of four-option items with the same items rewritten as three-option items. Using item-analysis data to eliminate the distractor with the lowest response rate, we compared three- and four-option versions of 41 multiple-choice items administered to two student cohorts over two subsequent academic years. Removing the non-functioning distractor resulted in minimal changes in item difficulty and discrimination. Three-option items contained more functioning distractors despite having fewer distractors overall. Existing distractors became more discriminating when infrequently selected distractors were removed from items. Overall, three-option items perform equally as well as four-option items. Since three-option items require less time to develop and administer and additional options provide no psychometric advantage, teachers are encouraged to adopt three-option items as the standard on multiple-choice tests.

Introduction

In nursing education, multiple-choice questions (MCQs) are one of the most popular written assessment formats. Single best-answer MCQs consist of a question, two or more choices from which examinees must choose the correct option (the distractors), and one correct or best response. While MCQs are often criticized for largely assessing factual recall over higher cognitive thinking (Pamplett and Farnhill, 1995), MCQs still offer many advantages when compared with other types of written assessment. Despite what many teachers believe, MCQs are adaptable to different, although not all, levels of learning outcomes (Gronlund and Waugh, 2008). High quality MCQs present clinical vignettes to students that mimic actual clinical problems and assess application of knowledge rather than simple factual recall (Case and Swanson, 2003). Therefore, well constructed MCQs can accurately discriminate between high- and low-ability students (Schuwirth and van der Vleuten, 2003). MCQs are objective and they allow teachers to test a wider range of content and educational objectives than many other written assessment methods. Additionally, MCQs allow teachers to efficiently assess large numbers of candidates as they are easy to administer and score (McCoubrie, 2004). Furthermore, because of this broader sampling of content and because MCQ test items can be subjected to post-test review using item analysis procedures, MCQ tests have higher validity than other test methods such as short-answer or essay-style questions (Gronlund and Waugh, 2008).

Four-option MCQs remain the standard in nursing, both on in-house developed tests (Tarrant et al., 2006) and in test banks and text books used in nursing education (Masters et al., 2001). In other health-science disciplines, such as medicine, five-option items are more common (Haladyna and Downing, 1993). Although measurement specialists have long discovered that there are few evidence-based reasons for MCQs to have four or five options, many introductory books on item writing continue to recommend this practice and a majority of teachers continue to follow this recommendation (Owen and Froman, 1987). Three-options items however, have many advantages over four- and five-option items, including less time required to construct items and less testing time required. Alternately, with less time required to complete three-option items, teachers are able to increase the number of items administered on a test and thereby increase the amount of content tested (Haladyna and Downing, 1993). Furthermore, researchers have shown that in both teacher-generated (Tarrant et al., 2009) and professionally-developed (Haladyna and Downing, 1993) four- and five-option MCQs, students rarely select more than two or three of the options.

In most nursing programmes, the amount of content that requires assessment can be overwhelming. A substantial proportion of a teacher’s time is spent on developing written assessments and since a substantial proportion of those assessments will likely contain MCQs, it is important that teachers are basing those practices on the best available research evidence. Additionally, student numbers are generally increasing to meet workplace shortages, while at the same time the number of available teaching faculty is often getting smaller (Broome, 2009). Therefore, because of their efficiency and ability to assess different learning outcomes, MCQs are likely to continue to remain an important component of written assessment in many nursing programmes for the foreseeable future. Thus if the time required to develop multiple-choice tests can be reduced without reducing the reliability and validity of the assessment, this is an important consideration for nursing faculty.