Item Sampling, Guessing, Partial Information and Decision-Making in Achievement Testing

Abstract

This paper addresses the problem of determining the number of items in a test and the cutting score to make a binary decision on mastery with prescribed precision. Previous attempts to solve this problem have been based on simplistic models assuming all-or-none knowledge. The same approach as in those papers is adopted here, but from the standpoint of a finite state model of test behaviour. The required number of items is shown to be smaller than previous results suggested. In contrast with previous attempts, this method is applied to true-false and multiple-choice tests responded to in the conventional mode, and to multiple-choice tests responded to in answer-until-correct and Coombs’ modes. The effects of varying guessing behaviour on the part of the examinees are also studied. It is shown that the smallest number of items needed to make a mastery decision with a predetermined error rate occurs when the testees respond without guessing. This behaviour is unlikely to be followed by examinees, but it is also shown that this lower limit can be reached at by administering multiple-choice tests under Coombs’ response mode.