Advanced Psychometric Methods for Developing and Evaluating Cut-Point-Based Indicators

Abstract

In this study, we offer some advice on the use of empirical techniques to create categorical indicators from responses to multiple survey questions that ostensibly compose scales. We discuss the use of advanced psychometric methods, including confirmatory factor analysis for ordered categorical measures (CFA-OCM) and item response theory (IRT), to expand 12 previously published criteria for cut-point development and evaluation offered by Moore et al. (Indicators of Child Well-Being: The Promise for Positive Youth Development. Annals of the American Academy of Political and Social Science. Special Issue: Positive Development: Realizing the Potential of Youth. 591:125–145, 2004). We present an application of this advice by developing categorical cut-points for the social skills and behavior problems subscales of the U.S. National Survey of Children’s Health (NSCH) Social Competence Scale. We used data from the 2007 NSCH, a large cross-sectional, random-digit-dial telephone survey of a representative sample of the noninstitutionalized population of U.S. children 0–17 years of age. Parents of children aged 6 to 17 years (n = 63,364) responded to 4 questions about their children’s behavior problems and 4 questions about their children’s positive social skills. IRT analyses indicated that IRT model-based scores of +1.8 and −0.7 provided suitable high and low cut-points, respectively, for the behavior problems subscale. These cut-points had marginal reliabilities of 0.70 and 0.67 respectively, and corresponded to raw subscale scores of 13 and 8 IRT analyses indicated that an IRT model-based score of −1.8 provided a suitable cut-point for identifying children with low levels of positive social skills. This cut-point had a marginal reliability of 0.63 and corresponded to a raw subscale score of 13. For both subscales and cut-point sets, frequency distributions demonstrated that the cut-points identified sufficient proportions of children generally as well as within demographic subgroups. Our findings indicate that analysts can use raw subscale scores and cut-points of 13 and 8 to identify children with high, moderate, and low levels of behavior problems. Our results demonstrate that, for the positive social skills subscale, the subscale’s psychometric properties only support a single cut-point of 13 that identifies children with low levels of positive social skills. Our study highlights the iterative, empirically-based process that cut-point development should follow and shows the value of modern test theory-based psychometric methods to inform this process.