Elsevier

Journal of Clinical Epidemiology

Volume 124, August 2020, Pages 118-125
Journal of Clinical Epidemiology

Original Article
Unlike ROC analysis, a new IRT method identified clinical thresholds unbiased by disease prevalence

https://doi.org/10.1016/j.jclinepi.2020.05.008Get rights and content
Under a Creative Commons license
open access

Highlights

  • Cutoff points based on receiver operating characteristic (ROC) analysis may vary with disease prevalence.

  • ROC cutoff points do not necessarily reflect clinical thresholds.

  • A new item response theory (IRT)-based method identified thresholds not biased by prevalence.

  • The IRT method was evaluated using 3,000 simulated data sets.

  • The IRT method was applied to the Hospital Anxiety Depression Scale in a real data example.

Abstract

Objective

This study introduces a new method to establish clinical thresholds for multi-item tests, based on item response theory (IRT), as an alternative to receiver operating characteristic (ROC) analysis. The performance of IRT method was examined and compared with the ROC method across multiple simulated data sets and in a real data set.

Study Design and Setting

Simulated data sets (sample size: 1,000) varied in means and variability of the test scores and the prevalence of disease. The true clinical threshold was defined as a predetermined location on the latent trait underlying the questionnaire, with its corresponding expected test score. The real data set (sample size: 295) comprised Hospital Anxiety Depression Scale (HADS) depression scores and Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition major depressive disorder (MDD) diagnoses.

Results

The IRT method recovered the clinical thresholds without bias, whereas the ROC method identified thresholds that were biased by the prevalence of disease. Mild MDD was clinically diagnosed in 23%, moderate MDD in 12%, and severe MDD in 14% of the participants. The IRT method identified the following HADS depression score thresholds for mild, moderate, and severe MDD: 10.7, 13.2, and 15.1, respectively.

Conclusion

The new IRT method identifies clinical thresholds that are unbiased by disease prevalence.

Keywords

Clinical threshold
Diagnostic threshold
Cutoff score
Item response theory
Receiver operating characteristic analysis

Cited by (0)

Conflict of interest: none.