Original Article
Using the entire cohort in the receiver operating characteristic analysis maximizes precision of the minimal important difference

https://doi.org/10.1016/j.jclinepi.2008.07.009Get rights and content

Abstract

Objective

We compared the minimal important difference (MID) values obtained by the receiver operating characteristics (ROC) curve approach using different strategies on four outcome measures to guide the optimal use of ROC curve.

Study Design and Setting

Studies of two psychometric scales (Rhinoconjunctivitis Quality-of-Life Questionnaire [RQLQ] and Chronic Respiratory Questionnaire [CRQ]) and two clinimetric indices (Pediatric Ulcerative Colitis Activity Index [PUCAI] and Pediatric Crohn's Disease Activity Index [PCDAI]) instruments provided prospective longitudinal data. The MID was calculated from 7- and 15-point global ratings of change dichotomized in multiple ways, using the ROC curve method. Analysis was performed twice: first, using only the two groups adjacent to the dichotomization point (e.g., including only patients who had a small vs. moderate change); and second, using the entire cohort split at the same cutoff (e.g., including both unchanged subjects with those with small change vs. those who experienced moderate or large change combined).

Results

Using the entire cohort, rather than just those with ratings adjacent to the dichotomization point, yielded more precise and sensible MID estimates. With one exception, high precision was obtained when using the ROC curve method for any cutoff on the rating scale.

Conclusion

When calculating the MID using the ROC curve method, the use of the entire cohort maximizes precision.

Introduction

The minimal clinically important difference (MCID) was defined as the smallest change in an outcome measure perceived as beneficial by patients or physicians, and which leads to a change in the patient's management, assuming minimal toxicity and cost [1], [2]. Clearer thinking subsequently led to increased emphasis on identifying whose perspective the minimally important difference (MID) is based on and substitution of “MID” for MCID [3], [4]. For patient-reported health-related quality-of-life measures, the MID describes the patient's perspective. However, disease activity indices rely on clinicians' judgment and, thus, the MID describes the clinicians' perspective. Such clinimetric indices rely on judgment by experienced clinicians who interpret the clinical symptoms, signs, imaging, and laboratory data to arrive at a gestalt of the extent of disease activity [5], [6].

Approaches to determining the MID can be usefully categorized as distribution-based and anchor-based [7]. The external criterion most widely used in the anchor-based approaches is a global rating of change (GRC), which is a Likert-type scale scored by the patient or the physician, from “large deterioration” through “no change” and to “large improvement” [8]. The original statistical strategy to ascertain the MID using the GRC calculates the mean change of patients who rate themselves as having a small change (e.g., 1–3 points on a 15-point scale in which 7 represents large deterioration and +7 large improvement; or 1 point on a 7-point scale in which -3 represents large deterioration and +3 large improvement; in both “0” means “no change”) [1]. Others used the mean change of patients graded as 2–3 [2], [9] or 5 [10], [11] points on a -7 to +7-point scale or used GRC scales with varying numbers of categories [12], [13], [14], [15].

As early as 1986 [13], investigators have recognized the benefit of an alternative statistical strategy that uses the receiver operating characteristics (ROC) curve method to define MID [11], [13], [14], [16], [17], [18], [19], [20], [21]. This approach offers a group-level interpretation (area under the curve [AUC]) and potential for ascertaining the diagnostic accuracy of different change scores for correctly classifying improved vs. nonimproved patients. The ROC approach always relies on a dichotomous standard, the choice of which must be carefully determined. Investigators use criteria, such as “improved” or not, or “therapy changed” or not, as the external standard [13], [17], [18]. A clearer description of the magnitude of change that occurred may be achieved with a GRC that includes more than two categories [14], [19], [20]. The optimal threshold to dichotomize such a GRC to ascertain the minimal amount of change that is important, and which groups to include in the analysis, remains uncertain.

Here, we evaluate the ROC method to ascertain the MID using two psychometric instruments and two clinimetric indices. We examine the impact of alternative dichotomization thresholds of the GRC, and repeat each analysis twice, once with only the groups adjacent to the dichotomization point and once with the combined cohort.

Section snippets

Materials and methods

For this study, we used prospectively collected longitudinal datasets of the following well-established instruments: the Pediatric Ulcerative Colitis Activity Index (PUCAI) [22], the Pediatric Crohn's Disease Activity Index (PCDAI) [19], [23], the Rhinoconjunctivitis Quality-of-Life Questionnaire (RQLQ) [24], and the Chronic Respiratory Questionnaire (CRQ) [2], [25].

On the 7-point GRC (-3 to +3) used in the clinimetric indices (PUCAI and PCDAI), “0” was referred to as “no-change,” “1” as “small

Results

The correlations between change in the relevant instruments/domains and the global ratings were 0.50, for the RQLQ, 0.61 for the dyspnea domain of the CRQ, 0.50 for the fatigue domain of the CRQ, 0.53 for the emotions domain of the CRQ, 0.72 for the PCDAI, and 0.85 for the PUCAI. All included indices and domains also satisfied criteria for the correlation of the baseline and the follow-up scores with the GRC.

Using the ROC approach, the different MID values obtained in analysis using all data

Discussion

This study provides novel data to guide utilization of the ROC approach in finding the MID. The ROC approach for determining the MID from GRC has several advantages over the most popular alternative, namely the mean value of those who report small but important changes [1]. The ROC approach is more suitable for data that is not normally distributed, and can use the entire data set, thus, maximizing precision. Most importantly, by using the mean change, individuals who score lower than the mean

Summary

Using two patient-based and two physician-based instruments, we present novel data that should guide future identification of the MID. For the use of anchor-based approach, an external criterion that quantifies a gradient of change, such as the 15- or the 7-point GRC facilitates establishing the validity of the anchor and allows identification of small, moderate, and large changes. The latter two may represent the minimal difference needed to institute expensive or toxic therapy. Several

Acknowledgments

No funding was received for this study. Dr Beaton was supported by a CIHR New Investigators award during the conduct of this research. Dr. Holger J. Schünemann was funded by a European Commission: The human factor, mobility and Marie Curie Actions. Scientist Reintegration Grant (IGR 42192).

References (29)

  • H.J. Schunemann et al.

    Commentary—goodbye M(C)ID! Hello MID, where do you come from?

    Health Serv Res

    (2005)
  • D.E. Beaton et al.

    Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research

    Curr Opin Rheumatol

    (2002)
  • D.E. Beaton

    Understanding the relevance of measured change through studies of responsiveness

    Spine

    (2000)
  • G.R. Norman et al.

    Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life

    Med Care

    (2001)
  • Cited by (0)

    View full text