Original articles
Assessing Smallest Detectable Change Over Time in Continuous Structural Outcome Measures: Application to Radiological Change in Knee Osteoarthritis

https://doi.org/10.1016/S0895-4356(99)00109-2Get rights and content

Abstract

Interpreting changes in continuous structural outcome measures is a common problem in clinical research and in daily practice. We propose a method for estimating whether difference observed between two successive measures in an individual constitutes a statistically relevant change or a change induced by variability. This statistically relevant change is based on an analysis of reproducibility. The continuous structural outcome measure investigated as an example was joint space width (JSW) measurement on standard X-rays, which is known to be the primary end-point for assessing structural osteoarthritis progression. The results of the present study demonstrate that cutoffs are closely dependent on all sources of variabilities in JSW measurement such as joint positioning, radiographic procedure, and the measurement process itself. Therefore, we suggest to determine cutoffs for each study using a representative sample of the population studied and using the procedures and methods of measurement of the specific study. This approach may easily be extended to other continuous structural outcome measures.

Introduction

In clinical trials and longitudinal epidemiological studies, continuous outcome measures are frequently used. An important drawback of continuous outcome measure is the difficulty in interpreting the changes observed. For example, what is the meaning of a decrease in pain of 10 mm on a visual analogue scale or of a 0.03-mm increase per year in carotid arterial intima-media thickness?

A pivotal step in translating clinical research into practice is summarizing data from these studies in terms of proportions of patients who progress, a concept that can be easily appraised by clinicians, investigators, and regulatory authorities. Hence, defining a cutoff above which patients are considered to progress is necessary.

In the field of symptomatic assessment, methods have been developed for estimating the minimally important difference for measurement of health status. These methods have focused on the patient as the arbitrator of what constitutes the minimal important difference in symptoms 1, 2, 3.

In the field of structural assessment (e.g., bone mineral densitometry measurement by dual energy X-ray absorptiometry to measure progression of osteoporosis, or measurement of carotid arterial intima-media thickness by ultrasound to measure progression of atherosclerosis) 4, 5, 6, the patient's judgment cannot as easily be used to determine what is a minimal important difference in structures. A drug that affects the long-term course of a disease may have no direct short-term effect on symptoms, and differences in structural assessment do not necessarily imply symptomatic benefits over the course of a study.

To assess the validity of a measure, we could use methods that investigate its predictive validity. These studies must 1) be longitudinal, 2) specify the clinically relevant event that defines the end-point (e.g., vertebral fracture and bone densitometry, myocardial infarction, and cholesterol level).

However, some structural measurements could confront the problem of difficulties in identifying an appropriate standard (i.e., clinically relevant event) to which they can be related. In osteoarthritis (OA), some preliminary studies have evaluated the predictive validity of joint space width (JSW) progression by considering the need for articular replacement as the clinically relevant end-point. However, the decision for articular replacement depends partly on factors unrelated to disease progression (health service system, concurrent medical condition, etc.) and is therefore not recommended as an outcome variable [7]. As a consequence, there is presently no consensus on definition of radiological progression in OA for a group of patients or for an individual even if some authors have proposed an arbitrary minimal change of 1 or 2 mm in JSW [8].

Whatever the choice of end-point, the results of all the studies evaluating the predictive validity of an outcome measure are summarized in dichotomous variable (i.e., increase >0.03 mm per year in carotid arterial intima-media thickness). However, to be clinically relevant in terms of predictive validity, this cutoff has to be at least over the measurement error.

Therefore, for a specific structural outcome measure two possibilities could be encountered:

  • 1.

    Some longitudinal studies have evaluated predictive validity and propose relevant cutoffs. In this case, it is important to check whether this cutoff is superior to the measurement error.

  • 2.

    The predictive validity of the measure has not yet been evaluated. In this case, it is interesting to evaluate the level of measurement error because it is obvious that a clinically relevant cutoff should at least be over this measurement error. In this situation, we cannot state with certainty what amount of change is really clinically important and we will have to retain statistical evaluation of the smallest detectable change [9].

The aim of this study is therefore to propose a method based on a reproducibility study to estimate cutoffs that assess the smallest detectable change. The latter then allows to distinguish between patients with probably true organic change (i.e., with a change greater than the variability inherent both in repeating the imaging procedure and in the measurement process itself) and patients who remained stable (i.e., with a change possibly explained by the measurement error alone).

We illustrate this aim using the measurement of joint space width in medial knee osteoarthritis, which is considered as the primary end-point for assessing structural OA progression (i.e., alteration of joint structure) in clinical trials and epidemiological studies 7, 8, 10, 11.

Section snippets

Definition of cutoffs for the assessment of individual progression

Measurement error is derived from several sources, including error due to the measurement process (reader, measuring instrument), the radiographic procedure, and patient positioning 12, 13, 14. The variability of the measurement error can therefore be estimated from measurements using X-rays repeated over a short period, assuming that no organic change can occur over this period.

Let Xij be the measurement made at time j (j = 1, 2) on subject i. If there is no organic change from time 1 to time

Definition of cutoffs for the assessment of progression of a group of patients

The two cutoffs ICOL and ICOU are of interest when dealing with individual changes. If we no longer focus on individual progression but on the mean progression for a group of n patients, the variable of interest is then the mean difference from time 1 to time 2: D̄ = (1/ni=1n (Xi1Xi2). D̄ is a random variable normally distributed with mean 0 and variance ϵ2n as the random variables Xi1 − Xi2 (i = 1,…, n) are independent and identically distributed. Therefore, 95% of the mean differences

Example

To illustrate the methodology described, we estimated cutoffs defining minimal relevant radiological change in JSW is a sample of medial knee OA patients. Thirty patients with medial OA were radiographed in anteroposterior weight-bearing extended view. Characteristics of these patients are shown in Table 1. These data, including Lequesne's functional index [16], were obtained by one of us (GRA) before the first X-ray except for the Kellgren and Lawrence grade [17], which was determined by PR.

Discussion

Structural continuous outcome measures are widely used in several diseases such as osteoarthritis, osteoporosis, and atherosclerosis to assess disease progression. Interpreting changes in these continuous outcome measures is therefore a common problem in clinical research and in daily practice.

To assess progression in OA, JSW is considered as the primary end-point. Cutoffs on JSW measurements are necessary to decide whether the difference observed between two successive measures in the same

References (23)

  • B. Burnand et al.

    Indexes and boundaries for “quantitative significance” in statistical decisions

    J Clin Epidemiol

    (1990)
  • D.A. Redelmeier et al.

    Assessing the minimal important difference in symptomsA comparison of two techniques

    J Clin Epidemiol

    (1996)
  • D. de Bacquer et al.

    Intra-individual variability of fibrinogen levels

    J Clin Epidemiol

    (1997)
  • D.A. Redelmeier et al.

    Assessing the clinical importance of symptomatic improvements. An illustration in rheumatology

    Arch Intern Med

    (1993)
  • H.K. Genant et al.

    Noninvasive assessment of bone mineral and structureState of the art

    J Bone Miner Res

    (1996)
  • G. Howard et al.

    Cigarette smoking and progression of atherosclerosisThe atherosclerosis risk in communities (ARIC) study

    JAMA

    (1998)
  • H.N. Hodis et al.

    The role of carotid arterial intima-media thickness in predicting clinical coronary events

    Ann Intern Med

    (1998)
  • R. Altman et al.

    Recommendations from a task force of the Osteoarthritis Research SocietyDesign and conduct of clinical trials in patients with osteoarthritis

    Osteoarthritis Cartilage

    (1996)
  • P.A. Dieppe et al.

    A two-year placebo-controlled trial of non steroidal antiinflammatory therapy in osteoarthritis of the knee joint

    Br J Rheumatol

    (1993)
  • R. Rau et al.

    A new method of scoring radiographic change in rheumatoid arthritis

    J Rheumatol

    (1998)
  • Recommendations for the registration of drugs used in the treatment of osteoarthritis

    Ann Rheum Dis

    (1995)
  • Cited by (57)

    • Impact of repeated measures of joint space width on the sample size calculation: An application to hip osteoarthritis

      2015, Joint Bone Spine
      Citation Excerpt :

      However, in a previous OA study [4], a sample of 10% of the whole radiographs was considered to be representative to determine the reliability of the radiographic procedure (in ECHODIAH, 507 radiographs were analyzed). Secondly, this approach based on SDD allows to determine statistical cut-offs of radiological progression for the calculation of sample size but not clinically relevant cut-offs, i.e., a change related to clinically relevant structural outcome measures in OA [18]. However, recently an expert committee failed to propose a data-driven definition of clinically relevant change [1].

    • Assessment of Health Outcomes

      2012, Kelley's Textbook of Rheumatology: Volume 1-2, Ninth Edition
    • Measurement of renal volume using respiratory-gated MRI in subjects without known kidney disease: Intraobserver, interobserver, and interstudy reproducibility

      2011, European Journal of Radiology
      Citation Excerpt :

      In the Bland-Altman analysis of reproducibility, CoR is an estimation (expressed in absolute units) of the overall precision which permits the calculation of 95% limits of agreement [9,10]. Finally, Ravaud et al. have demonstrated that the CoR coincide with the threshold we defined before, i.e. the smallest detectable difference [11]. On the other hand, if we cannot exclude a systematic error (i.e. when the bias is significantly different from zero), we must take into account this source of variation and consider the sum of bias and CoR as a measure of the overall variability (as it was done in this study).

    • Influence of the experience of the reader on reliability of joint space width measurement. A cross-sectional multiple reading study in hip osteoarthritis

      2011, Joint Bone Spine
      Citation Excerpt :

      The SDD is then defined as 1.96 SD of the change between measurements and provides an absolute estimate of measurement error. Thus, a change in JSW during a trial greater than the SDD reflects radiological progression, that is to say a change greater than the variability of the measurement process [7]. Confidence intervals (CI)– 95% – of SDDs were estimated with bootstrapping methods.

    View all citing articles on Scopus
    View full text