Original articlesAssessing Smallest Detectable Change Over Time in Continuous Structural Outcome Measures: Application to Radiological Change in Knee Osteoarthritis
Introduction
In clinical trials and longitudinal epidemiological studies, continuous outcome measures are frequently used. An important drawback of continuous outcome measure is the difficulty in interpreting the changes observed. For example, what is the meaning of a decrease in pain of 10 mm on a visual analogue scale or of a 0.03-mm increase per year in carotid arterial intima-media thickness?
A pivotal step in translating clinical research into practice is summarizing data from these studies in terms of proportions of patients who progress, a concept that can be easily appraised by clinicians, investigators, and regulatory authorities. Hence, defining a cutoff above which patients are considered to progress is necessary.
In the field of symptomatic assessment, methods have been developed for estimating the minimally important difference for measurement of health status. These methods have focused on the patient as the arbitrator of what constitutes the minimal important difference in symptoms 1, 2, 3.
In the field of structural assessment (e.g., bone mineral densitometry measurement by dual energy X-ray absorptiometry to measure progression of osteoporosis, or measurement of carotid arterial intima-media thickness by ultrasound to measure progression of atherosclerosis) 4, 5, 6, the patient's judgment cannot as easily be used to determine what is a minimal important difference in structures. A drug that affects the long-term course of a disease may have no direct short-term effect on symptoms, and differences in structural assessment do not necessarily imply symptomatic benefits over the course of a study.
To assess the validity of a measure, we could use methods that investigate its predictive validity. These studies must 1) be longitudinal, 2) specify the clinically relevant event that defines the end-point (e.g., vertebral fracture and bone densitometry, myocardial infarction, and cholesterol level).
However, some structural measurements could confront the problem of difficulties in identifying an appropriate standard (i.e., clinically relevant event) to which they can be related. In osteoarthritis (OA), some preliminary studies have evaluated the predictive validity of joint space width (JSW) progression by considering the need for articular replacement as the clinically relevant end-point. However, the decision for articular replacement depends partly on factors unrelated to disease progression (health service system, concurrent medical condition, etc.) and is therefore not recommended as an outcome variable [7]. As a consequence, there is presently no consensus on definition of radiological progression in OA for a group of patients or for an individual even if some authors have proposed an arbitrary minimal change of 1 or 2 mm in JSW [8].
Whatever the choice of end-point, the results of all the studies evaluating the predictive validity of an outcome measure are summarized in dichotomous variable (i.e., increase >0.03 mm per year in carotid arterial intima-media thickness). However, to be clinically relevant in terms of predictive validity, this cutoff has to be at least over the measurement error.
Therefore, for a specific structural outcome measure two possibilities could be encountered:
- 1.
Some longitudinal studies have evaluated predictive validity and propose relevant cutoffs. In this case, it is important to check whether this cutoff is superior to the measurement error.
- 2.
The predictive validity of the measure has not yet been evaluated. In this case, it is interesting to evaluate the level of measurement error because it is obvious that a clinically relevant cutoff should at least be over this measurement error. In this situation, we cannot state with certainty what amount of change is really clinically important and we will have to retain statistical evaluation of the smallest detectable change [9].
The aim of this study is therefore to propose a method based on a reproducibility study to estimate cutoffs that assess the smallest detectable change. The latter then allows to distinguish between patients with probably true organic change (i.e., with a change greater than the variability inherent both in repeating the imaging procedure and in the measurement process itself) and patients who remained stable (i.e., with a change possibly explained by the measurement error alone).
We illustrate this aim using the measurement of joint space width in medial knee osteoarthritis, which is considered as the primary end-point for assessing structural OA progression (i.e., alteration of joint structure) in clinical trials and epidemiological studies 7, 8, 10, 11.
Section snippets
Definition of cutoffs for the assessment of individual progression
Measurement error is derived from several sources, including error due to the measurement process (reader, measuring instrument), the radiographic procedure, and patient positioning 12, 13, 14. The variability of the measurement error can therefore be estimated from measurements using X-rays repeated over a short period, assuming that no organic change can occur over this period.
Let Xij be the measurement made at time j (j = 1, 2) on subject i. If there is no organic change from time 1 to time
Definition of cutoffs for the assessment of progression of a group of patients
The two cutoffs ICOL and ICOU are of interest when dealing with individual changes. If we no longer focus on individual progression but on the mean progression for a group of n patients, the variable of interest is then the mean difference from time 1 to time 2: D̄ = (1/n)Σi=1n (Xi1 − Xi2). D̄ is a random variable normally distributed with mean 0 and variance as the random variables Xi1 − Xi2 (i = 1,…, n) are independent and identically distributed. Therefore, 95% of the mean differences
Example
To illustrate the methodology described, we estimated cutoffs defining minimal relevant radiological change in JSW is a sample of medial knee OA patients. Thirty patients with medial OA were radiographed in anteroposterior weight-bearing extended view. Characteristics of these patients are shown in Table 1. These data, including Lequesne's functional index [16], were obtained by one of us (GRA) before the first X-ray except for the Kellgren and Lawrence grade [17], which was determined by PR.
Discussion
Structural continuous outcome measures are widely used in several diseases such as osteoarthritis, osteoporosis, and atherosclerosis to assess disease progression. Interpreting changes in these continuous outcome measures is therefore a common problem in clinical research and in daily practice.
To assess progression in OA, JSW is considered as the primary end-point. Cutoffs on JSW measurements are necessary to decide whether the difference observed between two successive measures in the same
References (23)
- et al.
Indexes and boundaries for “quantitative significance” in statistical decisions
J Clin Epidemiol
(1990) - et al.
Assessing the minimal important difference in symptomsA comparison of two techniques
J Clin Epidemiol
(1996) - et al.
Intra-individual variability of fibrinogen levels
J Clin Epidemiol
(1997) - et al.
Assessing the clinical importance of symptomatic improvements. An illustration in rheumatology
Arch Intern Med
(1993) - et al.
Noninvasive assessment of bone mineral and structureState of the art
J Bone Miner Res
(1996) - et al.
Cigarette smoking and progression of atherosclerosisThe atherosclerosis risk in communities (ARIC) study
JAMA
(1998) - et al.
The role of carotid arterial intima-media thickness in predicting clinical coronary events
Ann Intern Med
(1998) - et al.
Recommendations from a task force of the Osteoarthritis Research SocietyDesign and conduct of clinical trials in patients with osteoarthritis
Osteoarthritis Cartilage
(1996) - et al.
A two-year placebo-controlled trial of non steroidal antiinflammatory therapy in osteoarthritis of the knee joint
Br J Rheumatol
(1993) - et al.
A new method of scoring radiographic change in rheumatoid arthritis
J Rheumatol
(1998)
Recommendations for the registration of drugs used in the treatment of osteoarthritis
Ann Rheum Dis
Cited by (57)
Changes in knee joint load indices from before to 12 months after arthroscopic partial meniscectomy: A prospective cohort study
2016, Osteoarthritis and CartilageImpact of repeated measures of joint space width on the sample size calculation: An application to hip osteoarthritis
2015, Joint Bone SpineCitation Excerpt :However, in a previous OA study [4], a sample of 10% of the whole radiographs was considered to be representative to determine the reliability of the radiographic procedure (in ECHODIAH, 507 radiographs were analyzed). Secondly, this approach based on SDD allows to determine statistical cut-offs of radiological progression for the calculation of sample size but not clinically relevant cut-offs, i.e., a change related to clinically relevant structural outcome measures in OA [18]. However, recently an expert committee failed to propose a data-driven definition of clinically relevant change [1].
Assessment of Health Outcomes
2012, Kelley's Textbook of Rheumatology: Volume 1-2, Ninth EditionMeasurement of renal volume using respiratory-gated MRI in subjects without known kidney disease: Intraobserver, interobserver, and interstudy reproducibility
2011, European Journal of RadiologyCitation Excerpt :In the Bland-Altman analysis of reproducibility, CoR is an estimation (expressed in absolute units) of the overall precision which permits the calculation of 95% limits of agreement [9,10]. Finally, Ravaud et al. have demonstrated that the CoR coincide with the threshold we defined before, i.e. the smallest detectable difference [11]. On the other hand, if we cannot exclude a systematic error (i.e. when the bias is significantly different from zero), we must take into account this source of variation and consider the sum of bias and CoR as a measure of the overall variability (as it was done in this study).
Influence of the experience of the reader on reliability of joint space width measurement. A cross-sectional multiple reading study in hip osteoarthritis
2011, Joint Bone SpineCitation Excerpt :The SDD is then defined as 1.96 SD of the change between measurements and provides an absolute estimate of measurement error. Thus, a change in JSW during a trial greater than the SDD reflects radiological progression, that is to say a change greater than the variability of the measurement process [7]. Confidence intervals (CI)– 95% – of SDDs were estimated with bootstrapping methods.