The Validity of Prospective and Retrospective Global Change Criterion Measures

doi:10.1016/j.apmr.2005.07.290

Archives of Physical Medicine and Rehabilitation

Volume 86, Issue 12, December 2005, Pages 2270-2276

Presented as a poster to the American Physical Therapy Association Combined Sections Meeting, February 2004, Nashville, TN.

https://doi.org/10.1016/j.apmr.2005.07.290 Get rights and content

Abstract

Schmitt J, Di Fabio RP. The validity of prospective and retrospective global change criterion measures.

Objective

To assess the validity of retrospective versus prospective criterions of change.

Design

Single cohort pretest-posttest design.

Setting

Physical or occupational therapy outpatient clinics.

Participants

Volunteer sample of 211 patients with upper-extremity musculoskeletal problems.

Interventions

Not applicable.

Main Outcome Measures

Disabilities of the Arm, Shoulder, and Hand questionnaire, the Shoulder Pain and Disability Index, the Patient-Rated Wrist Evaluation, the Medical Outcomes Study 12-Item Short-Form Health Survey; global disability rating (GDR), retrospective global rating of change (GRC), and patient satisfaction.

Results

Correlations were calculated among the baseline, 3-month follow-up, and change scores for each outcome measure with the change criterion instruments. Retrospective GRC and patient satisfaction ratings showed moderate correlations with the 3-month follow-up scores, but nonsignificant correlations with baseline scores. By contrast, the prospective GDR criterion showed significant correlations with both baseline and 3-month follow-up scores ranging between 0.3 and 0.4 (absolute value).

Conclusions

Retrospective self-report measures of change do not accurately reflect true change over time. The retrospective GRC and patient satisfaction were heavily influenced by current (posttreatment) status whereas the prospective global change measure reflected both baseline and posttreatment status equally and thus appeared to be a more valid measure of change over time. This study demonstrates the need for an alternative criterion for establishing true individual change.

Section snippets

Participants

Patients with a physician’s diagnosis of a musculoskeletal upper-extremity problem were recruited at the initial visit to local physical and occupational (hand therapy) outpatient clinics. All patients 18 years and older who could read and understand English were eligible for the study. Subjects were excluded if they had primary or coexisting systemic conditions including multiple sclerosis, cancer, rheumatoid arthritis, stroke, or mental or cognitive impairment. The recruitment goal was at

Results

A summary of the subjects’ demographic information is listed in table 1. The patients had a variety of diagnoses involving the upper extremity as is seen in many outpatient orthopedic practices. The most common diagnoses were shoulder pain (n=45 patients, 21% of the sample), shoulder tendonitis (n=33, 16%), shoulder impingement (n=16, 8%), and elbow tendonitis (n=24, 11%).

Six percent of the SF-12 questionnaires had missing items (typically 1 item was omitted). The correlation of the imputed

Discussion

Retrospective global change instruments do not accurately measure change over time. The retrospective measures used in this study were heavily influenced by current (posttreatment) status. As predicted, baseline (pretreatment) status had little or no impact on the retrospective ratings. Van Stel et al⁴⁰ reported similar results in a validation study for the Quality of Life for Respiratory Illness Questionnaire. These findings have implications for researchers who want to categorize patients

Conclusions

Criteria for global change are used frequently in research and clinical practice. Retrospective global change and satisfaction questions appear to reflect follow-up status rather than true change over time. Change scores from a prospectively measured global functional status question reflected both baseline and follow-up status fairly equally, but still accounted for a small to moderate amount of the variance in commonly accepted functional outcome measures for the upper extremity. This study

Acknowledgments

We thank the physical and occupational therapists at Park Nicollet Clinic and the Institute for Athletic Medicine for their support in recruiting patients.

References (55)

G. Guyatt et al.
Measuring change over timeassessing the usefulness of evaluative instruments
J Chronic Dis
(1987)
B. Kirshner et al.
A methodological framework for assessing health indices
J Chronic Dis
(1985)
J.G. Wright et al.
A comparison of different indices of responsiveness
J Clin Epidemiol
(1997)
G. Stucki et al.
Relative responsiveness of condition-specific and generic health status measures in degenerative lumbar spinal stenosis
J Clin Epidemiol
(1995)
R. Jaeschke et al.
Measurement of health status. Ascertaining the minimal clinically important difference
Control Clin Trials
(1989)
G.R. Norman et al.
Methodological problems in the retrospective computation of responsiveness to changethe lesson of Cronbach
J Clin Epidemiol
(1997)
N.H. Williams et al.
Extending the Aberdeen Back Pain Scale to include the whole spinea set of outcome measures for the neck, upper and lower back
Pain
(2001)
E. Chang et al.
Comparison of the responsiveness of lupus disease activity measures to changes in systemic lupus erythematosus activity relevant to patients and physicians
J Clin Epidemiol
(2002)
N. Smidt et al.
Corticosteroid injections, physiotherapy, or a wait-and-see policy for lateral epicondylitisa randomised controlled trial
Lancet
(2002)
J.S. Schmitt et al.
Reliable change and minimum important difference (MID) proportions facilitated group responsiveness comparisons using individual threshold criteria
J Clin Epidemiol
(2004)

G.H. Guyatt et al.

A critical look at transition ratings

J Clin Epidemiol

(2002)

R.G. Marx et al.

Clinimetric and psychometric strategies for development of a health measurement scale

J Clin Epidemiol

(1999)

J.C. MacDermid

Outcome evaluation in patients with elbow pathologyissues in instrument development and evaluation

J Hand Ther

(2001)

J.C. MacDermid et al.

Responsiveness of the short form-36, disability of the arm, shoulder, and hand questionnaire, patient-rated wrist evaluation, and physical impairment measurements in evaluating recovery after a distal radius fracture

J Hand Surg [Am]

(2000)

D. Beaton et al.

Assessing the reliability and responsiveness of 5 shoulder questionnaires

J Shoulder Elbow Surg

(1998)

A.J. Beurskens et al.

Measuring the functional status of patients with low back pain. Assessment of the quality of four disease-specific questionnaires

Spine

(1995)

R.A. Deyo et al.

Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation

Control Clin Trials

(1991)

P.W. Stratford et al.

Health status measuresstrategies and analytic methods for assessing change scores

Phys Ther

(1996)

B. Middel et al.

How to validate clinically important change in health-related functional status. Is the magnitude of the effect size consistently related to magnitude of change as indicated by a global question rating?

J Eval Clin Pract

(2001)

S.J. Taylor et al.

Responsiveness of common outcome measures for patients with low back pain

Spine

(1999)

N.C. Santanello et al.

What are minimal important changes for asthma measures in a clinical trial?

Eur Respir J

(1999)

P.W. Stratford et al.

Sensitivity to change of the Roland-Morris Back Pain Questionnairepart 1

Phys Ther

(1998)

M. Ross

Relation of implicit theories to the construction of personal histories

Psychol Rev

(1989)

J.H. van der Lee et al.

Clinimetric properties of the motor activity log for the assessment of arm use in hemiparetic patients

Stroke

(2004)

P.R. Dunkl et al.

Responsiveness of fibromyalgia clinical trial outcome measures

J Rheumatol

(2000)

D. Locker et al.

Assessing the responsiveness of measures of oral health-related quality of life

Community Dent Oral Epidemiol

(2004)

J.L. Hoving et al.

Manual therapy, physical therapy, or continued care by a general practitioner for patients with neck pain. A randomized, controlled trial

Ann Intern Med

(2002)

Cited by (85)

Achieving satisfactory functional outcomes in conservatively treated proximal humerus fractures: relationship between shoulder range of motion and patient-reported clinical outcome scores
2024, JSES International
Proximal humerus fractures are common osteoporotic fractures. Postinjury outcome measures include objective clinician-measured range of motion (ROM) and subjective patient-reported outcome measures (PROMs), but the relationship between both has not been established. This study aimed to determine the relationship between shoulder ROM and PROMs and establish which ROMs correlated most with PROMs.
A prospective cohort study was conducted on patients with acute proximal humerus fractures. Surgical intervention, open or pathological fractures, neurovascular compromise, polytrauma, or delayed presentations were excluded. Correlation and regression analyses between active ROM and PROMs (Quick Disabilities of Arm, Shoulder and Hand [QuickDASH] and Oxford Shoulder Score [OSS]) at 1-year postinjury were explored. ROM cutoffs predicting satisfactory PROM scores were established.
Fifty-five patients were recruited. Moderate correlations were observed between PROMs and flexion, extension, and abduction, but not internal and external rotation. Multivariate analysis showed significant relationships between PROMs and flexion [QuickDASH: adjusted coefficient (AC): −0.135, P = .013, OSS: AC: 0.072, P = .002], abduction [QuickDASH: AC: −0.115, P = .021, OSS: AC: 0.059, P = .005], and extension [QuickDASH: AC: −0.304, P = .020] adjusting for age, gender, Neer classification, injury on dominant side, and employment. Achieving 130° flexion, 59° extension, and 124° abduction were correlated with satisfactory OSS/QuickDASH scores, respectively.
Overall, holistic assessment of outcomes with both subjective and objective outcomes are necessary, as shoulder flexion, extension, and abduction are only moderately correlated with PROMs. Attaining 130° flexion, 59° extension, and 124° abduction corresponded with satisfactory functional outcomes measured by OSS/QuickDASH and can guide rehabilitation.
Validation of a pictorial-based range of motion self-assessment tool in shoulder arthroplasty
2023, Seminars in Arthroplasty JSES
Range of Motion (ROM) assessment is a key method of assessing recovery progress in shoulder arthroplasty. The development of a reliable patient-based self-assessment tool for ROM could reduce the need for clinician dependent modes of outcome assessment after shoulder arthroplasty, allowing for greater optimization of healthcare resources including physician and clinic time. The primary aim of this study was to test the validity and reliability of a pictorial-based patient self-assessment of ROM in postshoulder arthroplasty in terms of the correlation between patient-based and clinician-based ROM assessments. The secondary aim was to determine the relationship between patient- and clinician-based ROM assessment with corresponding patient-reported outcome measures (PROMs).
This was a cross-sectional study of 126 patients identified from a list of patients scheduled for an elective shoulder arthroplasty postsurgery follow-up visit. Patients completed a pictorial-based ROM self-assessment questionnaire where they identified pictorial diagrams best representing their active shoulder ROM. Shoulder ROM was then assessed by a clinician with a goniometer, and PROMs including the Oxford Shoulder Score, American Shoulder and Elbow Surgeons score, pain score, and satisfaction score were measured. Intraclass correlation coefficients (ICCs) were used to assess interobserver agreement between the clinician and patient-based ROM scores. Pearson correlation and multiple regression analyses were used to analyze the relationship between measured ROM and PROM scores.
Strong correlation was found between patient and clinician measured ROM for forward flexion (ICC = 0.70) and internal rotation (ICC = 0.72), while moderate correlation was noted for abduction (ICC = 0.66) and external rotation (ICC = 0.58). Patients tended to overestimate their shoulder ROM for all four measured movements. Both patient and clinician measured ROM were significantly correlated with PROMs, with forward flexion and abduction being the most correlated with PROMs (coefficient = 0.313-0.544) compared to other ROMs. The questionnaire difficulty was rated as “Very easy,” “Easy,” or “OK” by 96.8% (122/126) of patients.
This pictorial-based ROM self-assessment questionnaire found moderate to high agreement of patient-clinician measured ROM and correlation between measured ROM and PROMs. This easy-to-use self-assessment tool may prove useful in increasing convenience and compliance to outcome assessment in shoulder arthroplasty.
Establishing the Substantial Clinical Benefit in a Non-Shoulder Hand and Upper Extremity Population for the QuickDASH and PROMIS Upper Extremity and Physical Function Computer Adaptive Tests
2022, Journal of Hand Surgery
Citation Excerpt :
It remains possible that different GRC anchors might result in estimates for the SCB outside of our estimated range. Additionally, retrospective administration of GRCs has been shown to reflect posttreatment status to a greater degree than the change from baseline, raising concerns regarding recall and experimenter biases.35 We attempted to mitigate this by collecting the anchor question and PRO instrument responses in real-time at each surgeon visit as a part of routine care.
It is unclear what score changes on the abbreviated Disabilities of the Arm, Hand, and Shoulder (QuickDASH), Patient-Reported Outcomes Measurement Information System (PROMIS) upper extremity (UE) computer adaptive test (CAT), and PROMIS physical function (PF) CAT represent a substantial improvement. We calculated the substantial clinical benefit (SCB) for these 3 instruments in a non-shoulder hand and upper extremity population.
Adult patients treated between March 2015 and September 2019 at a single academic tertiary institution were identified. The QuickDASH, PROMIS UE CAT v2.0, and PROMIS PF CAT v2.0 scores were collected using a tablet computer. Responses to the QuickDASH both at baseline and follow-up 6 ± 4 weeks later, and a response to the anchor question “Compared to your first evaluation at the University Orthopaedic Center, how would you describe your physical function level now?” were required for inclusion. A second anchor question querying treatment-related improvement was also used. The SCB was calculated using an anchor-based approach comparing the mean change difference between groups reporting no change and a maximal change for both anchor questions.
Of 1,119 included participants, the mean age was 48 ± 17 years, 53% were women, and half were recovering from surgery. Score changes between baseline and follow-up were significantly different between groups reporting no improvement and maximal improvement on both anchor questions. The SCB values ranged between 16.9 and 22.8 on the QuickDASH, 5.9 and 7.1 on the UE CAT, and 3.5 and 6.7 on the PF CAT.
These score improvements for the QuickDASH, UE CAT, and PF CAT represent a substantial clinical improvement in a non-shoulder hand and upper extremity population.
These SCB estimates may assist with the interpretation of outcome scores at a population level.
Present state bias in transition ratings was accurately estimated in simulated and real data
2022, Journal of Clinical Epidemiology
Citation Excerpt :
In contrast, the Devji III criterion correctly classified these datasets as “definitely no,” but failed to appreciate the absence of PSB in dataset 4. Since Guyatt et al. [3] introduced the equal-but-opposite correlations and regression coefficients rules, a number of studies have used these criteria, often coming to the conclusion that their transition ratings were subject to PSB (e.g., [8–10,24–27]). As the Guyatt criteria lack recommended cut-off values, they are difficult to interpret.
Patient-reported transition ratings are supposed to reflect the change between a previous baseline health state and a present follow-up state, but may reflect the present state to a greater extent. This so-called “present state bias” (PSB) potentially threatens the validity of transition ratings. Several criteria have been proposed to assess PSB. We examined how well these criteria perform and to which extent confirmatory factor analysis (CFA) for categorical data provides an accurate assessment of the degree of PSB.
We simulated multiple samples with baseline and follow-up item responses to a hypothetical questionnaire, and transition ratings. The samples varied with respect to various distributional characteristics and the degree of PSB. The performance of criteria proposed in the literature, and a new CFA-based criterion, were evaluated by the proportion of explained variance in PSB. In addition, four real datasets were analyzed.
The known criteria explained 36–74% of the variance in PSB. A new CFA-based criterion, namely the ratio of the factor loadings of the transition ratings plus one, explained 81–98% of the variance in PSB across the samples.
Present state bias in transition ratings can be estimated accurately using CFA.
“Looking Under the Hood” of Anchor-Based Assessment of Clinically Important Change: A Machine Learning Approach
2021, Value in Health
The Global Assessment of Change (GAC) item has facilitated the interpretation of change in patient-reported outcomes, providing an anchor for computing minimally important differences. Construct validity has been documented via disease-specific patient-reported outcomes change. We examined what domains, sociodemographic characteristics, attributions of change, and cognitive-appraisal processes are reflected in GAC ratings.
This secondary analysis examined data from 1,481 chronically ill patients and caregivers surveyed at baseline and 17 months. Items queried change since baseline in overall disease symptoms (GAC) and in physical, emotional, and social functioning. Candidate predictors included sociodemographic factors, health-related quality-of-life domains, change attributions, and quality-of-life appraisal processes. Least absolute shrinkage and selection operator and bootstrapping tested 77 predictors’ effectiveness and stability.
GAC worsening was notably associated with being disabled (β = −0.24) and having difficulty paying bills (β = −0.13). GAC was better explained by the physical domain than the emotional or social (β = 0.67, 0.10, and 0.03, respectively; R²_adj = 0.63) after sociodemographic-covariate adjustment. In a separate model (R²_adj = 0.18), GAC variance was explained by attributions about changing health and changing response of one’s health team, goals related to solving healthcare problems and maintaining activities, and appraisal about things getting better (β = −0.14, 0.08, −0.07, 0.05, 0.21, respectively; p_range ~0.0005–0.05) after adjustment.
The GAC primarily reflects the physical domain, and the GAC reflects attributions, goals, and patterns of emphasis related to change in health and healthcare. Commonly unmeasured factors have some bearing on GAC scores and can facilitate the interpretation of change.
Comparative Responsiveness of the PROMIS Pain Interference Short Forms With Legacy Pain Measures: Results From Three Randomized Clinical Trials
2019, Journal of Pain
The Patient-Reported Outcomes Measurement Information System Pain Interference (PROMIS-PI) scales are reliable and publicly accessible; however, little is known about how responsive they are to detect change in clinical trials and how their responsiveness compares with legacy measures. The study purpose was to evaluate responsiveness for the PROMIS-PI scales and to compare their responsiveness with legacy pain measures. We used data from 3 clinical trials totaling 759 participants. The clinical trials included patients with chronic low back pain (n = 261), chronic back or osteoarthritis pain (n = 240), and a history of stroke (n = 258). At both baseline and follow-up, participants completed PROMIS-PI scales and legacy pain measures (Brief Pain Inventory Interference scale; Pain, Enjoyment, General Activity scale; 36-Item Short Form Survey (SF-36) Bodily Pain scale; and Roland-Morris Disability Questionnaire). We measured global ratings of pain change, both prospectively and retrospectively, as anchors to identify patients as improved, unchanged, or worsened. Responsiveness was assessed with standardized response means, statistical tests comparing change groups, and area under the curve analysis. The PROMIS-PI scales had largely comparable responsiveness with the Brief Pain Inventory Interference and Pain, Enjoyment, General Activity scales. The 4 PROMIS-PI short forms had comparable responsiveness. For all pain questionnaires, responsiveness varied based on the study population and whether pain improved or worsened.
This article presents 1) how responsive the PROMIS-PI scales were to detect change over time in the context of 3 clinical trials and 2) how their responsiveness compared with legacy pain measures. The findings can help researchers and clinicians choose between different patient-reported pain outcome measures.

View all citing articles on Scopus

Supported in part by the American Physical Therapy Association Orthopaedic Section.

No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the author(s) or upon any organization with which the author(s) is/are associated.

View full text

Original articleThe Validity of Prospective and Retrospective Global Change Criterion Measures

Abstract

Objective

Design

Setting

Participants

Interventions

Main Outcome Measures

Results

Conclusions

Section snippets

Participants

Results

Discussion

Conclusions

Acknowledgments

J Chronic Dis

J Chronic Dis

J Clin Epidemiol

J Clin Epidemiol

Control Clin Trials

J Clin Epidemiol

Pain

J Clin Epidemiol

Lancet

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

J Hand Ther

J Hand Surg [Am]

J Shoulder Elbow Surg

Measuring the functional status of patients with low back pain. Assessment of the quality of four disease-specific questionnaires

Spine

Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation

Control Clin Trials

Health status measuresstrategies and analytic methods for assessing change scores

Phys Ther

How to validate clinically important change in health-related functional status. Is the magnitude of the effect size consistently related to magnitude of change as indicated by a global question rating?

J Eval Clin Pract

Responsiveness of common outcome measures for patients with low back pain

Spine

What are minimal important changes for asthma measures in a clinical trial?

Eur Respir J

Sensitivity to change of the Roland-Morris Back Pain Questionnairepart 1

Phys Ther

Relation of implicit theories to the construction of personal histories

Psychol Rev

Clinimetric properties of the motor activity log for the assessment of arm use in hemiparetic patients

Stroke

Responsiveness of fibromyalgia clinical trial outcome measures

J Rheumatol

Assessing the responsiveness of measures of oral health-related quality of life

Community Dent Oral Epidemiol

Manual therapy, physical therapy, or continued care by a general practitioner for patients with neck pain. A randomized, controlled trial

Ann Intern Med

Original article
The Validity of Prospective and Retrospective Global Change Criterion Measures