Skip to main content

Advertisement

SpringerLink
Log in

Threats to the Internal Validity of Spinal Surgery Outcome Assessment: Recalibration Response Shift or Implicit Theories of Change?

  • Published:
Applied Research in Quality of Life Aims and scope Submit manuscript

Abstract

A recalibration response shift will cause the patient to think about a self-report measure’s response options differently after a health state change. Commonly assessed using the retrospective-pretest design (“then-test”), recent guidelines suggest adjusting then-test estimates for competing explanations. This prospective longitudinal study investigated recalibration response shift after adjusting for implicit theories of change in patients undergoing spinal surgery. The Oswestry Disability Index (ODI) and Short Form-36 (SF-36) were collected before surgery and at 6 weeks and 3 months after spinal decompression surgery. Then-tests of the measures were also collected at all post-tests. Recalibration response shift was operationalized as the then-minus-pre difference score on the evaluative SF-36. Implicit theories of change were operationalized as the then-minus-pre difference score on the perception-based ODI. Improved vs. No-Effect patient groups were compared using the Minimally Important Difference (±15 points) as a cut-off on the Visual Analogue Scale (VAS) items for back and leg pain. Logistic regression analyses investigated whether recalibration response shift had an independent effect distinguishing patient groups, after adjusting for implicit theories of change. The sample (baseline n = 169, mean age 52, 39 % female) was well-educated, and 1/3 were working. All then-minus-pre difference scores were non-zero at each time point and were stable over time. In the adjusted models distinguishing Improved versus No Effect groups, then-minus-pre ODI difference scores were significant in the majority of the adjusted models at all timepoints, but only one then-minus-pre SF-36 difference score—for physical functioning recalibration—was significant and only at 6-weeks post-surgery. This suggests that implicit theories of change bias the estimation of post-surgical outcomes, but that recalibration response shift biased only the estimation of physical functioning and only at 6 weeks post-surgery. Recalibration response shift and implicit theories of change can both be sources of bias in patient-reported outcome assessment. Our findings suggest that implicit theories of change are a greater threat to validity in this patient sample. Future research using the then-test should control for implicit theories of change to minimize misspecification of effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Ahmed. (2004). Response shift influenced estimates of change in health-related quality of life poststroke. Clinical Epidemiology, 57(6), 10.

    Google Scholar 

  • Ahmed, S., Mayo, N. E., Wood-Dauphinee, S., Hanley, J. A., & Cohen, S. R. (2004). Response shift influenced estimates of change in health-related quality of life poststroke. Journal of Clinical Epidemiology, 57(6), 561–570.

    Article  Google Scholar 

  • Ahmed, S., Mayo, N. E., Wood-Dauphinee, S., Hanley, J. A., & Cohen, S. R. (2005). The structural equation modeling technique did not show a response shift, contrary to the results of the then test and the individualized approaches. Journal of Clinical Epidemiology, 58(11), 1125–1133.

    Article  Google Scholar 

  • Ahmed, S., Mayo, N. E., Corbiere, M., Wood-Dauphinee, S., Hanley, J., & Cohen, R. (2005). Change in quality of life in people with stroke over time: true change or response shift? Quality of Life Research, 14, 611–627.

    Article  Google Scholar 

  • Bernhard, J., Hurny, C., Maibach, R., Herrmann, R., & Laffer, U. (1999). Quality of life as subjective experience: reframing of perception in patients with colon cancer undergoing radical resection with or without adjuvant chemotherapy. Swiss Group for Clinical Cancer Research (SAKK). Annals of Oncology, 10(7), 775–782.

    Article  Google Scholar 

  • Chapman, G. B., Elstein, A. S., Kuzel, T. M., Sharifi, R., Nadler, R. B., Andrews, A., et al. (1998). Prostate cancer patients’ utilities for health states: how it looks depends on where you stand. Medical Decision Making, 18(3), 278–286.

    Article  Google Scholar 

  • Daltroy, L. H., Larson, M. G., Eaton, H. M., Phillips, C. B., & Liang, M. H. (1999). Discrepancies between self-reported and observed physical function in the elderly: the influence of response shift and other factors. Social Science & Medicine, 48(11), 1549–1561.

    Article  Google Scholar 

  • Deyo, R. A., Battie, M., Beurskens, A. J., Bombardier, C., Croft, P., Koes, B., et al. (1998a). Outcome measures for low back pain research. A proposal for standardized use. Spine (Phila Pa 1976), 23(18), 2003–2013.

    Article  Google Scholar 

  • Deyo, R. A., Battie, M., Beurskens, A. J., Bombardier, C., Croft, P., Koes, B., et al. (1998b). Outcome measures for low back pain research. A proposal for standardized use. Spine, 23(18), 2003–2013.

    Article  Google Scholar 

  • Fairbank, J. C., & Pynsent, P. B. (2000). The oswestry disability index. Spine, 25(22), 2940–2952.

    Article  Google Scholar 

  • Finkelstein, J. A., Razmjou, H., & Schwartz, C. E. (2009). Response shift and outcome assessment in orthopedic surgery: is there is a difference between complete vs. partial treatment? Journal of Clinical Epidemiology, 82, 1189–1190.

    Article  Google Scholar 

  • Gudex, C., Dolan, P., Kind, P., & Williams, A. (1996). Health state valuations from the general public using the visual analogue scale. Quality of Life Research, 5(6), 521–531.

    Article  Google Scholar 

  • Hagedoorn, M., Sneeuw, K. C., & Aaronson, N. K. (2002). Changes in physical functioning and quality of life in patients with cancer: response shift and relative evaluation of one’s condition. Journal of Clinical Epidemiology, 55(2), 176–183.

    Article  Google Scholar 

  • Hagg, O., Fritzell, P., & Nordwall, A. (2003). The clinical importance of changes in outcome scores after treatment for chronic low back pain. European Spine Journal, 12(1), 12–20.

    Google Scholar 

  • Heidrich, S. M., & Ryff, C. D. (1993). The role of social comparisons processes in the psychological adaptation of elderly adults. Journal of Gerontology, 48(3), 127–136.

    Article  Google Scholar 

  • Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychology Measurement, 3(1), 1–23.

    Article  Google Scholar 

  • Jansen, S. J., Stiggelbout, A. M., Nooij, M. A., Noordijk, E. M., & Kievit, J. (2000). Response shift in quality of life measurement in early-stage breast cancer patients undergoing radiotherapy. Quality of Life Research, 9(6), 603–615.

    Article  Google Scholar 

  • Kurd, M. F., Lurie, J. D., Zhao, W., Tosteson, T., Hilibrand, A. S., Rihn, J., et al. (2012). Predictors of treatment choice in lumbar spinal stenosis: a spine patient outcomes research trial study. Spine, 37(19), 1702–1707.

    Article  Google Scholar 

  • Li, Y., & Rapkin, B. (2009). Classification and regression tree analysis to identify complex cognitive paths underlying quality of life response shifts: a study of individuals living with HIV/AIDS. Journal of Clinical Epidemiology, 62, 1138–1147.

    Article  Google Scholar 

  • Li, Y., & Schwartz, C. E. (2011). Data mining for response shift patterns using recursive partitioning tree analysis. Quality of Life Research, 20(10), 1543–1553.

    Article  Google Scholar 

  • Mancuso. (1995). Does recollection error threaten the validity of cross-sectional studies of effectiveness? Medical Care, 33(4 Suppl), 12.

    Google Scholar 

  • Middel. (2006). Recall bias did not affect perceived magnitude of change in health-related functional status. Journal of Clinical Epidemiology, 59(5), 9.

    Article  Google Scholar 

  • Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2009). Tests of measurement invariance failed to support the application of the “then-test”. Journal of Clinical Epidemiology, 62(11), 1173–1180.

    Article  Google Scholar 

  • Norman, G. (2003). Hi! How are you? Response shift, implicit theories and differing epistemologies. Quality of Life Research, 12(3), 239–249.

    Article  Google Scholar 

  • Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Medical Care, 41, 582–592.

    Google Scholar 

  • Oort, F. J. V. M., & Sprangers, M. A. G. (2005). An application of structural equation modeling to dtect response shifts and true change in quality of life data from cancer patients undergoing invasive surgery. Quality of Life Research, 14, 599–609.

    Article  Google Scholar 

  • Ostelo, R. W., Deyo, R. A., Stratford, P., Waddell, G., Croft, P., Von Korff, M., et al. (2008). Interpreting change scores for pain and functional status in low back pain: towards international consensus regarding minimal important change. Spine (Phila Pa 1976), 33(1), 90–94.

    Article  Google Scholar 

  • Pearson, A., Lurie, J., Tosteson, T., Zhao, W., Abdu, W., Mirza, S., et al. (2012). Who should have surgery for an intervertebral disc herniation? Comparative effectiveness evidence from the spine patient outcomes research trial. Spine, 37(2), 140–149.

    Article  Google Scholar 

  • Postulart, D., & Adang, E. M. (2000). Response shift and adaptation in chronically ill patients. Medical Decision Making, 20(2), 186–193.

    Article  Google Scholar 

  • Razmjou, H., Yee, A., Ford, M., & Finkelstein, J. A. (2006). Response shift in outcome assessment in patients undergoing total knee arthroplasty. The Journal of Bone and Joint Surgery. American Volume, 88(12), 2590–2595.

    Article  Google Scholar 

  • Rees, J., MacDonagh, R., Waldron, D., & O’Boyle, C. (2004). Measuring quality of life in patients with advanced cancer. European Journal of Palliative Care, 11(3), 104–106.

    Google Scholar 

  • Rijken, M., Komproe, I. H., Ros, W. J. G., Winnubst, J. A. M., & van Heesch, N. C. A. (1995). Subjective well-being of elderly women: conceptual differences between cancer patients, women suffering from chronic ailments and healthy women. British Journal of Clinical Psychology, 34, 289–300.

    Article  Google Scholar 

  • Ring, L. H., Höfer, S., Heuston, F., Harris, D., & O’Boyle, C. A. (2005). Response shift masks the treatment impact on patient reported outcomes (PROs): the example of individual quality of life in edentulous patients. Health and Quality of Life Outcomes, 3, 55.

    Article  Google Scholar 

  • Sajobi, T. T., Lix, L. M., Clara, I., Walker, J., Graff, L. A., Rawsthorne, P., et al. (2012). Measures of relative importance for health-related quality of life. Quality of Life Research, 21, 1–11.

    Article  Google Scholar 

  • Sajobi, T. T., Lix, L. M., Schwartz, C. E., Quaranto, B. R., & Finkelstein, J. A. (2012). Effect of missing data on relative importance measures for response shift detection: an example from an orthopedic population. Quality of Life Research, 21(Supplement 1), 2–3.

    Google Scholar 

  • Schwartz, C. E., & Rapkin, B. D. (2004). Reconsidering the psychometrics of quality of life assessment in light of response shift and appraisal. Health and Quality of Life Outcomes, 2, 16.

    Article  Google Scholar 

  • Schwartz, C. E., & Sprangers, M. A. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science & Medicine, 48(11), 1531–1548.

    Article  Google Scholar 

  • Schwartz, C. E., & Sprangers, M. A. G. (2010). Guidelines for improving the stringency of response shift research using the then-test. Quality of Life Research, 19, 455–464.

    Article  Google Scholar 

  • Schwartz, C. E., Wheeler, H. B., Hammes, B., Basque, N., Edmunds, J., Reed, G., et al. (2002). Early intervention in planning end-of-life care with ambulatory geriatric patients: results of a pilot trial. Archives of Internal Medicine, 162(14), 1611–1618.

    Article  Google Scholar 

  • Schwartz, C. E., Merriman, M., Reed, G., & Hammes, B. (2004). Measuring patient treatment preferences in end-of-life care research: applications for advance care planning interventions and response shift research. Journal of Palliative Medicine, 7(2), 233–245.

    Article  Google Scholar 

  • Schwartz, C. E., Sprangers, M. A. G., Carey, A., & Reed, G. (2004). Exploring response shift in longitudinal data. Psychology and Health, 19(1), 51–69.

    Article  Google Scholar 

  • Schwartz, C. E., Merriman, M. P., Reed, G., & Byock, I. (2005). Evaluation of the missoula-VITAS quality of life index—revised: research tool or clinical tool? Journal of Palliative Medicine, 8(1), 121–135.

    Article  Google Scholar 

  • Schwartz, C. E., Sprangers, M. A., Oort, F. J., Ahmed, S., Bode, R., Li, Y., et al. (2011). Response shift in patients with multiple sclerosis: an application of three statistical techniques. Quality of Life Research, 20(10), 1561–1572.

    Article  Google Scholar 

  • Schwartz, C. E., Sajobi, T., Lix, L., Quaranto, B. R., Finkelstein, J. A. (2013). Changing values, changing outcomes: The influence of reprioritization response shift on outcome assessment after spine surgery. Quality of Life Research. doi:10.1007/s11136-013-0377-x.

  • Sprangers, M. A., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: a theoretical model. Social Science & Medicine, 48(11), 1507–1515.

    Article  Google Scholar 

  • Sprangers, M. A., Van Dam, F. S., Broersen, J., Lodder, L., Wever, L., Visser, M. R., et al. (1999). Revealing response shift in longitudinal research on fatigue–the use of the thentest approach. Acta Oncologica, 38(6), 709–718.

    Article  Google Scholar 

  • Toyone. (2005). Patients’ expectations and satisfaction in lumbar spine surgery. Spine, 30(23), 5.

    Article  Google Scholar 

  • Ware, J. E., Jr., & Sherbourne, C. D. (1992). The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30(6), 473–483.

    Article  Google Scholar 

  • Wikby, A., Stenstrom, U., Hornquist, J. O., & Andersson, P. O. (1993). Coping behaviour and degree of discrepancy between retrospective and prospective self-ratings of change in quality of life in type 1 diabetes mellitus. Diabetic Medicine, 10(9), 851–854.

    Article  Google Scholar 

  • Willis, G. B. (2005). Cognitive interviewing. Thousand Oaks: Sage Publications.

    Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge helpful discussions with Bruce Rapkin, Ph.D., Aileen Davis, Ph.D., Gary Naglie, M.D., FRCP(C), FGSA, and Alex Kiss, Ph.D., on an earlier draft of this manuscript; and Aimee Gallant for her assistance with data collection.

Author information

Authors and Affiliations

  1. Division of Orthopaedics, Sunnybrook Health Sciences Center, Sunnybrook Center for Spinal Trauma, University of Toronto, Toronto, ON, Canada

    Joel A. Finkelstein

  2. DeltaQuest Foundation, Inc, 31 Mitchell Road, Concord, MA, 01742, USA

    Brian R. Quaranto & Carolyn E. Schwartz

  3. Department of Medicine and Orthopaedic Surgery, Tufts University Medical School, Boston, MA, USA

    Carolyn E. Schwartz

  4. Department of Orthopaedic Surgery, Tufts University Medical School, Boston, MA, USA

    Carolyn E. Schwartz

Authors
  1. Joel A. Finkelstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian R. Quaranto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carolyn E. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carolyn E. Schwartz.

Electronic supplementary material

Below is the link to the electronic supplementary material.

S1

(PDF 13.8 kb)

S2

(PDF 8.74 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Finkelstein, J.A., Quaranto, B.R. & Schwartz, C.E. Threats to the Internal Validity of Spinal Surgery Outcome Assessment: Recalibration Response Shift or Implicit Theories of Change?. Applied Research Quality Life 9, 215–232 (2014). https://doi.org/10.1007/s11482-013-9221-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11482-013-9221-2

Keywords

Search

Navigation