Original Article
A new approach to combining clinical relevance and statistical significance for evaluation of quality of life changes in the individual patient

https://doi.org/10.1016/j.jclinepi.2009.03.016Get rights and content

Abstract

Objective

Empirical investigation of formerly proposed criteria for relevant changes of health-related quality of life (QOL) regarding their use for monitoring changes in the individual patient. Suggestion of a new criterion trying to overcome the drawbacks of former criteria.

Study Design and Setting

QOL data were collected longitudinally in 160 cancer patients receiving chemotherapy at an oncological outpatient unit, giving rise to a total of 975 QOL assessments. QOL was measured using the European Organization on Research and Treatment of Cancer Quality of Life Core Questionnaire. Several formerly suggested criteria of relevant change (distribution based, anchor based) were compared in terms of both prevalence and statistical significance of the resulting relevant changes.

Results

When considering criteria of relevant change suggested in the literature, high proportions (average: 42.3–48.3%) of reputedly relevant changes were found. The majority of these changes (average: 55.8–62.2%) were statistically insignificant. Combination of an increased threshold for clinical relevance with the concept of statistical significance resulted in a more meaningful change criterion.

Conclusion

Formerly recommended thresholds of relevant change in QOL appear to be unduly low when focusing on the individual patient. A modified criterion is therefore suggested for this case. However, more research is needed for validation and refinement of the proposed criterion.

Introduction

What is new?

Key message

  1. When dealing with the quality of life (QOL) of individual subjects rather than groups, the usual concept of minimal important difference (MID) needs reconsideration and possibly redefinition to account for the issue of statistical significance.

What does this paper add?
  1. When applying the usual MID criteria to the evaluation of QOL changes in individual subjects, over 50% of the reputedly relevant changes proved to be statistically insignificant in a sample of cancer patients completing the European Organization on Research and Treatment of Cancer Quality of Life Core Questionnaire.

  2. Similar results are to be expected with other samples and instruments, as the above finding is mainly a consequence of the instrument's reliability.

  3. A substantial increase of the threshold for “relevant change” is proposed to resolve this problem.

Implications—what should change now?
  1. Incorporation of the concept of statistical significance (type one error) appears essential when defining relevant change for individual subjects. Additional inclusion of the type two error would be desirable, but to allow for this more emphasis should be placed on the validation of criteria assessing the clinical significance of QOL changes.

Although in the past two decades quality of life (QOL) research was dominated by studies involving groups of patients, there has been a growing interest in the individual patient and his or her QOL in recent years. This shift of focus was probably fostered by the new possibilities computer technology offers for data collection, evaluation, and visualization [1], [2], [3]. These new, user-friendly computer systems with data entry by the patient allow the routine assessment and monitoring of patient-reported outcomes, including QOL, not only in the highly equipped setting of a clinical trial but also in the daily clinical practice where the focus usually lies on the individual patient [4].

To benefit from the new possibilities, physicians and other medical staff need a certain amount of guidance. This does not only apply for the practical use of the computer system but likewise for the interpretation of the QOL data which, in contrast to clinical variables, are not very familiar to most medical users [5]. One possibility is to provide cut-off values for QOL scores, for example, “within normal range” and “below normal range.” Although such benchmarks may be helpful for first orientation, the large interindividual variation of QOL scores limits the use of such a classification. However, as QOL data are usually assessed repeatedly over time, an evaluation of an individual's QOL need not be confined to the actual QOL scores but can also make use of the time course. In particular, an evaluation of differences of QOL scores at adjacent time points is of interest. These changes may be classified according to their size and often a partition into three categories will be sufficient for practical purposes: no relevant change, relevant improvement, and relevant deterioration.

Various criteria for defining clinically relevant change have been proposed. These may be broadly classified into two groups: anchor-based and distribution-based criteria [6], [7], [8]. Anchor-based criteria make use of additional information obtained from the patient (who is asked to rate the relevance of perceived changes in health status or well-being) or from the clinical context (e.g., a laboratory or physiological measure or a rating by the clinician). Comparison of this information with the actually measured differences on the QOL scale allows estimation of the so-called minimal important difference (MID) [9] or minimal important change, the threshold to be used to distinguish relevant changes from irrelevant ones. Distribution-based methods, on the other hand, use the statistical distribution of the QOL data to derive threshold values for “relevant” changes. Probably the most often used distribution-based change measures are multiples of the standard deviation (SD). Measures involving the scale reliability, such as the standard error of measurement (SEM), are also referred to as distribution-based criteria. However, as this type of measure addresses measurement precision rather than clinical relevance its appropriateness as a criterion of relevant change is doubtful [10].

MIDs have been derived for a variety of QOL measures and diagnostic groups, often by combining anchor-based and distribution-based methods. In oncology, MIDs have been established for various instruments of the FACIT system [7], [11], [12] and for the European Organization on Research and Treatment of Cancer Quality of Life Core Questionnaire (EORTC QLQ-C30) [13], [14], among others. Recently efforts have been made to unify the rather confusing multitude of MIDs and two distribution-based parameters have been suggested as rather crude, pragmatic benchmarks: 1/2 SD [15] and 10% of the range of a scale [16].

The change criteria described above and the resulting recommendations for MIDs have been developed primarily for patient groups rather than individual patients. Although some researchers have pointed out that these criteria will basically retain their validity when being used with individual subjects [17], [18], others stress the differences between the two types of application [6], [19]. In particular, although for patient groups the sample size can, by an appropriate power analysis, be determined in such a way that clinical relevance of a change also entails statistical significance, this is not possible in the case of an individual subject. Rather in this case, the threshold for a significant change is entirely determined by the scale reliability or, more exactly, its SEM. Hence clinical relevance, as defined above, does not imply statistical significance and consequently the change criteria described above should not simply be transferred to the case of N = 1 without further consideration.

Up to now, few articles have dealt with criteria suitable for defining clinically relevant QOL changes in the individual subject. An exception is the integrated approach proposed by Crosby et al. [20]. The suggested criterion is defined as the maximum of two thresholds, one governing clinical relevance and the other controlling statistical significance. The first threshold is based on MIDs derived for patient groups, the second is based on significance levels for individual subjects. The authors demonstrated the usefulness of their criterion in a large sample of obesity patients using a QOL scale with rather high reliability (Cronbach's α = 0.965). It remains open how the criterion will perform for QOL instruments with lower reliability, as is usually the case for measures used in oncology and many other fields of application.

Wyrwich et al. [21] also considered the evaluation of intraindividual changes in health-related QOL and suggested the use of one SEM as a criterion of relevant change. The authors validated their criterion by means of anchor-based methods and recommended its use both for individual subjects and groups. However, there is very little protection against type one errors as the criterion is based on a two-sided alpha level of about 0.3 (Z-score of 1).

In summary, there seems to be little consensus about the appropriate criteria to be used for evaluating QOL changes in the individual subject. Moreover, there appears to be limited experience how the suggested criteria work in practice. Also, research with oncological patients and cancer-specific QOL measures regarding this issue is scarce.

To shed some more light on this issue, we collected QOL data longitudinally in a sample of 160 cancer patients, giving rise to a total of 815 QOL changes. We used the EORTC QLQ-C30 [22], one of the most widely used cancer-specific QOL instruments, for the assessment of QOL. Aim of the study was a critical comparison of several formerly suggested criteria of relevant change with regard to their usefulness for evaluating changes in the individual subject, taking into account both clinical relevance and statistical significance. As a consequence of this comparison, we propose a modified change criterion and put it up for discussion.

Section snippets

Sample

The study consecutively included all cancer patients being admitted to chemotherapy treatment at the outpatient unit of the Department of Internal Medicine, Kufstein County Hospital, within a time period of 12 months. Inclusion criteria were an age between 18 and 80 years, German speaking, no overt cognitive impairments, expected survival time of at least 6 months and informed consent. Clinical data were extracted from the medical records.

Assessment of QOL

QOL was assessed by means of the EORTC QLQ-C30. This

Patient characteristics

A total of 187 cancer patients had been entered into the study by the reference date for analysis. Of these, 160 had at least two assessment times and were thus eligible for the evaluation of changes. Their mean age was 62.9 ± 11.8 years (mean ± SD), 52.5% were female. About a quarter of the patients (25.6%) had a diagnosis of lung cancer, 21.9% had breast cancer, 10.6% hematological malignancies, 8.7% colorectal cancer, and 33.1% other types of cancer. A quarter of the patients (26.3%) had

Discussion

The primary goal of this study was a critical inspection of formerly proposed criteria of relevant change in QOL with regard to their use for evaluating changes in the individual subject. The study was based on a sample of cancer outpatients whose QOL was assessed by the EORTC QLQ-C30, mainly for monitoring purposes in the daily clinical practice.

Our main finding was that the usual change criteria developed for samples of patients are apparently no longer appropriate when considering QOL

Acknowledgment

We want to thank Sandra Ehrenstrasser for her contribution to data collection. Furthermore, we thank the “Jubilaeumsfond” of the Austrian National Bank and the “Qualitaetsfoerderprogramm” of the federal state of Tyrol for financial funding of part of the project.

References (30)

  • G. Guyatt et al.

    A critical look at transition ratings

    J Clin Epidemiol

    (2002)
  • D.L. Berry et al.

    Computerized symptom and quality-of-life assessment for patients with cancer part I: development and pilot testing

    Oncol Nurs Forum

    (2004)
  • B. Holzner et al.

    Computerized assessment of quality of life in patients undergoing chemotherapy

    Qual Life Res

    (2004)
  • J.B. Jones et al.

    Issues in the design of Internet-based systems for collecting patient-reported outcomes

    Qual Life Res

    (2007)
  • G. Velikova et al.

    Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial

    J Clin Oncol

    (2004)
  • Cited by (23)

    View all citing articles on Scopus
    View full text