Assessing the construct validity and responsiveness of Preference-Based Measures (PBMs) in cataract surgery patients

The analyses used data from the Predict-CAT study, a cohort study of cataract surgery patients. Eligible participants were aged 50 or over, were able to understand and complete the PBMs, and were approaching either their first cataract surgery on either eye or a second surgery on the fellow eye. Participants were recruited from two NHS trusts (University Hospitals Bristol NHS Foundation Trust and Gloucestershire Hospitals NHS Foundation Trust) in the South West of England at the time of being listed for cataract surgery or at a pre-operative assessment appointment.

Participants attended two study visits, before and after surgery. The post-operative appointment was scheduled to take place 6–8 weeks after cataract surgery, although in practice there was some variation. All participants completed the Cat-PROM5 and ICECAP-O. The Cat-PROM5 is a five-item questionnaire designed to measure the HRQL impact of cataract surgery [22]. The Cat-PROM5 is responsive to changes in vision following cataract surgery (Cohen’s d = –1.45) [22], although is not preference based and thus cannot be used in CUA. It is currently being piloted in the National Ophthalmology Database Cataract Surgery Audit [23], its use having been encouraged by NICE [24]. Cat-PROM5 is comparable or performs better than the widely used CATQUEST-9SF nine item questionnaire [25].

For the remaining measures, participants were randomised in a 1:1:1 allocation to complete either the EQ-5D-3L, EQ-5D-3L + VIS or the EQ-5D-5L. Randomisation used an automated allocation when participants were added to the study database. Data collected also included socio-demographic information, medical history, assessment of visual function, and an ocular examination with pupil dilation.

A description of the measures and their scoring is provided in the Supplementary material. Lower Cat-PROM5 scores reflect better quality of life (QOL), whereas higher PBM scores are better. Both the EQ-5D-5L-CW [9] and EQ-5D-5L-VSE [8] algorithms were used to score the EQ-5D-5L.

Clinical (visual acuity, diabetic status, first or second eye surgery, complications) and socio-demographic characteristics (age, gender) of the sample were summarised. Descriptive statistics were generated for PBM indices at baseline and follow-up. We estimated the proportion of participants scoring the maximum and minimum scores at baseline. A threshold of 15% of was chosen [26] to define potentially problematic ceiling (maximum) or floor (minimum) effects. Large proportions of patients reporting the highest or lowest value at baseline reduces the potential to demonstrate either improvement or decline in condition following cataract surgery.

A PBM is responsive if changes in the index score reflect known changes in health [20]. These changes are defined using external indicators (anchors) of either clinical or patient-reported change, but they must be relevant to the condition. After surgery, patients completed two questions that asked about their perceived benefit of surgery and change in visual QOL. These were appended to the Cat-PROM5 post-operative questionnaire. These response options were used as anchors and participants were categorised into the following groups:

Perceived benefit of surgery

Change in visual QOL

The no change/worsening response options were combined due to few participants reporting worsening [perceived benefit N = 34 (2.81%), visual QOL N = 27 (2.23%)].

Change in visual acuity was used as a clinical anchor. Based on a change in monocular habitual near visual acuity threshold (− 0.2 LogMAR), participants were categorised as either improving or experiencing no change/worsening. This was based on clinical expertise, previous literature [34] and data from the National Ophthalmology Database Audit [23].

For each PBM, change in PBM index was calculated between baseline and follow-up for each patient. Mean difference was compared between groups gaining significant benefit and those not changing by a significant amount/worse off using analysis of covariance (ANCOVA). Covariates included age, gender, diabetic status and complications.

Effect sizes were calculated for each PBM index score. These statistics quantify the difference between pre- and post-surgery scores in standardised units, enabling the comparison of the PBMs. Effect size is the change score divided by the standard deviation at baseline (Cohen’s d). Where no comparative data are available, effect sizes can be interpreted using Cohen’s thresholds [27, 28]. These are 0.20 small change, 0.50 moderate change and 0.80 large change [27]. Effect sizes for participants worsening/experiencing no benefit were expected to be less than those experiencing benefit. An effect size smaller than 0.2 would be expected when no change/worsening occurred.

We examined the construct validity and responsiveness of the PBMs using the properties reported in Brazier et al. [20].

The following criteria were tested.

Descriptive statistics for the PBMs are reported in Table 2. The EQ-5D-3L + VIS had the highest mean PBM index at both timepoints. All mean PBM scores increased between baseline and follow-up. Variability was greatest for the EQ-5D-3L, with the largest standard deviations observed for this measure. Variability for ICECAP-O and EQ-5D-3L + VIS were lowest.

For the EQ-5D-3L, 27.1% (118/435) of participants scored one at baseline (index profile 11111), greatly exceeding the threshold indicating a potentially problematic ceiling effect. The EQ-5D-5L marginally exceeded the 15% threshold also (69/439, 15.7%). Of the generic PBMs, the ICECAP-O had the smallest ceiling effect (123/1308, 9.4%). The EQ-5D-3L + VIS was marginally lower (38/436, 8.7%). A high proportion of participants scored 0.961 on the EQ-5D-3L + VIS (74/436, 17.0%). This corresponds to an index profile of 111112 (‘no problems’ on all EQ-5D-3L domains and ‘some problems’ on the vision bolt-on domain). As usually observed, all PBM distributions were negatively skewed, with more participants reporting good health. No participants scored the lowest possible PBM score.

Correlation coefficients between the PBMs were all strong (> 0.5; Table 3). Moderate associations were observed between the Cat-PROM5 and the EQ-5D-3L + VIS, EQ-5D-5L-VSE and the ICECAP-O but not the EQ-5D-3L and EQ-5D-3L-CW. As expected, the correlation coefficient between the vision-specific EQ-5D-3L + VIS and Cat-PROM5 was largest. In regard to correlations between PBMs and visual acuity, the relationships were either small (0.1–0.3 for the EQ-5D-3L + VIS, EQ-5D-5L-VSE and EQ-5D-3L-CW) or insubstantial (< 0.1 for the ICECAP-O and EQ-5D-3L).

For the previous cataract surgery and baseline visual acuity known-groups, the mean differences in PBM scores were small, but in the expected direction (Table 4). For the ocular comorbidities known-group, the mean differences in PBM scores were also small, and not consistently in the expected direction. In almost all analyses the confidence interval spanned zero.

For the improvement in QOL anchor, the mean difference in scores was in the expected direction for all PBMs, but the EQ-5D-5L mean difference was closer to zero and the confidence interval for that PBM included zero (Table 5). All PBMs identified moderate (EQ-5D-3L + VIS) or small (other PBMs) effect sizes in patients who reported QOL improvements. Unexpectedly, there was a small positive effect size for the EQ-5D-3L + VIS in patients who stated that their QOL had not improved.

For the perceived benefit of surgery anchor, the mean difference in scores was again in the expected direction for all PBMs. However, the mean difference was largest for the ICECAP-O and that was the only PBM where the confidence interval excluded zero. All PBMs identified small effect sizes in patients who reported significant benefit from surgery. Again, there was a small positive effect size for the EQ-5D-3L + VIS in patients who stated that they had no significant benefit from surgery.

For the visual acuity anchor, mean differences in PBM scores were close to zero. The EQ-5D-3L + VIS and ICECAP-O identified small-positive effect sizes in patients whose visual acuity improved. For patients with little or no improvement in visual acuity, effect sizes were similar across all PBMs.

The EQ-5D-3L and EQ-5D-5L did not perform well across almost every measure of validity and responsiveness and had the largest ceiling effects (Table 6). The EQ-5D-3L + VIS had a lower ceiling effect and better convergent validity with the Cat-PROM5. It was able to differentiate between patient groups who did and did not report benefit from surgery and improved visual QOL after surgery. However, it also identified small positive effect sizes in patients who reported no benefit or no improved visual QOL after surgery. The ICECAP-O also had a low ceiling effect and there was some evidence of convergent validity with the Cat-PROM5. It performed best on many measures of responsiveness.

Predict-CAT is a large cohort study that resulted in a detailed dataset describing the patient-reported impact of cataracts before and after surgery. The core EQ-5D measures did not perform well across the tests of validity and responsiveness conducted. There was little evidence that the EQ-5D-5L is more responsive than the EQ-5D-3L. The ICECAP-O was more responsive than the EQ-5D measures to post-operative improvements in visual QOL and the perceived benefit of surgery, although the effect sizes were small. None of the PBMs were responsive to changes in visual acuity.

This is the first published use of the ICECAP-O in cataract patients and the first that allows the comparison of the EQ-5D-5L, EQ-5D-3L and the EQ-5D-3L + VIS. This large cohort was mostly representative of UK cataract surgery patients, with a similar median age (75) and baseline visual acuity (0.5 LogMAR) as the UK National Ophthalmology Database Audit 2018 [23] (median age 76.3, visual acuity 0.5 LogMAR). The audit data comprised 50% of UK cataract surgeries undertaken in 2017–2018. Whilst the EQ-5D-3L + VIS has not been used extensively, there is ongoing interest in the development of EQ-5D bolt-ons to fill perceived gaps in the core measures [35]. There is also considerable debate about which EQ-5D version and scoring algorithm should be used to measure self-reported health [36] and to inform decision-making [10]. Another strength is the patient-reported and objective measures of change in visual acuity collected in the study. It could be argued that patient perceived benefits are the outcome that should be targeted, as these might not correspond directly to clinical change. This study was able to test the responsiveness of the PBMs to both of these outcomes, replicating findings that visual acuity is not associated with generic PBMs [37‐39].

A limitation of the study is that the three versions of the EQ-5D questionnaire were completed by different patient cohorts. If participants were to have completed every questionnaire, response burden would have been excessive. These cohorts were randomly assigned, relatively large and had similar baseline characteristics, nevertheless it is possible that some observed differences in validity and responsiveness might be due to chance. In addition, the study comprised cataract patients only. All participants had surgery, so experienced some change in clinical condition. Including a control group of cataract patients on the waiting list for surgery might have provided a more robust assessment of responsiveness. When evaluating the PBM performance, judgements were made on a series of thresholds and statistical tests. In some cases, decisions were based on marginal results. Whilst using arbitrary cut-offs is perhaps crude, decisions made on the triangulation of evidence is also subjective [40].

The EQ-5D-3L ceiling effect at baseline was larger than previous studies in cataract patients 19.3% [41] 23% [42]), although not as pronounced as that observed by Gandhi et al. [18] (51%). The EQ-5D-5L ceiling effect reported by Gandhi et al. [18] (46%) also exceed the Predict-CAT results. Gandhi et al. [18] reported the performance of the EQ-5D-5L and EQ-5D-3L in a small cohort (n = 148) of cataract surgery patients and similarly found the EQ-5D measures to be inferior to alternative PBMs (HUI3 and SF-6D). The EQ-5D-5L was scored using four algorithms. Gandhi et al. [18] concluded the EQ-5D-5L is the preferred version due to its superior responsiveness (irrespective of scoring algorithm), however, our study does not support this. Gandhi et al. [18] did not examine responsiveness in relation to change in either patient-reported or objectively measured vision. Comparing the EQ-5D-5L scoring algorithms, EQ-5D-5L-VSE utilities were greater than the EQ-5D-5L-CW obtained values in our study. This is consistent with published comparisons [43]. Despite the addition of two more response categories, the five-level version showed no advantage over the three level. Neither version was consistently better when examining responsiveness, which is compatible with the mixed evidence available thus far [36].

This is the first study to administer the EQ-5D-3L, EQ-5D-3L + VIS, EQ-5D-5L and ICECAP-O concurrently and in a longitudinal study. Furthermore, the two available EQ-5D-5L scoring algorithms were applied [8, 9]. The addition of the vision bolt-on appears to improve the responsiveness of the EQ-5D-3L in this patient population, however it also seems responsive to the process of surgery in the absence of benefit. It was also the only EQ-5D variant to discriminate between participants with poorer and good visual acuity. The anchors used to measure patient-reported change required a reflection on their condition pre-surgery. This may introduce recall bias. In addition, assessing responsiveness as the difference between two assessments of your ‘health today’ perhaps does not reflect the change attributable to surgery. Complications or other negative experiences might be missed for example.

The poor association between vision and HRQL highlights challenges interpreting and appraising evidence of construct validity and responsiveness of PBMs. Firstly, PBMs have been valued by the general population, but clinical outcomes and other patient-reported outcomes are not. Associations between these measures might be improbable as a result. Furthermore, PBMs measure aspects of health and wellbeing unrelated to the condition. They are therefore not intended to be strongly associated with condition specific measures or clinical outcomes. Irrespective of this, there are certain properties that we would expect of a PBM. These include a small ceiling effect among a group of patients seeking care for visual problems affecting their QOL and being able to differentiate between patients who do and do not report improved QOL after a procedure of proven effectiveness, like cataract removal. The problems are probably related, with a high ceiling effects for the EQ-5D-5L and EQ-5D-3L leading to a lack of responsiveness.

It seems that the EQ-5D-3L or EQ-5D-5L should not be the sole PBMs used in studies evaluating the cost-effectiveness of cataract surgery. This analysis has revealed evidence of limited responsiveness and poor construct validity in both EQ-5D-3L and EQ-5D-5L amongst cataract patients. This should be reflected in interpretations of cost-effectiveness analysis of interventions in cataract surgery patients. There is currently no evidence of the ICECAP-O’s content validity in this patient group. This was not within the remit of this study, but future qualitative work could be conducted to explore this. Future work could explore the suitability and performance of the ICECAP-A [44] in cataract patients given the potential importance of capability wellbeing in this context. The ICECAP-A measures capability wellbeing in adults as opposed to the focus on older adults in the ICECAP-O. Developed using qualitative research with UK adults, the five domains cover similar capabilities to the ICECAP-O, with some items reworded or the focus changed.

Whilst the ICECAP-O seems to be more responsive than the EQ-5D in cataract surgery patients, it cannot be used to generate QALYs. Yet without further methodological developments, neither can the EQ-5D-3L + VIS. There is no five-level vision bolt-on available, meaning a revised bolt-on is required, with concurrent robust valuation and validation. The methodological rigour and resources required to develop and value a bolt-on item challenges the feasibility of developing one for every condition that the EQ-5D is reportedly unsuitable. The endorsement of the EQ-5D for use in economic evaluation is largely justified by the need for a comparable measure of benefit. Bolt-ons lack comparability with core EQ-5D scores, however. Developing measures that are sufficiently broad to measure health-related wellbeing in all common conditions, without the need for bolt-ons should be prioritised. Finally, the ability to conclude what the best PBM is in cataract surgery patients should be informed by evidence comparing all PBMs available, such as the SF-6D [45] and HUI [46].