The WHOQOL-BREF: a modern psychometric evaluation of its internal construct validity in people with multiple sclerosis

Patients were recruited as part of the Trajectories of Outcomes in Neurological Conditions (TONiC) study investigating predictors of quality of life in chronic neurological disease. Subjects with MS were recruited from 23 centres across the UK and were asked to complete a questionnaire pack which included the WHOQOL-BREF amongst a variety of PROM’s designed to map on to the International Classification of Functioning, Disability and Health (ICF) Brief Core Set for MS [15]. The project aimed to recruit subjects across a broad range of ages, diseases subtypes and disabilities. Patients who were physically unable to complete the questionnaires were permitted to use a scribe to report their answers on the form. All participants received written information and informed consent was obtained from all individual participants included in the study prior to enrolment.

The Rasch model provides a systematic and objective method to evaluate a set of criteria consistent with fundamental measurement, that is the type of measurement normally found in the physical sciences [16]⁠. These criteria include the stochastic (probabilistic) ordering of items, monotonicity (increase in item responses consistent with underlying trait), local item independence (zero correlation between items when conditioned on the score), unidimensionality and group invariance (no difference in response to item by group membership when at the same level of (in this case) QoL [17]. Thus data from a scale are tested against the requirements of the Rasch model in a process widely known as Rasch analysis. Full details of this process are given elsewhere [18].

⁠Recent methodological developments have updated the previously published guidelines. Thus, local item independence was examined by constructing a residual item correlation matrix between all items. The residual of an item is the difference between the estimate of item difficulty given the model, and the observed item difficulty, standardised to a mean of zero and standard deviation of 1. Residual correlations of + 0.2 above the average correlation are considered an indicator of a breach of the local independence criteria [19]⁠. That is, there is something else, other than the trait being measured (e.g. QoL) that is causing an association between the items. This may be due to the fact that the items are close replications of one another, or multidimensionality is present. When local item dependency is observed, it is accommodated through a strategy of grouping items, either along the lines of a priori known domains where the grouping is consistent with a testlet, or from other evidence where the grouping is post hoc, contingent on the analysis, and referred to as ‘super items’ [20, 21]. This latter approach can include two super items, created by allocating alternative items to each super item, on the basis that as the scale (domain) was supposed to be summated into a single score, then alternative items should demonstrate a near perfect latent correlation between the two, while absorbing most, if not all, local dependency within the item set. Analysis using two super items has the advantage of providing a more robust conditional chi-square test of fit, together with the proportion of variance retained in a bi-factor equivalent solution, consistent with the explained common variance (EVC) in the bi-factor literature [22‐24]⁠. A bi-factor solution is where the latent estimate is based upon the first (Rasch) factor, upon which all items load but also load onto secondary factors EVC should be > 0.9 if the scale is to be considered essentially unidimensional, that is greater than 90% of the variance is common and retained in the latent estimate.

The data were further evaluated for differential item functioning (DIF) by age group, gender, MS subtype and duration [25]⁠. Analysis of DIF examines whether each item performs equally across different subgroups, given the same level of the underlying trait. So, contextual characteristics who have the same level of quality of life. Evidence of DIF was sought from graphical displays of group-specific item characteristic curves, and statistically if the p-value derived from an ANOVA analysis was significant at the 5% level with a Bonferroni correction applied. Where a testlet or super-item solution is obtained, and DIF is shown to be still present, the substantive nature of this DIF is tested by comparing unadjusted and adjusted person estimates. Should the t test of this comparison (for paired or repeated measures) be significant, then an effect size of the difference is calculated which should be less than 0.1, in which case DIF is deemed to be small and no action is taken [26]. All analyses were performed using RUMM2030 software [27].

Targeting of each scale was assessed by person-threshold (transition between categories) distribution plots, which plot the range of QoL recorded by subjects against the range measured by each scale. Spearman’s Rho correlation coefficients were calculated to ascertain concurrent validity between the derived scales and the established scales included in the TONiC study: the EQ5D and Leeds MS QoL (LMSQoL) [28, 29]. Finally, the precision of the various domains of the WHOQOL-BREF was examined with respect to the standard error of measurement (SEM) and the smallest detectable difference (SDD) [30, 31]⁠.

Thus, the properties of the WHOQOL-BREF were first examined according to the original structure of physical, psychological, environment and social health subscales, then two new domains were considered reflecting the stems of the items, that is ‘impact’ and ‘satisfaction’ and finally, the data were examined for a 24-item total solution. Where a solution was identified to fit the assumptions of the Rasch model, a conversion table was calculated to enable transformation of the original ordinal scores to an interval scale without the use of specialist software.

3186 people with MS were recruited into the study by mid-2017 and had returned the baseline questionnaire. Mean age was 49.8 years (SD 11.8) and mean duration of MS (since diagnosis) was 11.2 years (SD9.6). Almost three quarters (73.2%) were female. Three fifths (60.3%) had a Relapsing Remitting form of MS (RRMS), 11.7% Primary Progressive; 22.9% Secondary Progressive, and 5.1% a rapidly evolving form of RRMS. Almost half (49.8%) had an EDSS of 0–4.0; 38.5% were at level 4.5–6.5; 6.75% at level 7–7.5 and 4.9% had an EDSS ≥ 8.0.

Data from the WHOQOL-BREF were fit to the Rasch model in a series of analyses representing different grouping of items shown in Table 1, ideal values for fit statistics derived from Rasch analysis literature are provided within the table [19, 32]. The requirement for local independence of items was found to be breached for all groups. For example, in the 24-item-based scale, the items ‘How much do you enjoy life’ and ‘To what extent do you feel your life to be meaningful’, had a residual correlation of 0.481. For the physical subscale, whose average residual correlation was − 0.16, the items ‘How well are you able to get around’ and ‘How satisfied are you with your ability to perform your daily activities’ had a residual correlation of 0.198. As a consequence, in practice all the item groups (domains) were resolved into two super items.

The 7-item physical subscale (Table 1, Analysis 1) showed significant misfit to the model, including a breach of the local independence solution, multidimensionality, and substantial DIF by age, duration and disease subtype.

All thresholds were ordered, supporting the monotonicity requirement. The two ‘super-item’ approach resolved the fit, with the two latent estimates perfectly correlated, and the explained common variance at 1.0, indicating that no unique variance had been discarded to achieve a unidimensional latent estimate which was confirmed by the t test (Analysis 2). However, DIF remained. Graphically it was hard to determine where the DIF was present, but the MS subtype showed some slight deviation for rapidly evolving (RE) RRMS (Fig. 1). Substantive DIF was thus tested by contrasting the estimates from a split solution (RE + the rest) against an unsplit solution, anchored by the split super-item parameters. The effect size of such a contrast was 0.032. As such no further action was taken for DIF on the physical subscale.

The 6-item psychological scale also showed misfit to the model, with local dependencies, multidimensionality and significant DIF (Analysis 3). All item thresholds were ordered. The two super items showed good fit to the model (Analysis 4). While duration was invariant, DIF was present for age, gender and MS subtype. Graphical interpretation was not helpful, but it appeared that females showed the most deviation from the expected curve, and so a super item was split by gender. The effect size for the difference between estimates was 0.031, no further action was taken for DIF and the original unsplit estimate used.

The three-item social relationships scale (Analysis 5) failed to show fit to the Rasch model. With just three items, the test for unidimensionality was underpowered. The reliability of the scale remained below that considered a minimum for group use. As such, no resolution was attempted.

The original 8 item environment scale (Analysis 6) showed misfit to the model, multidimensionality and DIF by age and disease subtype. The two super items demonstrated adequate fit within a bi-factor equivalent solution, having shed some 4% of the variance to achieve a unidimensional latent estimate (Analysis 7). DIF was absent for all items.

It is also possible to view the 24-item set from a different perspective, based upon the format of the stem of each question. In this way two separate scales assessing impact (items 3–15 and 26) and satisfaction (items 16–25) may be considered. The 14-item impact scale (Analysis 8) did not fit the Rasch model. While all thresholds were ordered, the scale was multidimensional, DIF manifested for all contextual factors, and local item dependencies were present. A two super-item approach resolved the issue (Analysis 9). While DIF was present for age and disease subtype, no variation was observed from the graphical interpretation.

A similar result as found for the 10-item satisfaction scale. Initially the data did not fit the model and showed evidence of multidimensionality, local item dependencies, two disordered thresholds and DIF on all contextual factors. (Analysis 10). Combining items into two super items resulted in satisfactory fit and unidimensionality, having discarded none of the variance (Analysis 11). DIF was still in evidence for age and disease subtype, although it was almost impossible to visualize any difference in the group-specific item characteristic curves (e.g. see Fig. 2) and, given earlier findings and their magnitude of effect size observed, no further action was taken. The targeting of the satisfaction domain was good, showing a slightly higher level of satisfaction than the average of the scale, which itself showed a near perfect distribution across the trait (Fig. 3).

The 24-item scale showed considerable misfit, multidimensionality, local dependencies, and DIF across all contextual factors, often for several items (Analysis 12). Two strategies were considered to see if a total score was valid: a testlet strategy based upon the four domains, and a strategy based upon alternative items allocated to two super items.

Creating 4 testlets according the 4 original subscales failed to resolve fit to the model (Analysis 13). The psychological and social testlets were dependent, DIF was present for all contextual factors, and 8% of the variance had to be discarded in order to achieve a unidimensional latent estimate (although adequate under a bi-factor solution).

In contrast, the two super-item analysis displayed near perfect fit, with excellent reliability, and none of the variance was discarded to obtain the unidimensional latent estimate (Analysis 14). DIF remained on all but the duration factor, but here the sample size was influencing small group differences that were not substantive. For example, splitting the youngest age off in one super item, and comparing unsplit with (Anchored) split age gave an effect size for the difference in person estimates of 0.05.

Transformations of the ordinal raw score of the above scales to interval scale latent metrics are available in Table 2.

The SEM and SDD of each domain are shown in Table 3. The original domains (excluding social which has insufficient reliability), require a difference between two groups/times of the order of 16–18% of their operational ranges to overcome the error, whereas, in descending order, the satisfaction, impact and total (24-item) domains require between 14 and 8% of their operational ranges to be above error.

While all one-way ANOVA’s showed a significance level of < 0.001 for discrimination of each domain across subtype, the non-discriminating pairwise results from the post hoc Bonferroni provide some insight (Table 4). Generally, the significant difference was driven by the contrast between Relapsing Remitting, and other subtypes. None of the domains showed a post hoc significant difference between Primary and Secondary Progressive subtypes. The only post hoc significance on the psychological domain was between Primary Progressive and Relapsing Remitting, yet the effect size of this difference was just 0.23. For the physical domain, the effect size for the difference between Primary Progressive and Relapsing Remitting was 0.54.

Correlations between the derived scales and existing subscales included in the TONiC study are shown in Table 5, all correlations were found to be significant with a p-value of < 0.001.