Test–retest reliability of the EQ-5D-5L and the reworded QOLIBRI-OS in the general population of Italy, the Netherlands, and the United Kingdom

The sample consists of 1864 persons of Italy, the Netherlands, and the UK. The data were collected as part of the Collaborative European Neuro Trauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study [17]. The participants were recruited by an international market research agency from existing large internet panels.

The data were collected at two time points within 6 months. At the first time point (T1), a sample that represented the general population by age (18–75), gender, and education in the selected countries, was asked to fill out the web-based questionnaire (from 29th June till 31st July 2017). At the second time point (T2), a random subset of the T1 sample respondents was re-contacted (from 3rd until16th February 2018).

Panel participation was on a volunteer basis, once participating the data capture system did not allow skipping questions. Participants received an incentive in the form of cash or points from the market research company.

We selected respondents who experienced no change of chronic conditions between T1 and T2 and assumed that represented no real changes in their health state.

Spearman correlation was used to test the correlation between reworded QOLIBRI-OS items and EQ-5D-5L dimensions. It was hypothesized that similar domains measuring similar items of HRQoL should be strongly correlated (> 0.5) [19].

Bland–Altman (BA) plots were used to visually examine the test–retest agreements for EQ-5D-5L summary index, the EQ VAS, and reworded QOLIBRI-OS level sum score. A BA plot shows whether there are significant systematic differences between T1 and T2, for example, whether the second measurement constantly under- or overestimates compared to the first one.

Test–retest reliability of both EQ-5D-5L and reworded QOLIBRI-OS dimensions/items scores were calculated using Gwet’s AC2 test. Gwet’s AC, proposed by Gwet [20] corrects the chance-agreement probability so that it is consistent with the inclination of random rating from the observed rating. Gwet’s AC1 is for use with nominal data and Gwet’s AC2 is for use with ordinal data. Gwet’s AC addresses the paradoxical behavior of Cohen’s Kappa where low Kappa is coupled with a high agreement. Because Kappa cannot reflect real agreement with very high (or low) trait prevalence [21, 22]. Health data of the general population typically are extremely unbalanced with a high prevalence of traits such as “no problem”. Radical and linear weights are selected to calculate Gwet’s AC2 for EQ-5D-5L and reworded QOLIBRI-OS dimensions/items [23]. We consider Gwet’s AC2 higher than 0.8, between 0.8 and 0.6, between 0.6 and 0.4, between 0.4 and 0.2, and lower than 0.2, to be indicative of almost perfect, substantial, moderate, fair, and slight to none agreement [24]. The percentage of agreement between test and retest was also calculated.

Test–retest reliability of the EQ-5D-5L summary index, the EQ VAS, and the reworded QOLIBRI-OS level sum score were calculated by Intraclass Correlation Coefficient (ICC, two-way random effects, absolute agreement). Because ICC prefers data normality and stable variance [25, 26], our data were transformed using Tukey’s “Ladder of Powers” to acquire near-normality [27]. As an alternative to ICC, Lin’s Concordance Correlation Coefficient (CCC) was calculated to approximate data normality [28]. We consider ICC and CCC values higher than 0.9, between 0.9 and 0.75, between 0.75 and 0.5, and lower than 0.5 are respectively indicative of excellent, good, moderate, and poor reliability [29].

Z test [30] after Fisher’s r-to-z transformation was used to compare Gwet’s AC, ICC, and CCC between countries. The significant level was set at 0.05.

All statistical analyses were carried out using R version 3.6.3.

1864 respondents completed the T1 and T2 questionnaire, of which 1171 (62.8%) respondents (IT: 377, NL: 390, UK: 404) reported no change of health status and were included in the analysis.

Of these 1171 respondents, 71 people did not have corresponding answers with regards to gender at T1 and T2, had more than two years difference of age, and/or level of education backward from T1 to T2. Twenty-two respondents spent less than a minute to finish the entire questionnaire, which we consider unrealistically short. Chi-square tests between the 93 and the rest of respondents in each instrument suggest no significant difference in the distribution of EQ-5D-5L and reworded QOLIBRI-OS domains (Supplementary file 1), we thus did not exclude these 93 respondents in our analysis and assumed that information at T1 is correct. Characteristics of the respondents at T1 are shown in Table 1. There are significant differences between countries in the distribution of age, level of education, occupational status, and level of income.

Summary distributions of the EQ-5D-5L dimensions, the EQ VAS, the EQ-5D-5L summary index, and the reworded QOLIBRI-OS items are shown in Fig. 1. Overall, 38%, 46%, and 47% of respondents respectively in Italy, the Netherlands, and the UK reported a perfect health state (profile “11,111” for EQ-5D-5L) at T1; at T2 the correspondent percentages were 35%, 48%, and 44%. Median (IQR) EQ-5D-5L summary index scores were 0.88(0.23), 0.89(0.19), 0.88(0.23) respectively in Italy, the Netherlands, and the UK at T1, and 0.84(0.23), 0.91(0.21), 0.85(0.23) respectively at T2. 63%, 63%, and 54% of respondents, respectively in Italy, the Netherlands, and the UK reported EQ VAS higher than 80 at T1, at T2 the correspondent percentage were 62%, 59%, and 55%. Median EQ VAS were 80, 81, and 80, respectively in Italy, the Netherlands, and the UK at both T1 and T2.

With regard to the reworded QOLIBRI-OS, less than 10% of the respondents in each country reported being very satisfied with their health (“555,555”) at both T1 and T2 (5%, 8%, and 8%, respectively, in Italy, the Netherlands, and the UK at T1; 4%, 10%, and 9% at T2). Median (IQR) reworded QOLIBRI-OS level sum score was 22.0(6), 24.0(6), 22.5(8), respectively in Italy, the Netherlands, and the UK at T1, and 22.0(5), 24.0(5), 23.0(8) at T2, respectively.

Among all respondents with a perfect EQ-5D-5L health state (“11,111”), less than 20% responded being very satisfied on the reworded QOLIBRI-OS (15% at T1 and 17% at T2). Spearman correlation showed strong correlations between EQ-5D-5L index and reworded QOLIBRI-OS level sum score (0.52 at T1 and 0.53 at T2) and between domain “Anxiety/Depression” of the EQ-5D-5L and “Emotion” of the reworded QOLIBRI-OS (0.51 at both T1 and T2.). However, for the domains that are most comparable between the two instruments (“Usual activities” of the EQ-5D-5L and “Function in daily life” of the reworded QOLIBRI-OS)—the Spearman correlation did not show strong correlations between the responses on these domains (0.45 at both T1 and T2).

BA plots for the EQ VAS, the EQ-5D-5L summary index, and reworded QOLIBRI-OS level sum score (Fig. 2) suggested no significant systematic differences between T1 and T2 and permit further examinations.

Table 2 presents the test–retest reliability results for the EQ-5D-5L dimensions, the EQ-5D-5L summary index, the EQ VAS, the reworded QOLIBRI-OS items, and the reworded QOLIBRI-OS level sum score.

Gwet’s AC2 for the EQ-5D-5L dimensions was the highest in Self-care (> 0.9) in all three countries, followed by Mobility and Usual activity in Italy, showing almost perfect reliability in these dimensions. Gwet’s AC2 in Pain/Discomfort and Anxiety/Depression were lowest, showing substantial reliability. The ICC and the CCC for the EQ-5D-5L summary index and the EQ VAS respectively are fairly similar. They range from 0.73 to 0.84 for the EQ-5D-5L summary index and 0.61 to 0.68 for EQ VAS, showing moderate to good reliability in the three countries.

Gwet’s AC2 for the reworded QOLIBRI-OS items were lower than Gwet’s AC2 for the EQ-5D-5L dimensions, ranging from 0.39 to 0.63, suggesting substantial to fair reliability in different items. The ICC and the CCC range from 0.69 to 0.78, showing moderate to good reliability in the three countries.

Between countries, there were significant differences in the test–retest reliability in most domains of EQ-5D-5L, EQ-5D-5L summary index, EQ VAS, and reworded QOLIBRI-OS. (Supplementary file 2).

A strength of our study was that we adopted Gwet’s AC to access test–retest reliability. Compared to the commonly used Kappa coefficient, Gwet’s AC does not depend on the assumption of normal distribution and independence between raters, and it adjusts the chance-agreement probability when trait distribution is unbalanced. It offers a better estimation of test–retest reliability in the general population where trait distribution is extremely unbalanced. A second strength was that we compared the test–retest reliability of EQ-5D-5L and reworded QOLIBRI-OS in the general population of three countries.

Our study has some limitations too. First, the 93 (8%) respondents that had non-corresponding answers on gender, age, and education between T1 and T2 and/or spent less than 1 min to finish the questionnaire were included in the analysis. Including these respondents in the study might have affected the reliability of the results, even though we found no significant difference in the distribution of their responses on the EQ-5D-5L, and reworded QOLIBRI-OS compared to the rest. Second, the data were collected through the internet. The disadvantages related to web-based questionnaires include sampling issues and response rate [36]. Thirdly, the time interval between T1 and T2 in our studies was six months, which could allow for true health changes to occur [37]. Moreover, season variations have an impact on physical and mental health of a population [38]. These factors may have affected the test–retest reliability of the EQ-5D-5L, and reworded QOLIBRI-OS found in our study, they could also explain the significant differences of test–retest reliability between countries. Two weeks is generally considered the most appropriate time interval to investigate test–retest reliability [39], but no evidence has been found against longer time intervals. We sampled from the general population which is generally healthy and stable, and we excluded respondents who experienced a change in chronic health status from T1 to T2. We, therefore, believe that the time interval used in our study does not jeopardize the results on test–retest reliability. Finally, the use of monetary incentives may decrease response bias by design, but it may also increase response bias when accepting an incentive is correlated with education [40] and race [41]. In our study, the sample at T1 is designed to represent the socio-demographic of the general population. We are thus less worried about the bias induced by monetary incentives.

It is also worth noticing that using the EQ-5D-5L summary index and the reworded QOLIBRI-OS level sum score comes with some limitations too. Firstly, we used the value set for EQ-5D-5L of the UK for Italy because that of Italy was not available, different results might be produced using a different value set. Secondly, the reworded QOLIBRI-OS level sum score summary score with equal weight of each item. The use of equal weights means that each dimension is given equal importance in value judgement, which is not necessarily the case in different populations. However, no value set is available for the reworded QOLIBRI-OS yet.