Influence of elicitation procedure and phrasing on health state valuations in experience-based time trade-off tasks among diabetes patients in China

The study was conducted among Chinese adults (18 years and above) diagnosed with type 2 diabetes. Participants were consecutively recruited and allocated to one of two TTO tasks on an alternating basis by letting every second participant respond to an open-ended task and every second participant respond to an iteration-based task. The sample size of 80 participants in each task was decided based on the results of a previous study investigating the difference between the open-ended and the iteration-based TTO valuation tasks, where the respondents valued hypothetical health states [28]. To detect a statistically significant mean difference of 0.1, with a standard deviation (SD) of 0.18 in each group, a minimum of 50 respondents in each groups would be needed (α = 0.05 and 1 − β = 0.8).

Individual face-to-face interviews were conducted in March 2016 at the departments of endocrinology of the public hospitals Jiangning Hospital and Nanjing 1st Hospital located in Jiangning and Yuhuatai districts, Nanjing City, Jiangsu province, China. In Jiangning Hospital (a district county hospital), most patients came from the local area, whereas the Nanjing 1st Hospital (a city-level hospital) had patients from different areas of Jiangsu province and also other parts of the country. Patients represent mixed socioeconomic compositions.

Semi-structured interview questionnaires for open-ended and iteration-based TTO tasks were developed and pilot tested in English and then translated into Chinese and further pilot tested. In both interview questionnaires, the interview started by collecting information about the respondents’ age, sex, duration of diabetes, type of treatment, and comorbidities by asking if the respondent has any other disease diagnosed by a doctor. Information about ethnic group, marital status, having children, education, occupation, and additional comments from the respondent were collected at the end of the interview.

First, the respondents were asked to answer a self-rated health (SRH) question framed as “How is your health today? Is it ‘very good’, ‘good’, ‘fair’, ‘bad’ or ‘very bad’?” [29]. Thereafter, each respondent classified their own health status in five dimensions (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression) and five severity levels for each dimension by answering the EQ-5D-5L instrument [30]. The respondents also reported their own overall health status on a visual analogue scale (EQ VAS) where 100 represents best imaginable health state and 0 represents worst imaginable health state. Next, the respondents completed the TTO task. Finally, the level of difficulty in understanding the task was also assessed through this question “How would you rate the difficulty in understanding this task? Would you say ‘very easy’, ‘easy’, ‘neither easy nor difficult’, ‘difficult’ or ‘very difficult’?”

Pilot interviews were conducted by the first author at Karolinska Institutet in Stockholm, in Beijing and at Nanjing Medical University (NMU) in January and February 2016. The first author and two other interviewers from NMU, trained by the first author, conducted the interviews.

Prior to the interview, participants were informed about the purpose of the study, that participation in the study was voluntary and anonymous, that they were free to leave at any time and that their response would have no impact on the health care they receive. Verbal informed consent was obtained from each participant in accordance with the ethical standards of the institutional research committee Nanjing Medical University Ethics Committee.

Data was entered by all interviewers, and the responses to the open-ended questions were translated to English. The data was cross-checked by the first author for quality control. To ensure the confidentiality of personal information, respondents’ names, and contact information were not collected. Ethical approval was obtained by Nanjing Medical University Ethics Committee (2016; # 254).

The trade-off was between x years in full health and 10 years in the respondent’s current health state. The individual TTO value was calculated by dividing the number of years in full health (x) by the number of years in the current health state. Worse health was implied by a lower x value (i.e., a shorter period of time in full health was equivalent to 10 years in the present state). In the first analysis, the original open-ended and the iteration-based TTO tasks were included. In an additional analysis, the modified open-ended TTO task was included. This allowed us to compare two effects in relation to the iteration-based procedure: the effect of both procedure and phrasing, and the effect of procedure only. Dummy variables were created to represent the type of TTO task in the regression analyses with the iteration-based task as reference.

We employed multiple linear regression, using ordinary least squares (OLS), to investigate the association between the type of TTO task and the TTO values when controlling for other variables. To obtain the akaike information criterion (AIC) from SPSS we also fitted the same model, using maximum likelihood estimation (MLE), i.e. a generalized linear model with a normal probability distribution and identity link function.

The covariates included in the analyses were age, sex, EQ VAS score, socioeconomic status, and interviewer (Table 1). The first author was selected as reference (Interviewer 1) since this interviewer conducted more pilot interviews and trained the other interviewers. The EQ VAS score was used to control for the respondents’ subjective overall health since it was expected that respondents with more severe health problems would be more likely to trade off years, and have lower TTO values. Socioeconomic status included educational level (below primary school as reference) and occupational status (nonemployed as reference). Outliers were identified by plotting the relationship between TTO values and EQ VAS score, TTO values and SRH levels, as well as EQ VAS score and SRH levels. Adjusted R² and AIC were employed to test the goodness-of-fit of the models; the larger the adjusted R² and the lower the AIC, the better the goodness-of-fit of the model. Since the residual plot and Breusch–Pagan test both indicated heteroscedastic residuals, White’s robust standard errors were applied to correct for heteroscedasticity [32]. Statistical analyses were carried out by SPSS Statistics version 23.

The level of difficulty in understanding the TTO tasks was investigated by binary logistic regression. Those who answered difficult or very difficult were coded as having difficulties in understanding the task. The type of TTO task, interviewer, and educational level were selected as explanatory variables. Odds ratios (OR) representing the odds for that the TTO task would be indicated as difficult was examined. The Hosmer and Lemeshow test was employed to check the goodness-of-fit of the model, with nonsignificance of the test representing a good model fit.

Calculations of the mean number of years (SD) that the participants were willing to trade off showed 3.04 years (2.76) in the iteration-based task, 1.26 years (1.78) in the open-ended task and 2.61 years (2.37) in the modified open-ended task.

Without controlling for other variables, the mean TTO value was significantly higher (0.162) for respondents participating in the open-ended TTO task compared to those participating in the iteration-based TTO task (Table 3, Model 1). These results were robust against controlling for other variables in the regression models with coefficients between 0.162 and 0.173 (Table 3, Models 1–7). There were no statistically significant effects on age, sex and interviewer. When controlling for EQ VAS score, a higher EQ VAS score was associated with a significantly lower willingness to trade off years in the TTO task (Table 3, Model 4). When controlling for clinical variables, no statistically significant effects were found (data not shown). There were no systematic differences between educational groups in TTO values. However, those with primary school had significantly higher TTO values compared to those with below primary school as their highest educational level (Table 3, Models 5 and 6). Those employed had statistically significantly lower TTO values than those not employed (Table 3, Model 6). Age, sex, EQ VAS score, socioeconomic status, and interviewer affected the main effect coefficient modestly and increased the R² from 0.116 to 0.186, while AIC dropped from − 27.145 to − 28.651, indicating a better goodness-of-fit of the model.

With and without controlling for other variables, there was no statistically significant difference in the TTO value between the respondents to the modified open-ended and the iteration-based TTO tasks (Table 4, Models 1–7). However, there was a significant difference in the TTO value between the respondents to the original open-ended TTO task and to the modified open-ended TTO task (0.113 higher TTO values with the original TTO task compared to the modified task in the full model; data not shown). The effect of controlling for other variables in the models was similar to the effect in the first analysis.

In total, about 15% (n = 193) of the respondents indicated that the TTO tasks were difficult to understand. However, there were no statistically significant differences regarding the odds of reporting the task as difficult between the groups responding to the different TTO tasks (Table 5). When comparing respondents with different educational levels, those with the highest education level were significantly less likely to report difficulties with understanding the task compared to those with the lowest educational level (in Model 1 with the open-ended and iteration-based tasks). Those who were interviewed by Interviewer 3 were significantly less likely to report difficulties (Model 1). When entering the modified open-ended task, there was no difference between the interviewers (Model 2). In these analyses regarding the difficulties in understanding, the four outliers were included.

There are several potential explanations for our results. One possible explanation for the higher values elicited from the open-ended TTO task is loss aversion, which refers to the concept of people giving more weight to losses than to gains [27, 34, 35] and thus being more reluctant to give up life years in the TTO task than they are willing to gain higher quality of life for the lesser number of years. The effect of loss aversion has been suggested to be stronger in the open-ended task than in the iteration-based task, since the open-ended task puts more emphasis on the fact that something has to be given up [28]. Another possible explanation for the differences in values brought up in the literature refers to the phenomenon of scale compatibility, the idea being that responders are partly steered by what is the dominating scale. Compared to the iteration-based task, respondents in the open-ended TTO task have been seen to give more weight to the attribute of time than the potential improvement of health, since the response scale is more prominent in the open-ended task [28].

Common for both these explanations of the difference in outcome when eliciting TTO tasks in different ways is that they refer to certain differences in how the tasks are described. While there are no inherent differences regarding the core valuation of the two elicitation procedures of the TTO task—valuing a state of less than full health compared to one with full health, where the valuation is expressed in terms of a time trade-off—there may nevertheless be differences in how this has been described that might influence the valuations. Differences may concern how the task is introduced and described in words, but also how it is visually presented. Visual presentation, in turn, may vary both in terms of how scales are stressed (loss of life, gain in health) and how the very comparison is graphically illustrated (including variations in use of color). For instance, in our study the original open-ended TTO task was visualized by a single scale while the iteration-based TTO task was presented by the use of two bars, presented as Life A and Life B, marked with different numbers of years (Fig. 1). The iteration-based task started with the bar representing years in full health at 5 years (Life A), and the one representing the respondent’s present health at 10 years (Life B). Perhaps it comes more naturally to think of the single scale as representing one’s own life, while the two bars can more easily be perceived as hypothetical alternatives, thus somewhat shifting the choice from being about one’s own life to being about two hypothetical lives. As the modified open-ended TTO task was also described with two bars, even this task might have been perceived as hypothetical, which in turn might explain the similarity between this and the iteration-based task. Such perception can, of course, be strengthened or weakened depending on how the choice situation is described in words. We suggest that differences in how the tasks were phrased and visually presented in our study may have influenced the results [36].

An obvious difference between open-ended and iteration-based TTO tasks relates to the repetitiveness of the iteration-based approach. This may have two effects: on one hand, the iteration may provide a means to step by step target a valuation that accurately reflects the values of the respondents. On the other hand, the respondents might lose some of their resistance to the idea of trading off time, or at least, lost or not, behave in greater accordance with the conditions of the task: to trade off time as an expression of their valuation of the health state. It may also be the case that the open-ended TTO task better accommodates, and accounts for, a genuinely felt resistance to the very conditions of the task. Yet there are other differences between the tasks since the iteration-based task emphasizes the respondent’s current health state TODAY, whereas in the open-ended tasks, TODAY is not mentioned. This means that slightly different recall periods might be considered in the different tasks—say, the present day compared to this week or this month. However, all respondents were at the beginning of the interviews instructed to think about their health TODAY as EQ-5D-5L and EQ VAS have this recall period, so with this background the discrepancy between the two elicitation procedures is reduced. We have used formulations of questions currently used in other studies, and we cannot know the impact on this discrepancy in the present study [10, 31].

A strength of this study is that the tasks have been administrated through face-to-face interviews: it provided the possibility for the interviewers to get a better understanding of to what extent the respondents understood the question. It was also possible to identify the potential impact of the wording of the open-ended TTO tasks. However, face-to-face interviews may also present some potential problems related to the presence of an interviewer. For example, it was the impression of the interviewers that some respondents did not want to admit to the interviewer that they had health problems even though this was apparent in their responses to the background questions. This is consistent with previous studies showing that respondents report better health during an interview than in other administration modes [37, 38].

Another strength of the study is that it contained all background variables, except for religious beliefs, suggested by van Nooten et al. [39] for TTO studies. However, the background variables ethnic group, marital status, and having children could not be analyzed due to the lack of variation within these variables. There were no statistically significant differences in background variables between participants in the different tasks, which might be taken to suggest that the study design did not introduce any bias. However, even though we have adjusted for several possible confounders in the regression analysis, there could be other differences between the groups that may have influenced the results.

The interviews were conducted in the wards, which meant that interferences due to routine follow-ups and the visits of relatives could not be avoided. Interferences by relatives wanting to help out were avoided by asking the relatives not to interfere during the interview. Asking questions about diseases before the TTO task might influence the answers. In this study, we asked for how long the respondent had been diagnosed with diabetes and if the respondent had any other disease diagnosed by the doctor as well as the SRH question. We cannot say whether this influenced the willingness to trade off time. Another limitation is that the 37 respondents answering the modified open-ended task were not randomly selected.

In this study, we showed that how the TTO task is phrased and visually presented affects the valuations. However, we made changes both in the phrasing and in the visual presentation for the group that responded to the modified open-ended task, which leaves us no way of telling which factor had the greatest influence on the outcome compared to the original open-ended TTO task. Future studies should investigate this by separating the two modifications to be able to make comparisons.

Some previous studies show that Chinese people are conservative in reporting poor health-related quality of life [40, 41]. Of relevance to the generalizability of our results, it has been suggested that Chinese people have a tendency to be more grateful for life than other populations [40, 42‐44] and more directed at avoiding death (both Taoism and Buddhism are pursuing immortality)—living with poor health is preferred to a good death [42, 45]. Because death, and talking about death, can be seen as taboo in Chinese culture [45], it is possible that Chinese people are more likely to react negatively to the typical way to describe the open-ended TTO question compared to respondents from other cultures. The results can therefore not be simply generalized to other cultures.

In addition, we do not know whether differences similar to the ones found in this face-to-face study would occur in studies using other modes of administration. Further, we cannot tell whether we can generalize from patients with diabetes to other groups of patients.

Generally, the open-ended task has the advantage over the iteration-based as it is less burdensome for respondents, takes less time to complete, can be administrated in postal surveys, and hence costs for data collection will be lower. In the Chinese context, in the aspect of cultural relations to death (trading off years), the modified open-ended task produces as consistent estimates as the iteration-based which is used in the development of value sets for hypothetical EQ-5D-5L states [31]. The iteration-based TTO task has also been shown to be challenging in a nationally representative Chinese population survey used for estimation of a TTO value set for EQ-5D-3L [44].

Further research is needed to explore how the elicitation procedures as well as the phrasing and visual presentation of TTO tasks influence the valuation of experience-based health states in different cultural contexts using different modes of administration.