Reliability and validity of the Cancer Therapy Satisfaction Questionnaire in lung cancer

This study was approved by the Institutional Review Board of the Erasmus University Medical Center in Rotterdam, the Netherlands. Patients were recruited from a university hospital (Erasmus University Medical Center) and a large teaching hospital (Amphia hospital) specialized in lung cancer care located in the western part of the Netherlands. Patients were enrolled in our study if they met the following criteria: They provided written informed consent, were aged 18 years or older, and were treated with at least four cycles of pemetrexed monotherapy or in combination with cisplatin or carboplatin as either first or second line. Patients were excluded if they met the following criteria: They were not able to read Dutch or could not complete the questionnaire because of a physical or mental condition (which prohibited participation in the study). A sample size of at least 50 patients was needed in order to perform a validation study [11].

The CTSQ contains three domains covering 16 items: expectations of therapy (ET; five items), feelings about side effects (FSE; four items) and satisfaction with therapy (SWT; seven items). Each item was scored on a scale from one to five with a value of one corresponding with the worst response and a value of five representing the best response. Four items are reverse-coded. Domain score was calculated by the formula: (mean of completed item scores −1) × 25. This results in a domain score ranging from 0 to 100, with a higher score representing a better outcome on each domain.

The original CTSQ was translated into Dutch by TransPerfect Translations Inc. according to the forward/backward methodology following international guidelines [12]. Questions were translated in a forward manner (English to Dutch) by two independent native-speaking linguists of the target language experienced in the translation of quality of life instruments. A third independent native speaker reviewed these translations and selected the most appropriate translation of the items or provided an alternative version. Discrepancies, linguistic limitations or cultural differences were addressed. Back translation was performed by a fourth independent native speaker with proficiency in English. An oncologist determined whether the Dutch translation was in line with the medical terminology as used in the Netherlands. Finally, five respondents who received cancer treatment in the past 18 months asked to provide feedback on the Dutch CTSQ during an interview. The respondents’ overall impression of the instrument was that it was “easy to complete.” The respondents’ answers corresponded with the intended meanings of the items. During the translation process, some questions were slightly changed (i.e., not literally translated) to ensure conceptual equivalence and cultural relevance to facilitate correct use of Dutch grammar. Permission of use was granted by Pfizer Inc. the current owner of the intellectual rights of the CTSQ. A pre-assessment of the Dutch version was conducted in 14 patients with lung cancer (not included in this study) to assess whether the questions were understandable and acceptable for use in the study.

The European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30 (EORTC QLQ-C30) is a cancer-specific HRQoL instrument with demonstrated psychometric properties [13]. It consists of 30 items and incorporates a global health status/quality of life scale, five functional scales and a number of single items assessing additional symptoms or difficulties. Each of the QLQ-C30 domains is scored on a 0–100 scale, with higher scores on the functional scales being indicative of better HRQoL, whereas higher scores on the symptom scales are reflective of worse symptoms [14, 15].

The World Health Organization Quality of Life-BREF (WHOQOL-BREF) is a shorter version of the original WHOQoL-100 questionnaire. It is a generic QoL instrument and comprises 26 items divided over 4 domains: physical health, psychological health, social relationships, and environment and one facet: overall quality of life and general health. The WHOQOL-BREF domains are scored on a 4–20 scale and the facet on a 2–10 scale with higher scores indicating a better quality of life [16]. The WHOQOL-BREF is a well-established instrument that was developed for use in a wide range of disease areas and health problems [17].

All questionnaires were completed after patients finished their four-cycle therapy of chemotherapy. In addition to completing the instruments, respondents were asked to provide information about the frequency and severity of adverse events they have experienced (cancer or therapy related). We also collected sociodemographic information (age, gender, educational level, ethnicity, smoking status, and clinical history) and information about cancer stage, hospitalization (due to cancer or adverse effect of therapy), and the ECOG performance status.

Floor and ceiling effects were calculated in our study and were considered to be present if more than 15 % of the respondents achieved the lowest (floor effect) or highest (ceiling effect) possible domain score [11].

Construct validity was evaluated using Pearson’s rank correlation coefficient between questionnaire items and domains. Correlations of 0.40 or higher indicate a good correlation between items and domains [11].

Internal consistency reliability measures to which extent items within a domain correlate with each other to form a (multi-item) domain. Reliability coefficients for the CTSQ domains were estimated using Cronbach’s coefficient alpha where a reliability coefficient of 0.70 or higher was considered to be acceptable [11].

Known-groups validity comparisons were made for the CTSQ domains in relation to the number of different adverse events and its severity. Also the impact of therapy-related adverse events compared to cancer-related adverse events on CTSQ domain score was evaluated. For this analysis, the one-way ANOVA was used to determine whether there are any significant differences between the means of two or more independent groups.

The association between the CTSQ domains with domains of the EORTC QLQ-C30 and WHOQOL-BREF was assessed using Spearman’s correlation coefficients.

We assessed interpretability, which is defined as the degree to which one can assign qualitative meaning to quantitative scores. For each CTSQ domain, the MCID was calculated using the approach of 0.5 SD [18] and 1 standard error of measure (SEM) [19‐21]. MCID is the smallest change in an outcome that a patient would identify as important. The 0.5 SD benchmark of an outcome measure entails that patients improving more than 0.5 of the outcome score’s SD have achieved a minimally clinically important difference [22]. For the 1 SEM approach, we have used the internal consistency reliability estimates. In addition, results of the known-groups comparison were used to derive the MCID using the number of adverse events with Common Terminology Criteria for Adverse Events (CTCAE) grade 3 or 4 as an anchor. A p value below 0.05 was considered to be statistically significant.

All analyses were performed using SPSS version 21.0 (IBM Corporation, Armonk, NY).

Table 1 describes the characteristics of our study population. A total of 55 patients completed the questionnaires. The age of these patients ranged from 45 to 79 years, with a mean of 55.0 (SD 8.6). Forty-four patients indicated they had received a low level of education (80.0 %), and 32.7 % stated to be employed. The majority of these patients were diagnosed with adenocarcinoma of the lung (94.5 %), and 85.5 % had stage IV NSCLC. Almost all patients (98.2 %) had a good ECOG performance score (grade 0 or 1). The majority of patients received pemetrexed chemotherapy as a first-line treatment (85.5 %).

The mean scores of the ET and FSE domain were 55.6 (SD 22.5) and 52.2 (SD 23.8), respectively. The SWT domain had a mean score of 79.7 (SD 13.9), which was much higher compared to the mean scores of the other domains. No patients demonstrated the lowest possible domain score of 0.0. The floor effects for all domains were therefore zero. The FSE domain did not reach the highest possible score of 100, resulting in a negligible ceiling effect for this domain. For the ET and SWT domain, we observed a ceiling effect of 5.5 and 9.1 % respectively, which is below the common accepted limit of 15 % (Table 2).

Construct validity was supported for all 16 items as we observed a correlation of 0.40 or higher with their own hypothesized domain. However, we found that item 8 (chemotherapy would help you live longer) had a similar correlation with ET domain (0.50) as the SWT domain (0.48). All other comparisons showed good results, as these items correlated better with their own hypothesized domain than with competing domains (Table 3).

The internal consistency of the CTSQ domains is given in Table 4. All three domains met the reliability coefficient of 0.70 or higher. Cronbach’s alpha of the ET and FSE domains were both above 0.80 (0.83), except for the SWT domain (0.77). As presented in Table 3, we observed that item 8 had a similar correlation with the SWT domain as with the ET domain. We explored whether moving item 8 from the ET domain to the SWT domain would improve Cronbach’s alpha for both the ET and SWT domains. We found a slight increase in the alpha coefficients of both domains (ET: 0.86, SWT: 0.79).

Table 5 shows the known-groups validity comparisons for the CTSQ domains in relation to the number of different adverse events, its severity and chemo-related adverse events. None of these results were found to be significant. We observed an increasing number of grade 3 and 4 adverse events that corresponded with a decreasing mean score of the FSE domain. The same observation was found in the analysis where we looked at the percentage of adverse events that were related to chemotherapy. Also, frequency and severity of adverse events were not related to satisfaction with therapy.

The estimates of the MCIDs are given in Table 6. Estimates of the MCID for the ET and FSE domain were almost the same (0.5 SD: 11.75; 1 SEM: 9.69 and 0.5 SD: 12.4; 1 SEM: 9.28, respectively). The calculated estimates using the 0.5 SD approach were higher for both domains compared to the estimates using the 1 SEM approach. We observed a much lower estimate for the SWT domain (0.5 SD: 6.55; 1 SEM: 6.14) and a smaller difference between the estimates of the 0.5 SD and 1 SEM. The anchor-based MCID was estimated by calculating the average change in CTSQ score. For the ET domain, the estimate that was obtained using the number of grade 3 or 4 adverse events as an anchor was higher than the observed estimates using the 0.5 SD and 1 SEM approach (14.3). For the other two domains, we observed lower estimates when using the anchor-based method (SE: 8.5 and SWT: 5). However, results that were obtained using this method need to be interpreted carefully as the effect size could not be measured due to the low numbers of patients.

The correlation between the CTSQ domains and domains of the EORTC QLQ-C30 is shown in Table 7. We found the FSE domain correlated more strongly with the EORTC QLQ-C30 domains than the other two CTSQ domains. The highest correlations (r ≥ 0.40) were observed with global health status, role functioning, emotional functioning and the symptom domains fatigue, nausea and vomiting, and appetite loss. No correlation of 0.40 or higher was observed between the ET domain and the HRQoL domains. The SWT domain only significantly correlated with nausea and vomiting (r = −0.41). The negative correlations between the CTSQ and HRQoL domains indicate that higher scores of the CTSQ domains are associated with worse symptoms.

Results of the association between the CTSQ and WHOQOL-BREF domains are presented in Table 8. The domains of WHOQOL-BREF had the strongest correlations with the FSE domain. However, only the psychological domain had a correlation above 0.40 (r = 0.52).