Introduction
Since the 1980s, the number of quality of life (QoL) studies has increased dramatically [1] and QoL is becoming an increasingly important and integral outcome in clinical trials [2‐4]. To better interpret responses to QoL questionnaires, we need to examine the underlying cognitive processes [5], which in turn will increase the insight into the validity of QoL questionnaires. However, studies that systematically investigate these cognitive processes are scarce. In this study, we will examine the cognitive processes underlying cancer patients’ responses to QoL items by means of two theoretical models.
In the area of survey research, several models have been developed to describe the answering process to survey questions [6‐9]. We will use the model of Tourangeau and colleagues [9], because it largely resembles earlier survey models and excels in the elaborate description of each process. Second, to specifically address the cognitive processes underlying QoL assessment, we will also use the Rapkin and Schwartz’ QoL appraisal model [10]. Table 1 provides the cognitive components of both models. As can be seen, these models show great resemblance, as they share three cognitive processes and add one each. An important distinction is that the model of Tourangeau et al. [9] aims to explain the cognitive processes used for answering individual questionnaire items whereas the Rapkin and Schwartz’ model [10] focuses on the cognitive processes used in answering an entire QoL questionnaire.
Tourangeau et al.—Survey answering model | Rapkin and Schwartz—QoL appraisal model | |
|---|---|---|
A | Comprehension Paying attention to the question and accompanying instructions Interpreting the question Making assessments concerning the information sought | Frame of reference Assigning meanings to the questions Identifying experiences that are found to be relevant to the given responses |
B | Retrieval Recalling relevant information | Sampling strategy Retrieving relevant information |
C | Standards of comparison Judging each sampled experience against subjective standards of comparison | |
D | Judgment Judging the completeness or accuracy of the retrieved information Making inferences based on the process of retrieval Supplementing gaps in the retrieved information Combining the retrieved information into a single response Making estimates that adjust for omissions in retrieval | Combinatory algorithm |
Prioritizing and combining all relevant experiences to arrive at a QoL score | ||
E | Reporting and response selection | |
Editing the initial response for consistency, acceptability or other criteria | ||
Mapping the judgment onto a response category | ||
To the best of our knowledge, only Wyrwich and Tardino [11] have used the Rapkin and Schwartz model [10] in analyzing qualitative data on the cognitive processes underlying health-related quality of life (HRQoL) transition items. In this study, 41 chronically ill patients were interviewed by phone to identify their thought processes in answering transition items completed at an earlier occasion. A limitation of this study is that the transition assessments and the cognitive interview were not administered simultaneously. It is questionable whether respondents were able to recall their thought processes or rather reconstructed these processes during the interview.
To increase our understanding on how to interpret responses to QoL items, this study’s aim is to develop an analysis scheme capturing the cognitive processes underlying QoL assessment, based on the models of Tourangeau et al. [9] and Rapkin and Schwartz [10]. To elicit these cognitive processes, we will ask cancer patients to think aloud while completing QoL items prior to and following radiotherapy. In contrast to Wyrwich and Tardino [11], we will probe the patients directly after the completion of QoL items to reveal more information on their thought processes.
The specific objectives of this study are to examine whether (1) patients’ answers can be categorized according to the distinct cognitive components of the models, (2) a text fragment can be categorized exclusively in one cognitive component, and (3) the proposed models are exhaustive in capturing the cognitive processes underlying responses to QoL items or need to be extended.
Pilot study
The interview consisted of seven items selected from the 30-item EORTC QLQ-C30 [12], since it is too burdensome for patients to be cognitively interviewed about all items. Therefore, we first conducted a pilot study aimed at selecting the questionnaire items, the interview procedure and interview probes.
Material and methods
Sample
We selected four individuals with different health problems (mononucleosis, migraine, pelvis injury, repetitive strain injury) and with a social science background so that they were able to provide useful comments on the cognitive interview. These individuals were not part of the research team. The sample consisted of three women and one man (mean age 38.8 years, SD 12.4, range 25–55).
Questionnaire items
The seven items derived from the EORTC QLQ-C30 [12] were selected such that they covered global as well as specific content, and physical as well as emotional dimensions.
We thus included two items from the physical functioning scale (Do you have any trouble taking a short walk outside of the house?, Were you limited in doing either your work or other daily activities?), two items covering the most common symptoms (Have you had pain?, Were you tired?), one emotional functioning item (Did you feel depressed?), and two global items (How would you rate your overall health during the past week?, How would you rate your overall quality of life during the past week?). All items employed a one week time frame.
Cognitive interview
We used the Three-Step Test-Interview (TSTI) [13], combining the two subtypes of cognitive interview methods (think aloud interviewing and verbal probing techniques) [14]. In the first step, the respondents are instructed to think aloud while completing questionnaire items. In the second step, the interviewer poses focused questions aimed at remedying gaps in the data collected in the first step, e.g., pauses in which the respondent did not think aloud. The third step consists of semi-structured probes aimed at eliciting more information about the cognitive processes used. The first two subjects were interviewed retrospectively (after administering all seven items), the other two subjects were interviewed while probing concurrently (immediately after the think aloud response to each QoL item).
Interview probes
The interview probes were based on the models of Tourangeau et al. [9] and Rapkin and Schwartz [10]. Example probes within each cognitive component are illustrated in Table 2. To clarify patients’ responses we additionally posed non-leading probes like “Could you tell me more about that?” and “Could you explain that to me?”.
Cognitive component | Interview probes |
|---|---|
Comprehension/frame of reference | What does (target construct in item, e.g., quality of life) mean to you? |
Retrieval/sampling strategy | Can you tell me how you came to think of (aspect mentioned by respondent)? |
Standards of comparison | Did you compare yourself to someone or something? |
Judgment/combinatory algorithm | How did you arrive at your response? |
Reporting and response selection | Can you tell me why you choose the response category ‘a little’ and did not select the response category ‘quite a bit’? |
Results
Questionnaire items
After answering the first item covering physical functioning ‘Do you have any trouble taking a short walk outside of the house?’, the second physical item ‘Were you limited in doing either your work or other daily activities?’ did not add new information about the cognitive processes underlying patients’ assessment of their physical functioning. To better reflect the multidimensional character of the concept QoL, we replaced this item by the social functioning item ‘Has your physical condition or medical treatment interfered with your social activities?’. Second, since respondents did not endorse the item ‘Did you feel depressed?’ we came to realize that this item reflects a too extreme emotional state. To elicit more elaborate cognitive processes we replaced it by the more prevalent state “Did you worry?”.
Interview procedure
In probing retrospectively, the respondents indicated that they could not remember what they were thinking previously when answering the questionnaire items. This resulted in respondents reconstructing their answering process. Therefore we chose to probe concurrently. The disadvantage of this approach is that the think aloud process can be influenced by the semi-structured probing of the preceding item. Although this effect was not apparent in the pilot study, it cannot be ruled out [14].
Interview probes
The interview probes appeared to elicit the cognitive processes used in answering the QoL items, since the respondents understood the probes and answered them accordingly.
However, when respondents assessed their current functioning, they were often not aware of the fact that they used their prior functioning as a comparison. Direct probing about their standard of comparison proved troublesome in these cases. Since respondents mentioned their used standards spontaneously, direct probing about standards of comparison was often not needed.
Main study
Material and methods
Sample
At the Department of Radiotherapy of the Academic Medical Center (AMC) in Amsterdam, four radiotherapists consecutively recruited newly diagnosed cancer patients undergoing radiotherapy and matching our inclusion criteria: a minimum age of 18 years, fluent command of Dutch language and absence of cognitive impairments. Data saturation was reached based on 80 response processes of six patients (mean age 64.7 years, SD 9.0, range 49–75) (for further explanation see the Discussion). Four patients completed all seven QoL items at baseline and follow-up. The other two patients provided interpretable data for six items at both interviews. Overall, the interviews yielded 80 response processes useful for qualitative analysis ((four patients × seven items × two interviews) + (two patients × six items × two interviews)). All patients consented to participate. Four patients were men, two of them were diagnosed with prostatic cancer and two had esophageal cancer. One female patient was diagnosed with a gynecological tumor, the other with bladder cancer. One patient had elementary education, two patients had lower general secondary education, and three patients had lower or intermediate vocational education. The mean time between diagnosis and baseline interview was 62 days (SD 23.9, range 28–91).
Design
Patients were interviewed using the questionnaire items, TSTI procedure and probes resulting from the pilot study. Interviews were conducted prior to and following radiotherapy. Baseline interviews were administered on the day the patient had an appointment for the simulator, a medical procedure to plan the actual treatment. The follow-up interviews were either held at the last day of radiotherapy or in combination with an appointment with the patient’s treating radiotherapist, varying from 1 to 2 weeks after completion of radiotherapy. The mean duration of the interviews was 41 minutes (SD 11.3, range 30–71). The interviews were held at the AMC with only the patient and the interviewer being present and were conducted by the same interviewer (EB).1 During the interview, the interviewer kept the distinct cognitive processes of both models at the back of her mind and only probed for those cognitive components that did not emerge in the think aloud answers as provided by the patient. All interviews were tape-recorded and transcribed verbatim. After the interviews, the interviewer (EB) made field notes describing contextual information.
Analysis
Since the cognitive process models of Tourangeau et al. [9] and Rapkin and Schwartz [10] have not been specifically designed for the analysis of qualitative data, we had to develop a detailed plan of analysis. Content analysis of the interviews was carried out by at least two out of four researchers (EB, MK, FvZ, MS) independently and started directly after a patient had completed both interviews. To provide an open and unbiased account of the cognitive processes used in answering QoL items, we started all analyses inductively (data driven) by summarizing the salient content of each interview. Subsequently, all 80 text fragments were analyzed deductively (theory driven) according to a preliminary analysis scheme based on a combination of the two models using MAXqda software. The inductive and deductive analyses were compared in an iterative process. Since knowledge of the models might influence the inductive analyses, we actively searched for information that would not fit in or run counter to these models. After both interviews of each patient had been analyzed, all researchers involved discussed their findings. Frequently, this discussion yielded new insights that resulted in adaptations of the preliminary analysis scheme. Consequently, these amendments resulted in renewed analyses by all researchers of all text fragments and a renewed discussion of the findings. Finally, this iterative process resulted in a comprehensive analysis scheme based on mutual consensus. This analysis scheme is provided in Appendix 1, presenting all possible codes within each cognitive component and a description of each code. Please note that not all codes within each cognitive component will be applicable in analyzing the response process underlying each QoL item, which is illustrated in Appendix 2.
Results
Categorization of text fragments according to the cognitive models
(A) Comprehension/frame of reference The patients indicated what they understood the target construct to mean in 77 response processes. In 51 response processes the patients additionally mentioned what they considered to be the opposite of the target construct. For example:
How would you rate your overall quality of life during the past week? (range 1 (very poor) – 7 (excellent))“I would say a ‘3’. (…)2 Quality of life is being healthy and to be able to do everything you like. (…) I feel limited however, I can’t do a thing. I’m no longer self-reliant, I am dependent on others.”[Female (F), 71 years, gynecological cancer]
(B) Retrieval/sampling strategy In retrieving relevant information within the frame of reference, subjects can either recall positive or negative experiences. Sampled experiences could be distinguished in all 80 response processes. The patients retrieved far more negative (227) than positive (141) samples in their QoL assessments.
The period of time the patients considered in arriving at their answers could be identified in all response processes. In 11 response processes patients referred to two to three different periods, e.g., the time prior to cancer diagnosis and the past week. In 34 response processes, the patients referred to the past week, in agreement with the time-frame as employed by the EORTC QLQ-C30. Other coded periods are the period since cancer diagnosis (N = 27), period of radiotherapy (N = 18, follow-up), time prior to cancer diagnosis (N = 12), time since cancer treatment other than radiotherapy (N = 7), time since radiotherapy (N = 4, follow-up) and other periods, e.g., the future (N = 8).
The following excerpt exemplifies this cognitive component; in assessing his pain, the patient retrieves a negative sample (injection as part of hormonal therapy) by thinking about his last painful experience 2 days before (period).
Have you had pain?“Well, what is pain? When I get an injection in my stomach, with a very thick needle, like two days before. That hurts.”You immediately mentioned the injection, was that the first thing that came to your mind?“Yes, since that’s the last painful thing I have had this week.” [Male (M), 66 years, prostatic cancer]
(C) Standards of comparison To arrive at an answer, the patients spontaneously referred to a reference group in 77 response processes, of which 33 contained two to four different reference groups, e.g., self prior to cancer diagnosis and an expectation about the course of the disease. In the majority of the response processes, the patients made a comparison with themselves (N = 68). Most frequently (N = 47) the patients compared their current functioning with their own functioning prior to cancer diagnosis and treatment. In 15 response processes, other (cancer) patients were used as comparison, for example:
Did you worry?“(…) I have buried my brother, he also had prostatic cancer. But I know, all patients are different and he had metastases in his bones, and I haven’t”. [M-2, 66 years, prostatic cancer]
Additionally, patients compared themselves with an expectation (N = 19), an ideal (N = 5), other people the same age (N = 2) and other reference groups (N = 13), e.g., mentally ill patients.
(D) Judgment/combinatory algorithm This cognitive component is of particular relevance when respondents consider both positive and negative samples (N = 54). In arriving at a response to the QoL item, the respondents were found to emphasize the positive (N = 15) or negative samples (N = 10), or find a balance between both (N = 29). The following excerpt shows an example of a respondent emphasizing the positive sample.
Has your physical condition or medical treatment interfered with your social activities?“Not at all. Well, at the moment I am on sick leave. (…) I really enjoy my work, I have friends there. I consider my work to be a social activity. (…) The place where you have the majority of your social contacts during the day. (…) But I also have my social contacts at home. I can’t say I have no social contacts left. (…) Now there are other people I can visit, people I do not get around to normally.” [M, 49 years, esophageal cancer]
(E) Reporting and response selection In 67 response processes patients explained how they arrived at their answer. For example:
“I did worry a lot, now I worry to a lesser degree. So my answer would be ‘quite a bit’ or ‘a little’ (…) I’m convinced that everything will be all right when I’m being operated, so I’ll leave it at ‘a little’” [M, 71 years, esophageal cancer].
According to Tourangeau et al. [9] respondents may edit their response for consistency, acceptability or other criteria, e.g., the wish to avoid embarrassment or politeness to the interviewer. This way of editing a response was found in only two of the 80 response processes. In the following excerpt, the patient edits his initial QoL rating to make it consistent with his response to the preceding question in which he rated his overall health:
“(…) Since I’ve just rated my overall health as ‘very poor’, I have to do the same for my quality of life, don’t I?” [M, 71 years, esophageal cancer].
Exclusivity of categories
Systematic overlap was found between three cognitive components. First, between the components retrieval/sampling strategy and standards of comparison. This is partly in line with Rapkin and Schwartz’s model [10] according to which each sampled experience is judged against subjective standards of comparison. However, from the 368 samples mentioned, only 193 samples were judged against standards of comparison and thereby coded in both cognitive components. The remaining 175 samples were not compared to any reference group. Second, overlap was found between the components retrieval/sampling strategy and judgment/combinatory algorithm. This overlap is suggested by both models in that all sampled experiences need to be combined in a particular way to arrive at an answer. The following excerpt shows the overlap between these cognitive components:
“[Trouble taking a short walk] means that you dread walking or that you simply just can’t manage to get it done. I have experienced that once when I was operated on for my back2,3. And now I am not experiencing that kind of trouble at all1. I even prefer taking a longer walk than a short one4.” [M, 66 years, prostatic cancer]
1 Positive sample (Retrieval/Sampling strategy)2 Negative sample (Retrieval/Sampling strategy)3 Self – other [medical treatment in the past] (Standards of comparison)4 Emphasis within sampling – positive (Judgment/Combinatory algorithm)
Extension of models
Sixty-five response processes reflected editing processes for criteria other than those proposed by Tourangeau et al. [9]. These editing processes are interpreted as mitigating the initial response to make it acceptable for the respondent him/herself instead of making it socially acceptable. Therefore, we extended the cognitive component ‘reporting and response selection’ with three editing processes; self-protection (N = 34), self-presentation (N = 17) and normalization (N = 14). For example, in the following excerpt the patient edits his initial answer for self-protection in taking into account that his situation might deteriorate.
How would you rate your overall quality of life during the past week?“Actually I would rate my quality of life with a ‘7’ [excellent]. But life is full of surprises, they might find a metastasis in the future. So I can’t choose a ‘7’, I have to rate my quality of life with a ‘6’.” [M, 66 years, prostatic cancer]
The wish to present one-self in a specific way was also found to be a motive for respondents to edit the initial answer. For example:
Were you tired?“Yes, I suppose radiation treatment and chemotherapy caused that. In the beginning it didn’t cause that much trouble, but lately I do notice something’s going on. I get tired very easily, I haven’t got any energy left. I just want to sit on the couch the entire day. (…) I’ll say I’m tired ‘quite a bit’, ‘very much’ doesn’t suit me.” [M, 49 years, esophageal cancer]
In answering the QoL items, the patients referred to their ageing to normalize their deteriorated health and functioning. This process of editing for normalization resulted in reporting a mere absence of symptoms. For example:
Has your physical condition or medical treatment interfered with your social activities?“No, not at all. (…) Five weeks ago I’ve painted four houses from the outside, when I already knew about me having cancer. Now I don’t feel up to it anymore (…) Well, you need to close certain periods of your life. But that’s simply a part of getting older.” [M, 71 years, esophageal cancer]
Discussion
We have developed a comprehensive analysis scheme feasible for the qualitative analysis of the cognitive processes underlying responses to QoL items. Our patients were capable of verbalizing their cognitive processes used in answering the QoL items. The models of Tourangeau et al. [9] and Rapkin and Schwartz [10] combined proved applicable in the subsequent analysis of these cognitive processes since all cognitive components of both models could be discerned in our patients’ responses.
The fact that the interview probes we used to elicit more information about the cognitive processes were based on both models might account for the finding that all cognitive components as purported in the models could be distinguished in our data. However, in their think aloud answers, the patients frequently passed through the distinct cognitive components spontaneously, making further probing redundant. When we did use interview probes we formulated them as open and non-directive as possible.
In retrieving relevant information, the patients more often used negative samples than positive ones (sampling strategy). Considering the negative content of the QoL items (e.g., worry, pain), this can be considered a logical finding.
In accordance with the models, the only systematic overlap was found between the cognitive components retrieval/sampling strategy, standards of comparison and judgment/combinatory algorithm.
Although all cognitive components could be identified, the process of editing a response for consistency or acceptability (reporting and response selection) could only be discerned in two response processes. This can be explained by the fact that our patients might not consider the items intrusive or sensitive in a way that they felt the need to edit their response in order to provide a socially desirable response, as proposed by Tourangeau et al. [9]. However, our patients might consider the selected QoL items to be sensitive as they confront them with their deteriorating health. The use of processes to edit their responses for self-protection, self-presentation and normalization might reflect this sensitivity. In our analysis scheme, we extended the cognitive component ‘reporting and response selection’ with these editing processes to fully capture the cognitive processes underlying responses to QoL items.
Since this study, we continued using our analysis scheme to examine the cognitive processes underlying 84 QoL response processes of six other cancer patients. In analyzing their thought processes, our analysis scheme did not need revision, thereby confirming data saturation, which makes us feel confident about the scheme’s adequacy.
A remarkable finding merits attention. Rapkin and Schwartz’ model [10] describes the cognitive processes respondents use in answering an entire QoL questionnaire. It thereby assumes that respondents use the same cognitive processes in answering all individual questionnaire items. However, we found that the content of the cognitive components did not only differ per patient, but also per item within patients and over time. For example, patients compared themselves with other patients in one item, and referred to their own functioning prior to cancer diagnosis in another. Likewise, patients differed per item in the period of time they considered, the way they combined positive and negative samples and so forth. We therefore could use the response processes underlying each QoL item as unit of analysis rather than the individual patient. Consequently, we achieved data saturation based on 80 response processes instead of six cancer patients. Although this finding needs to be confirmed in more patients, it has implications for our insight into the cognitive processes underlying QoL items and for the design of qualitative QoL studies.
A range of studies have addressed the first component of the model of Tourangeau et al. [9]. In these studies, patients, elderly or healthy respondents were asked to complete a global item regarding QoL or health and were subsequently asked how they comprehended that question [16‐21]. Only Groves et al. [17] and Kaplan and Baron-Epel [21] additionally addressed the comparison category of Rapkin and Schwartz’s model [10], by asking respondents whether they used subjective standards of comparison. To the best of our knowledge, this study is the first to combine both models in a comprehensive analysis scheme.
The practical implications of our findings are twofold. Firstly, our findings may point to possible flaws or unintended responses which may help to improve questionnaires, e.g., in assessing their QoL patients do not refer to the period of time as employed by the questionnaire. Secondly, in examining the cognitive processes underlying baseline and follow-up assessments, we will be able to determine whether patients’ QoL evaluation processes remain stable or rather change over time. Such findings will increase the insight into the interpretation and validity of self-reported change in QoL.
Acknowledgements
This study was funded by the Dutch Cancer Society (UVA 2005-3197). We thank H. J. G. D. van den Bongard, E. G. Geijsen, M. C. C. M. Hulshof and B. R. Pieters from the Department of Radiotherapy of the Academic Medical Center (AMC) in Amsterdam for their help in patient accrual. We additionally thank C. Tishelman for helpful discussions about the design of the study.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.