Elsevier

Manual Therapy

Volume 15, Issue 1, February 2010, Pages 43-47
Manual Therapy

Original Article
Added value of qualitative studies in the development of health related patient reported outcomes such as the pain coping and cognition list in patients with sub-acute neck pain

https://doi.org/10.1016/j.math.2009.05.010Get rights and content

Abstract

Psychometric analyses, such as factor analysis, internal consistency and construct validity analysis, are well known and frequently applied methods in the development of health related patient reported outcomes. These statistical indexes shed very little light on how respondents interpret individual items, or on the meaning of their responses. In this study, the Pain Coping and Cognition List (PCCL), a quantitatively validated psychological questionnaire developed to assess chronic pain, has been subjected to a qualitative research method: the Three Step Test Interview (TSTI), an observational technique that aims to identify problems with self reported questionnaires. It consists of three phases: 1) concurrent thinking aloud; 2) a retrospective interview; 3) a semi-structured interview. Participants with sub-acute neck pain distinguished six different types of problems: long complicated formulations, composite questions, irrelevant questions, lacking frame of reference, problematic words, and wrongly interpreted questions. This study illustrates that qualitative methods have an added value when developing self-report questionnaires because some of the problems that were highlighted that cannot be identified using quantitative methods only. Therefore, we recommend that a full qualitative study should be an integral part of the development of questionnaires. The TSTI method is very useful for this purpose.

Introduction

When developing questionnaires it is important that they provide valid and reliable results. A prerequisite is that the questionnaire measures what it is supposed to measure. To address this issue of validation a golden standard is needed and this is not always available. It is also important that respondents understand the items in the questionnaires, and therefore give answers to each individual item as was intended by the developers of the questionnaire. Psychometric analyses, such as factor analysis, internal consistency and construct validity analysis, are well known and frequently applied methods in the development of health related patient-reported measurement instruments such as questionnaires. Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), for example, are also used to examine the underlying dimensions in a questionnaire, but in these quantitative methods statistical techniques are applied to assess the psychometrical characteristics of a questionnaire. In doing so, these techniques will highlight any problems that might occur in the items of a questionnaire, for example, in terms of a low item-total correlation, indicating that such an item does not fit satisfactorily into the questionnaire or into a dimension of a questionnaire. However, these statistical indexes shed very little light on how respondents interpret individual items, or on the meaning of their responses. Qualitative research methods (Collins, 2003) are well suited to study the interpretation of psychological self-administered questionnaires. For this purpose, the Three Step Test Interview (TSTI) can be used. The TSTI is a qualitative research instrument, specifically designed for testing self-report questionnaires. It can be used to identify problems that originate from a mismatch between, on the one hand, the ‘theory’ underlying the questions and, on the other hand, features of a respondent's actual behaviour (Van der Veer, 2003, Van der Veer et al., 2005).

In this study, a psychological questionnaire, the Pain Coping and Cognition List (PCCL) (Stomp-van der Berg et al., 2001), has been subjected to such a qualitative analysis (see Appendix). The PCCL was developed to measure coping, cognitions and locus of control in patients with chronic pain. It is a self-report questionnaire, and consists of 42 items to be answered on a 6-point Likert scale, ranging from ‘I completely disagree’ to ‘I completely agree’. The PCCL is a combination of the Pain Cognition List developed and validated by Vlaeyen et al. (1990), the Locus of Pain Control Questionnaire (Engstrom, 1983), translated and validated in Dutch (ter Kuile et al., 1993, ter Kuile et al., 1999), and the Coping Strategies Questionnaire (Rosenstiel and Keefe, 1983), translated and validated in Dutch (Spinhoven et al., 1994). The internal consistency, test–retest reliability and construct validity of all these individual questionnaires have been tested, and the psychometric characteristics have been found to be acceptable (Stomp-van der Berg et al., 2001). In the pilot phase it was checked whether a small number of participants understood the questions correctly. Although the PCCL has been validated quantitatively (Stomp-van der Berg et al., 2001), little research has focussed on how participants interpret the various items, and, consequently, how the data that this questionnaire produces should be interpreted. Although developed for patients with chronic pain, the PCCL is also suited for patients with other pain conditions, because the dimensions of the questionnaire are not only typical for chronic pain.

During a randomised controlled clinical trial on neck pain (Pool et al., 2006), participants with sub-acute neck pain were asked to complete the PCCL. In this study it appeared that most of the participants needed about 15–20 min to complete the questionnaire. However, some participants needed an extremely long period of thinking time. This might be due to the fact that they had difficulty in understanding the items, and/or that their thoughts and considerations fitted poorly into the response categories.

The objectives of this qualitative study were to find out what kind of problems patients with sub-acute neck pain experience when filling in the PCCL, and how they interpret the questions. It may lead to further adaptation of certain items in the questionnaire, or more extensive instructions for its use.

Section snippets

Participants

Participants with non-specific sub-acute neck pain, were recruited by one general practitioner, three physiotherapists, and one manual therapist, all working in a primary care setting. They informed the participants about the neck pain study and invited them to take part in the study. The inclusion criteria were the same as used in the clinical trial on neck pain (Pool et al., 2006). The Medical Ethics Committee of the VU University Medical Center in Amsterdam approved the study protocol.

The

Performance of the Three Step Test Interview

The think aloud method was a new experience for all of the participants in the present study, because none of them had ever filled in a questionnaire ‘thinking aloud’. For this reason, the first phase appeared to be difficult for some of the participants. A number of older participants clearly had more difficulty than some of the younger participants. Moreover, participants with a higher level of education appeared to be more critical with regard to the formulation of the questions. Moreover,

Discussion

Although the PCCL and its components have been validated quantitatively, this study illustrates that qualitative methods identify problems that cannot be retrieved by quantitative methods, such as factor analysis (EFA or CFA) or analysis of internal consistency. Based on these statistical techniques these items were not excluded from the questionnaire. However, 43% of the items (18 out of the 42!) still caused problems to a greater or a lesser extent. These problems were not due to the fact

Conclusion

Qualitative methods, such as the TSTI have an important added value, in addition to quantitative methods, in the development of questionnaires. The current study illustrates that in a questionnaire such as the PCCL, which has been validated quantitatively, by factor analysis, and analysis of internal consistency and construct validity, still contains considerable number of items which were problematic for participants with sub-acute neck pain. Therefore we recommend that qualitative methods

References (11)

There are more references available in the full text version of this article.

Cited by (0)

View full text