Correction
2 Jan 2014: Kyte DG, Draper H, Ives J, Liles C, Gheorghe A, et al. (2014) Correction: Patient Reported Outcomes (PROs) in Clinical Trials: Is ‘In-Trial’ Guidance Lacking? A Systematic Review. PLOS ONE 9(1): 10.1371/annotation/e7d2b920-2c2a-425e-b500-982cd72f4d64. https://doi.org/10.1371/annotation/e7d2b920-2c2a-425e-b500-982cd72f4d64 View correction
Figures
Abstract
Background
Patient reported outcomes (PROs) are increasingly assessed in clinical trials, and guidelines are available to inform the design and reporting of such trials. However, researchers involved in PRO data collection report that specific guidance on ‘in-trial’ activity (recruitment, data collection and data inputting) and the management of ‘concerning’ PRO data (i.e., data which raises concern for the well-being of the trial participant) appears to be lacking. The purpose of this review was to determine the extent and nature of published guidelines addressing these areas.
Methods and Findings
Systematic review of 1,362 articles identified 18 eligible papers containing ‘in-trial’ guidelines. Two independent authors undertook a qualitative content analysis of the selected papers. Guidelines presented in each of the articles were coded according to an a priori defined coding frame, which demonstrated reliability (pooled Kappa 0.86–0.97), and validity (<2% residual category coding). The majority of guidelines present were concerned with ‘pre-trial’ activities (72%), for example, outcome measure selection and study design issues, or ‘post-trial’ activities (16%) such as data analysis, reporting and interpretation. ‘In-trial’ guidelines represented 9.2% of all guidance across the papers reviewed, with content primarily focused on compliance, quality control, proxy assessment and reporting of data collection. There were no guidelines surrounding the management of concerning PRO data.
Conclusions
The findings highlight there are minimal in-trial guidelines in publication regarding PRO data collection and management in clinical trials. No guidance appears to exist for researchers involved with the handling of concerning PRO data. Guidelines are needed, which support researchers to manage all PRO data appropriately and which facilitate unbiased data collection.
Citation: Kyte DG, Draper H, Ives J, Liles C, Gheorghe A, Calvert M (2013) Patient Reported Outcomes (PROs) in Clinical Trials: Is ‘In-Trial’ Guidance Lacking? A Systematic Review. PLoS ONE 8(4): e60684. https://doi.org/10.1371/journal.pone.0060684
Editor: Antje Timmer, Bremen Institute of Preventive Research and Social Medicine, Germany
Received: June 27, 2012; Accepted: March 4, 2013; Published: April 1, 2013
Copyright: © 2013 Kyte et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: DK is supported by a National Institute for Health Research (NIHR), www.nihr.ac.uk, School of Primary Care Research (SCPR) PhD studentship. MC is a member of the MRC Midland Hub for Trials Methodology Research, University of Birmingham, UK (Medical Research Council Grant ID G0800808). No other funding sources reported. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Patient reported outcomes (PROs) such as health-related quality of life (HRQL) are increasingly assessed in clinical trials.[1]–[3] PROs provide researchers, clinicians and patients with important information regarding the effect of a disease and its treatment: on symptoms (for example, pain or fatigue) and on HRQL or satisfaction with care.[4] In general, patients participating in a trial do not directly benefit from completing a PRO questionnaire. This approach is adopted to ensure trial participants are not tempted to tailor their answers in order to influence the treatment they receive within a study, which is a potential source of bias.[5], [6] PRO results are therefore used to inform the care of future patients[6], who, with their clinicians, may use PRO data to inform significant health-care decisions. For example, between interventions offering similar survival or progression-free survival rates, or those that have differing trade-offs between therapeutic benefit and undesirable side-effects.[4] Thus, it is crucial that PROs are administered and processed in an un-biased way.
In order to ensure high quality PRO trial data, consistent and rigorous standardised data collection methods should be used throughout a trial.[7] The use of standardised methods should serve to minimise errors, measurement variability, missing data and systematic bias, thus contributing to the validity of trial results.[8] Local site staff require access to ‘in-trial’ (i.e. recruitment, data collection and data inputting, see Box S1) guidelines that clearly outline the standardised methods in-use, so that all study personnel may fully incorporate them into practice. Such guidelines should be contained within the trial protocol, supported by standard operating procedures (SOPs) where appropriate.
It is of concern, therefore, that anecdotal evidence - obtained during national quality of life training days run by the MRC Midland Hub for Trials Methodology in the UK - suggests that in-trial PRO guidelines are not routinely included within trial documentation and that, as a result, unstandardised PRO data collection may be common. Researchers also report feeling particularly uncomfortable that they receive no specific guidance on how to manage ‘concerning’ PRO data, i.e. data that might raise concern for the wellbeing of the trial participant in some way. Staff encountering such data - commonly represented by markedly low HRQL scores, or unexpected unprompted additional information recorded on the back of questionnaires - were therefore unsure where their responsibility should lie, or whether they should be viewing this information in the first place. In this situation, some described experiencing a ‘dual-role’ tension between their concurrent responsibilities as a clinician and researcher: the duty to act upon the information to benefit the patient verses that of protecting trial integrity by not intervening. In some instances, reports indicated that off-protocol concomitant interventions had been administered, some of which may not have been captured by standard trial reporting mechanisms. Such interventions have the potential to bias trial results. These anecdotal reports have since been supported by a recently completed qualitative study, in which we used semi-structured interviews to explore the experiences of 26 research nurses, research facilitators, trial coordinators and data managers across three NHS sites and two clinical trials units in the UK[9] (under review). This study confirmed a potential for bias associated with concerning PRO data, during both postal or clinic-based and self-reported or researcher/research nurse-assisted data collection.
These reports suggest a lack of in-trial PRO guidance, with a subsequent absence of systematic monitoring of potentially concerning PRO data and a resulting risk of bias. It is uncertain, however, whether they also reflect a deficiency in the published literature in this area. There are recent publications concerning the design of trials with a PRO outcome[7], [10] and, with the development of the CONSORT PRO extension[11], there is now guidance to improve PRO reporting: it remains unclear if the literature provides adequate coverage of in-trial issues.
The purpose of this study was to systematically review the current published in-trial PRO guidance, as no review of this kind had been previously undertaken. The objectives for our review were:
- To investigate the extent and content of the current in-trial PRO guidelines in publication.
- To determine if these guidelines adequately address questions raised by researchers involved in PRO data collection, surrounding the management of concerning PRO data.
Methods
Search strategy
The MEDLINE (Ovid), EMBASE, AMED and CINHAL+ databases were searched from inception to March 2012 (electronic search strategies are presented in full in Appendix S1). We also searched; the US Food and Drug Administration[12], European Medicines Agency[13], General Medical Council[14], Medical Research Council[15] and Royal College of Nursing[16] websites; PROQUEST (Thesis repository); Google; and made use of expert communication in an attempt to find additional potentially eligible papers not returned during the electronic database search. Records were first screened by title/abstract before full-text articles were retrieved for eligibility evaluation. Remaining articles were then subject to a citation search before a final hand-search of all reference lists.
Identification of eligible studies
Papers were deemed eligible if they included any form of in-trial guideline focused on PRO assessment during clinical trials. We defined the term ‘in-trial’ as relating to recruitment, data collection and data inputting activity, occurring from the first participant recruitment, through to inputting the final participant's data. The reviewers used the Oxford English Dictionary definition of the word ‘guideline’ during eligibility screening; “a general rule, principle, or piece of advice”.[17] Non-English papers were excluded. There were no other restrictions. All citations were downloaded into Endnote® software version 14, and duplicates deleted. DK screened all articles by title/abstract to determine their eligibility and AG reviewed a random sample of 10% in order to evaluate the reliability of the selection process. Agreement was high (Kappa = 0.903) and any discrepancies were resolved through discussion. Full text articles were retrieved following first round exclusions and were also subject to two independent eligibility reviews (DK 100%, AG 10%), this time with perfect agreement.
Data extraction
Data extraction occurred following the final selection of included articles.
DK and CL independently searched each paper to identify all sentences that provided any type of ‘guideline statement’ (which we defined as ‘an expression in words of a general rule, principle, or piece of advice') regarding PRO measurement (in-trial or otherwise). A consensus meeting was then held, to resolve any disagreements and finalise the selection. Each sentence, representing one ‘guideline statement’, was then extracted, as a text excerpt, into a mixed-method data analysis software package (Dedoose © 2011 SCRC) and tagged with its source data (Article title, Journal, Year of publication).
Data analysis
DK and CL undertook a qualitative content analysis[18] of the excerpts extracted from the included papers. All text excerpts were categorised according to an a priori coding frame, which was developed using a concept-driven strategy (i.e. codes were assigned based on the authors' prior knowledge of the literature and the study research questions). DK and CL piloted the coding framework, each independently applying the first draft to a random selection of the included papers[6], [7], [19] (n = 3 (17%)). Following the pilot, a meeting was held to discuss issues requiring clarification and to reach consensus regarding the data-driven changes that would improve the validity of the framework. Three of the co-authors (MC, HD and JI), who possess expertise in PRO design, implementation, reporting and ethics, checked and approved the face validity of the final coding frame. The definitive coding frame is presented in Figure 1. During the main analysis, DK and CL independently categorised each guideline statement according to the phase of trial activity to which it pertained, using a major dimension within the coding frame. These major dimensions were as follows; ‘Pre-Trial’, which included all content relating to the trial inception (including training logistics), up to the start of recruitment; ‘In-Trial’, denoting content directly related to the act of trial recruitment, data collection and inputting; ‘Post-Trial’, including activity taking place following data collection, for example, data analysis/reporting; ‘Future Research’, representing statements addressing the future direction of PRO research activity; and ‘Other’, used to identify guideline statements not captured in the main coding categories. Each individual guideline was also sub-categorised, as appropriate, in order to further identify its role within a given area.
Major categories in bold.
Throughout both the pilot and the main analysis phase, the reviewers met frequently to determine coding reliability for each paper and to seek consensus regarding coding disagreements. The reliability of coding application was determined using Cohen's kappa statistic.[20] Specifically, pooled kappa was employed, as it is the preferred method of calculating inter-rater agreement across a large number of coding items.[21] Face validity of the coding frame was further evaluated by determining the proportion of codes applied to the residuals (i.e., the ‘Other’ major- and sub-categories). A high level of residual coding may indicate that the main categories of the coding frame do not adequately describe the concept under study.[18] Whilst there are no firm guidelines regarding the desirable level of residual coding, we theorised that a figure of less than 5% would support the validity of our coding frame.
A protocol was not published or registered for this study. However, all reviewers followed a protocol detailing a priori determined search strategies, data extraction and data analysis methods.
Results
Included studies
The search strategy yielded 1273 citations from MEDLINE, EMBASE, AMED and CINHAL+, 89 citations were returned using other sources (PROQUEST, professional bodies, Google, expert communication) (PRISMA[22] flow diagram, Figure 2). In total, 41 full text articles were retrieved for review. 25 articles were excluded at this stage, as they contained no in-trial guideline statements. An additional 2 papers were included following the reference list and citation searches. A final total of 18 relevant articles were included in the analysis.
Study characteristics
The characteristics of the 18 included papers are summarised in Table 1. The majority of papers were concerned with the incorporation of PRO/HRQL measures into cancer trial design.[5], [6], [10], [23]–[28] Several considered PRO issues relating to pharmaceutical prescribing/labelling.[1], [7], [29]–[32] Two papers presented generalised guidance on using PRO/HRQL measures in clinical trials.[33], [34] Finally, one paper presented recommendations for PRO/HRQL assessment in allergy-related clinical trials.[19] The included articles were drawn from 16 different sources and the mean number of excerpts extracted from each paper was 58 (range 16–127).
Data synthesis
Over 1,110 guideline statements were extracted and coded following review of the 18 papers. The coding frame demonstrated reliability, with pooled kappa ranging from 0.86 to 0.97 across articles, and face validity, with overall residual coding at 1.2%. A summary of the final coding breakdown is presented in Table 2.
Major coding categories
‘In-trial’ guidance, whilst present in all papers, did not represent the major focus of any, accounting for 9.2% of guideline content across the articles reviewed. ‘Pre-trial’ guidelines were predominant throughout (72.2%), again present in all papers. ‘Post-trial’ guidance was the next most prevalent category (15.8%), presented across 13 articles.[1], [5], [7], [10], [19], [23], [24], [28]–[33] Statements pertaining to ‘future research’ represented 1.8% of guidelines (9 papers)[10], [19], [23], [24], [26]–[28], [30], [31] and the major category ‘Other’ was attributed to 1% of content (8 papers).[7], [10], [24], [26], [27], [29]–[31]
Sub-categories
In-trial.
There were no guideline statements addressing the management of concerning PRO data, or related questions including how additional information recorded on the back of questionnaires should be handled and who should have routine access to PRO data in the first instance. The majority of in-trial guidelines (61.2%) tackled notions surrounding quality control, compliance and the correct use of PROs.[1], [5]–[7], [10], [19], [24]–[31], [33], [34] Authors highlighted the importance of minimising missing items during data collection.[6], [34] A number of papers presented guidance aimed at improving compliance within a trial in order to maximise data quality: examples included the proposed education of local site staff, training of patients and use of real-time adherence monitoring [1], [5], [6], [10], [24], [25], [27]–[29], [34]. Other guidelines were concerned with piloting[27] and standardisation[28], [31] of data collection. Examples of suggested methods of standardisation included the following:
- A named individual, concerned with quality control, serving as a PRO data collection contact at each research site within a trial.[1], [6], [10]
- The use of standard scripts in interview- or telephone-based questionnaires.[31]
- Ensuring that patients complete questionnaires at the same pre-specified time point, usually selected so as to avoid the undue influence of a preceding event.[31], [33]
Where a trial participant is unable to complete their PRO questionnaire, a proxy (commonly a partner or close relative) may be asked to complete the form on their behalf. Discussion surrounding the role of proxies represented 16.5% of in-trial guidelines.[1], [5], [6], [23], [29], [31], [32] Authors mainly highlighted the situations in which proxy assessment was justified.[1], [5], [6], [29], [31], [32] The use of a proxy was generally promoted as a last resort [1], [6], [29], however it was acknowledged that proxy data was better than no data at all.[5], [29] The ideal identity of the proxy was discussed by two authors, who concluded that, if possible, the same person should be used throughout the trial[29] and they should be close enough to the patient to provide valid data.[32] Guidelines for the reporting of data collection represented 9.7% of in-trial content[5], [6], [10], [26], [29], [33] and were primarily concerned with the need to document reasons for non-compliance[5], [6], [10], [33] and the need to report whether or not a proxy was used[6], [29]. A small number of in-trial guideline statements (7.8%) focused on patient information, endorsing the use of a supplementary leaflet for patients to take home[6], and highlighting the importance of the investigator in ensuring the patient fully understands the role of PRO measurement.[34] Two papers by the same author[5], [6] presented guidelines suggesting that PRO data should not be used to influence management during a trial and one paper suggested that trial participants ought to be informed when data would be used for the benefit of future patients only.[10]
Pre-trial.
The majority of pre-trial guidelines (87%) were focused on study design, procedural issues (including training logistics) and the evaluation/selection of appropriate PRO measures.[1], [5]–[7], [10], [19], [23]–[34] Others (12.8%) were concerned with questionnaire development and validation, or with issues arising from questionnaire modification.[1], [7], [10], [19], [23], [24], [28]–[33]
Post-trial.
Most post-trial guidelines (66.7%) concentrated on data analysis, reporting and presentation issues.[1], [5], [7], [10], [19], [23], [24], [28]–[33] The remaining guidance in this area (33.3%) surrounded the interpretation of PRO data and related labeling claims.[5], [7], [19], [23], [24], [28]–[33]
Discussion
The purpose of this review was to investigate whether anecdotal claims (subsequently confirmed by data under review), highlighting a lack of in-trial PRO guidance, reflect a deficiency in the published literature in this area. Our main findings suggest there a minimal guidelines in publication focused on in-trial PRO activity and there are a complete lack of guidelines addressing the management of concerning PRO data.
Of the small number of in-trial guidelines that are in circulation, the majority appear to deal with the procedural issues associated with the prevention of missing data. This focus may be understandable given the detrimental effect missing data may have on a trial. Trial reports indicate that PRO questionnaires are commonly returned with incomplete entries and some may not be returned at all.[7] This data may not be missing at random and it represents a serious potential bias when present.[10] Therefore, it is encouraging there is some consensus in the guidelines reviewed. To reduce missing PRO data, authors recommended that:
- The investigator/research nurse should: (1) motivate the patients to complete all questionnaires in-full by ensuring they understand the purpose and importance of the PRO assessment within the trial, (2) check questionnaires for completeness and prompt patients to fill in any missing items, (3) show appreciation for the efforts of the patient in completing the questionnaire.[1], [5], [6], [25], [27]–[29], [33], [34]
- PRO data is best collected in clinic, in an environment that is private and free from distraction.[1], [24], [29], [34]
- A centrally managed PRO data monitoring system should be in place, coordinated at each site by a named individual, tasked with; evaluating compliance across trial locations, issuing data collection reminders to patients where needed and chasing-up missing items.[1], [6], [10], [25], [27], [28]
The guidance surrounding missing data is therefore comprehensive. In contrast, no guidelines appear to adequately address aspects surrounding the management of concerning PRO data. This may be a problem given this issue has been identified as key by those involved in PRO data collection, as it can result in dual-role tension and may risk the potential introduction of bias into a trial.
A PRO questionnaire may be the only outcome within a trial capable of identifying ‘tolerable’ symptoms such as participant anxiety or depression; and the research nurse checking the form may be the only individual to whom participants have disclosed how they feel. Understandably, nurses may feel it is their duty to intervene when faced with PRO data that raises concern for the participant. A problem arises if the intervention is non-medicinal; for example, words of comfort, or advice to visit one's general practitioner, or if the advice results in the participant self-medicating. Direct medicinal interventions are far more easily controlled-for during data analysis. Non-medicinal or self-directed interventions, that are selectively delivered in response to concerning PRO data, may influence patient well-being but remain unrecorded in the trial documentation: this may represent a hitherto unforeseen source of bias.
Research nurses have reported experiencing dual-role tension when handling PRO data. Dual-role tension arises when an individual's values and responsibilities as a researcher conflict with those associated with being a clinical practitioner. Assuming ethical norms have been followed and participant ‘risk and burden’ does not outweigh the potential benefit of trial participation [35], the nurse researcher may justifiably choose not to intervene when concerning PRO data is disclosed, in order to protect trial integrity. This decision may be driven by consequentialist values, geared toward achieving the greatest benefit at the lowest cost, and reasoning that the benefits of producing unbiased trial results outweigh the personal costs experienced by the ‘few’ participants who continue to (tolerably) suffer. Conversely, nurse practitioners are obliged to make the care of their patients their first concern, as outlined in the Nursing and Midwifery Council code of conduct[36], which compels them to take steps to address any evident suffering. This conflict between the two professional duties has been recognized elsewhere[37]–[39]. However, what sets PRO data collection apart from the management of other trial outcomes is the current lack of published, and trial-based, guidance in this area. In our experience, the trial protocol often contains clear guidelines surrounding the levels at which some clinical outcomes, blood pressure for example, need to reach before the data collector should become concerned.[9] There is usually also a clear system in place to manage participants whose clinical measurements exceed agreed limits. Equivalent guidance is not always provided for PROs. Thus, the researcher collecting/inputting PRO data may be left to determine independently, on a case-by-case basis, whether PRO results signal a risk to the participant that outweighs the benefit of trial involvement. We believe this situation places unreasonable demands upon the researcher and promotes inconsistency, as there is unlikely to be uniformity in decision-making across trial sites; this may adversely affect data quality. Our findings highlight the need to develop and publish specific guidelines that clearly outline how concerning PRO data should be handled, as there are none currently in circulation. PRO in-trial guidelines should be brought in line with those covering traditional clinical outcomes and should define the conditions under which the researcher may take remedial action, and the form this intervention might take.
Limitations
Non-English language papers were excluded from the review, which potentially lessens the generalisability of the results presented. However, this decision was taken as a key element of qualitative content analysis involved determining the implied or latent meaning of the material.[18] We questioned the validity of such analysis using material translated from the original language by a third party, as some latent meaning may be lost during the translation process. Our search strategy dictated that we carefully reviewed papers for their guideline content only if their title/abstract gave an indication that some aspect of in-trial activity might be discussed. It is possible that papers providing ‘in-trial’ guidance exist, which make no reference to in-trial activity in their title or abstract.
Conclusions
In-trial guidelines aimed at PRO recruitment, data collection and data inputting within clinical trials are lacking. No guidance appears to exist for researchers involved with the handling of concerning PRO data. This is a worry as this activity may be associated with considerable personal and professional anxiety and may risk the introduction of bias when the ethical tension generated, is resolved in favour of responding to the needs of the patient over the expectations of the trial. Further research is needed to produce guidelines aimed at supporting researchers so they can deal effectively with dual-role tensions, manage PRO data appropriately and facilitate unbiased data collection.
Acknowledgments
We would like to thank the National School of Primary Care Research and the MRC Midland Hub for Trials Methodology Research for assistance with the development of this review.
Author Contributions
Conceived and designed the experiments: DK MC HD JI. Performed the experiments: DK AG. Analyzed the data: DK CL. Wrote the paper: DK. Critical revision of manuscript: DK MC HD JI CL AG. Obtained funding: DK MC HD JI.
References
- 1. Calvert M, Freemantle N (2004) Use of health-related quality of life in prescribing research. Part 2: methodological considerations for the assessment of health-related quality of life in clinical trials. Journal of Clinical Pharmacy and Therapeutics 29: 85–94.
- 2. Lemieuz J, Goodwin PJ, Bordeleau LJ, Lauzier S, Theberge V (2011) Quality-of-life measurement in randomized clinical trials in breast cancer: An updated systematic review (2001–2009). Journal of the National Cancer Institute 103: 178–231.
- 3. Sajid MS, Tonsi A, Baig MK (2008) Health-related quality of life measurement. International Journal of Health Care 21: 365–373.
- 4. Calvert MJ, Freemantle N (2003) Use of health-related quality of life in prescribing research. Part 1: why evaluate health-related quality of life? Journal of Clinical Pharmacy and Therapeutics 28: 513–521.
- 5. Fayers P (1995) MRC quality of life studies using a daily diary card--practical lessons learned from cancer trials. [Review] [42 refs]. Quality of Life Research 4: 343–352.
- 6. Fayers PM, Hopwood P, Harvey A, Girling DJ, Machin D, et al. (1997) Quality of life assessment in clinical trials--guidelines and a checklist for protocol writers: the U.K. Medical Research Council experience. MRC Cancer Trials Office. European Journal of Cancer 33: 20–28.
- 7.
FDA (2009) Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. US Food & Drug Administration Website. Available: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. Accessed 1st March, 2012.
- 8. Chan A, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, et al. (2013) SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 346: 1–42.
- 9.
Kyte D, Draper H, Ives J, Keely T, Calvert M(under review) Inconsistencies in Health-Related Quality of Life (HRQL) data-collection in clinical trials – a potential source of bias? Interviews with research nurses and trialists.
- 10. Basch EM, Abernethy A, Mullins CD, Tiglao MR, Tunis SR (2011) Development of a guidance for including patient-reported outcomes (PROs) in post-approval clinical trials of oncology drugs for comparative effectiveness research (CER). Value in Health 14(3): A10.
- 11. Calvert M, Blazeby J, Altman DG, Revicki D, Moher D, et al. (2013) Reporting of Patient Reported Outcomes in Randomised Trials: the CONSORT PRO Extension. JAMA 309(8): 814–22 .
- 12.
FDA (2012) US Food & Drug Administration Website. Available: http://www.fda.gov/. Accessed 1st March, 2012.
- 13.
EMA (2012) European Medicines Agency website. Available: http://www.ema.europa.eu/ema/. Accessed 1st March, 2012.
- 14.
GMC (2012) General Medical Council website. Available: http://www.gmc-uk.org/. Accessed 1st March, 2012.
- 15.
MRC (2012) Medical Research Council website. Available: http://www.mrc.ac.uk/index.htm. Accessed 1st March, 2012.
- 16.
RCN (2012) Royal College of Nursing website. Available: http://www.rcn.org.uk/. Accessed 1st March, 2012.
- 17.
Oxford University Press (2012) Oxford Dictionairies Online. Oxford University Press, UK. Available: http://oxforddictionaries.com. Accessed 1st March, 2012.
- 18.
Schreier M (2012) Qualitative content analysis in practice. London: SAGE Publications Ltd.
- 19. Baiardini I, Bousquet PJ, Brzoza Z, Canonica GW, Compalati E, et al. (2010) Recommendations for assessing patient-reported outcomes and health-related quality of life in clinical trials on allergy: a GA(2)LEN taskforce position paper. Allergy 65: 290–295.
- 20. Cohen J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37–46.
- 21. De Vries H, Elliott MN, Kanouse DE, Teleki SS (2008) Using Pooled Kappa to Summarize Interrater Agreement across Many Items. Field Methods 20: 272–282.
- 22. Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Medicine 6: e1000097.
- 23. Fletcher A (1995) Quality-of-life measurements in the evaluation of treatment: proposed guidelines. British Journal of Clinical Pharmacology 39: 217–222.
- 24. Fletcher A, Gore S, Jones D, Fitzpatrick R, Spiegelhalter D, et al. (1992) Quality of life measures in healthcare. II: Design, analysis, and interpretation. BMJ 305: 1145–1148.
- 25. Hopwood P, Harvey A, Davies J, Stephens RJ, Girling DJ, et al. (1998) Survey of the Administration of quality of life (QL) questionnaires in three multicentre randomised trials in cancer. The Medical Research Council Lung Cancer Working Party the CHART Steering Committee. European Journal of Cancer 34: 49–57.
- 26. Kiebert GM, Curran D, Aaronson NK (1998) Quality of life as an endpoint in EORTC clinical trials. Statistics in Medicine 17: 561–569.
- 27. Moinpour CM, Feigl P, Metch B, Hayden KA, Meyskens FL Jr, et al. (1989) Quality of life end points in cancer clinical trials: review and recommendations. Journal of the National Cancer Institute 81: 485–495.
- 28. Movsas B (2003) Quality of life in oncology trials: a clinical guide. Seminars in Radiation Oncology 13: 235–247.
- 29. Chassany O, Sagnier P, Marquis P, Fullerton S, Aaronson N (2002) Patient-reported outcomes: The example of health-related quality of life - A European guidance document for the improved integration of health-related quality of life assessment in the drug regulatory process. Drug Information Journal 36: 209–238.
- 30. FDA (2006) Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health & Quality of Life Outcomes 4: 79.
- 31. Leidy NK, Revicki DA, Geneste B (1999) Recommendations for Evaluating the Validity of Quality of Life Claims for Labeling and Promotion. Value in Health 2: 113–127.
- 32. Revicki DA, Osoba D, Fairclough D, Barofsky I, Berzon R, et al. (2000) Recommendations on health-related quality of life research to support labeling and promotional claims in the United States. Quality of Life Research 9: 887–900.
- 33. Luo X, Cappelleri JC (2008) A practical guide on incorporating and evaluating patient-reported outcomes in clinical trials. Clinical Research and Regulatory Affairs 25: 197–211.
- 34. Poulter K (1997) The administration of quality of life questionnaries within clinical trials. Good Clinical Practice Journal 4: 22–25.
- 35.
WMA (2008) World Medical Association Delaration of Helsinki: Ethical pricinples of medical research involving human subjects. World Medial Association: 1-5 p.
- 36.
NMC (2008) The code: Standards of conduct, performance and ethics for nurses and midwives. The Nursing & Midwifery Council website. Available: http://www.nmc-uk.org/Documents/Standards/The-code-A4-20100406.pdf. Accessed 1st March, 2013.
- 37. Edwards M, Chalmers K (2002) Double agency in clinical research. Canadian Journal of Nursing Research 34: 131–142.
- 38. Houghton CE, Casey D, Shaw D, Murphy K (2010) Ethical challenges in qualitative research: examples from practice. Nurse Researcher 18: 15–25.
- 39. Holloway I, Wheeler S (1995) Ethical issues in qualitative nursing research. Nursing Ethics 2: 223–232.