You are here

  • Home
  • Archive
  • Volume 6, Issue 6
  • Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review

Article Text

Article menu

Download PDFPDF

XML
Research methods
Research
Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review
  1. Rebecca Mercieca-Bebber1,2,
  2. Michael J Palmer3,
  3. Michael Brundage3,
  4. Melanie Calvert4,
  5. Martin R Stockler1,5,
  6. Madeleine T King1,2
  1. 1Central Clinical School, Sydney Medical School, University of Sydney, Sydney, New South Wales, Australia
  2. 2Psycho-oncology Co-operative Research Group, School of Psychology, University of Sydney, Sydney, New South Wales, Australia
  3. 3Department of Cancer Care and Epidemiology, Cancer Research Institute, Queen's University, Kingston, Ontario, Canada
  4. 4Institute of Applied Health Research, University of Birmingham, Birmingham, UK
  5. 5NHMRC Clinical Trials Centre, University of Sydney, Camperdown, New South Wales, Australia
  1. Correspondence to Rebecca Mercieca-Bebber; Rebecca.Mercieca{at}sydney.edu.au

Abstract

Objectives Patient-reported outcomes (PROs) provide important information about the impact of treatment from the patients' perspective. However, missing PRO data may compromise the interpretability and value of the findings. We aimed to report: (1) a non-technical summary of problems caused by missing PRO data; and (2) a systematic review by collating strategies to: (A) minimise rates of missing PRO data, and (B) facilitate transparent interpretation and reporting of missing PRO data in clinical research. Our systematic review does not address statistical handling of missing PRO data.

Data sources MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL) databases (inception to 31 March 2015), and citing articles and reference lists from relevant sources.

Eligibility criteria English articles providing recommendations for reducing missing PRO data rates, or strategies to facilitate transparent interpretation and reporting of missing PRO data were included.

Methods 2 reviewers independently screened articles against eligibility criteria. Discrepancies were resolved with the research team. Recommendations were extracted and coded according to framework synthesis.

Results 117 sources (55% discussion papers, 26% original research) met the eligibility criteria. Design and methodological strategies for reducing rates of missing PRO data included: incorporating PRO-specific information into the protocol; carefully designing PRO assessment schedules and defining termination rules; minimising patient burden; appointing a PRO coordinator; PRO-specific training for staff; ensuring PRO studies are adequately resourced; and continuous quality assurance. Strategies for transparent interpretation and reporting of missing PRO data include utilising auxiliary data to inform analysis; transparently reporting baseline PRO scores, rates and reasons for missing data; and methods for handling missing PRO data.

Conclusions The instance of missing PRO data and its potential to bias clinical research can be minimised by implementing thoughtful design, rigorous methodology and transparent reporting strategies. All members of the research team have a responsibility in implementing such strategies.

  • patient-reported outcomes
  • health-related quality of life
  • missing data
  • quality assurance
  • systematic review
  • methodology

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2015-010938

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This systematic review collates practical strategies to minimise the problem of missing patient-reported outcome (PRO) data. Recommendations were retrieved from 117 multidisciplinary sources and potential drawbacks of each recommendation are presented.

    • Missing PRO data may be preventable in many cases by implementing rigorous study design and methodological strategies, as described in this review.

    • In some clinical research settings, missing PRO data is not avoidable due to deteriorating health status of the participants. Strategies to minimise the potential for bias caused by missing PRO data are described.

    • This paper discusses one aspect of PRO data quality: data completeness. Many other factors also contribute to high-quality PRO data, including but not limited to appropriateness of PRO measures, timing of PRO assessment, ensuring patients self-complete and clinical versus statistical significance of findings.

    • This review excludes non-English sources. The non-English publications may have been relevant; however, given the repetition of themes found in our 117 included sources we do not believe that these would significantly affect our findings.

    Patient-reported outcomes (PROs), including health-related quality of life (HRQOL) and specific symptoms, provide unique information about the effect of disease and treatment on the patient. PRO research evidence is crucial for informed clinical and policy decision-making, and is increasingly being used to inform labelling claims for medical products.1–3 The quality and value of PRO evidence is contingent on a number of factors, including: provision of a clear rationale for PRO assessment, the choice of PRO measure, the timing of PRO assessments, and ensuring the responses are the patient's own. One critical PRO quality assurance issue is missing data, defined as “…values that are not available and that would be meaningful for analysis if they were observed” (ref. 4, p. 1355). Conversely, researchers may measure ‘PRO assessment compliance’, which refers to the number of completed questionnaires received as a proportion of the number expected, given the study design, and the number of patients still alive and enrolled in the study.5 ,6 Both definitions acknowledge that questionnaires are not expected from patients who have died.4–6

    The practical and methodological issues associated with missing PRO data received considerable attention in the literature in the 1990s. An expert workshop on the prevention and analysis of missing PRO data in trials led by international cancer trials groups was held in 1996, with findings published in a dedicated special issue of Statistics in Medicine.7 Yet problems with missing PRO data persist; high rates of missing PRO data continue to be reported in clinical trials,8–10 and PRO compliance rates are sometimes so poor that PRO data are not analysed.11

    Persisting PRO compliance problems may reflect the sporadic attention the issue has received in the literature over the past 20 years,4 most of which is targeted to statisticians handling missing PRO data during analysis. This is problematic for four reasons: first, content targeted at statisticians may be conceptually and technically inaccessible to non-specialists; second, content addressing statistical handling of missing data does not acknowledge that some missing PRO data is preventable through study design and implementation; third, it promotes an attitude that the problem of missing data is the sole responsibility of the statistician; and fourth, appropriate statistical handling of missing PRO data is often contingent on other research data, and this will require consideration at the trial design stage. The broader research team should understand the issues associated with missing data, and their role in minimising related problems. This team includes individuals involved in study design and planning; recruitment; data collection; quality assurance; and analysing, interpreting or reporting of the results. To the best of our knowledge, there has not been a systematic review targeting the role of the broader research team in maximising PRO compliance rates, and minimising the problem of missing PRO data.

    This paper has two aims, and is accordingly structured in two parts:

    1. To summarise the problems created by missing PRO data in a format accessible to anyone involved in designing, conducting or analysing clinical research.

    2. To systematically review the multidisciplinary literature to identify and collate strategies relevant to the entire research team to:

      1. Maximise PRO compliance rates through study design and implementation;

      2. Reduce the potential for biased interpretation caused by missing PRO data through PRO-specific strategies for research design, implementation and reporting.

    Part 1: the problem of missing PRO data—a summary of the issues

    Missing PRO data create challenges for data analysis, and can compromise the interpretability and value of PRO findings for three major reasons: first, missed observations reduce study power.12 Studies with secondary PRO end points are usually sufficiently powered for PRO analyses when the sample size calculation is based on a survival primary end point (eg, progression-free survival) because these typically require larger sample sizes. However, a high proportion of missing PRO data will substantially reduce power and inflate standard error.13 This increases the risk of type 2 errors, that is, false-negative findings.

    Second, and more problematically, missing data may be related to the measured outcome (ie, HRQOL, pain, etc).12 For example, non-completers who dropped out of Southwest Oncology Group trials due to death had worse HRQOL at baseline, and at time of drop out than other participants.5 In many cases, this type of missing PRO data is unavoidable, yet it cannot be ignored as doing so may lead to biased estimates—the extent of which is impossible to calculate.13

    Third, the presence of missing data undermines randomisation, and makes intention-to-treat analyses (analysing according to randomised groups) less valid as missing data create a need to make assumptions about the data that are not always verifiable.14

    Difficulties in statistically handling missing PRO data

    There are many options for statistically handling missing PRO data. Each method makes assumptions about the missing data mechanism,15 which is a fairly technical system for classifying missing data according to their probable cause (see box 1). The challenge is to handle missing data in a way that closest resembles the true, albeit unverifiable, missing data mechanism, since the mechanism often has a greater impact on research results than does the proportion of missing data.16 To use a simple example—if PRO data are truly missing not at random (MNAR; eg, missing due to declining health), but the analysis method used assumes missing data are missing completely at random (eg, missing due to institution error) by excluding cases with missing data, then the analysed data represents only the better-performing patients. Therefore, in addition to some loss of study power, the findings may falsely indicate that PROs are more favourable than is the true case, thus potentially leading to biased interpretation of change over time within groups, or of between-group differences.13 If the missing data appear MNAR, and are handled and interpreted sensibly (within the specific clinical and study context), the risk of introducing bias is reduced. Although statistical approaches are available, it is critical to prevent missing data, where possible, rather than to rely solely on statistical approaches. Prevention, statistical handling, interpretation and transparent reporting of missing PRO data are complementary strategies. It is recommended that statistical handling of missing PRO data be undertaken by a statistician as the methods used are technical. Therefore, statistical handling of missing PRO data is not addressed in our systematic review below. Interested readers are referred to Bell and Fairclough17 for detailed discussion.

    Box 1

    The missing data mechanism

    ▸ Missing completely at random (MCAR)

    The probability of missing data is unrelated to past, current and future patient-reported outcome (PRO) scores/health status such as administrative errors.18 MCAR assumes the participants with missing data are a random sample of the whole sample.18 Therefore, assuming the study is adequately powered, the results should not be altered too much if the MCAR are ignored in analysis; however, the standard error of the estimates will be inflated.19 Many examples of MCAR are caused by poor study design and implementation, and are hence ‘preventable’ sources of missing PRO data.

    ▸ Missing at random (MAR)

    The probability of missing data depends on observed data or a fixed covariate, but not on the current (missing) or future PRO scores; for example, if a particular cultural group has a high proportion of missing data and patients from this group tend to have poorer PRO scores.13 Depending on whether the variable contributing to the likelihood of missing data is ‘informative’ (related to measured health outcome) or ‘ignorable’ (unrelated), using a statistical method that ignores MAR may distort the findings, potentially introducing bias.19 MAR is difficult to ascertain, but methods are available to test for (albeit with some uncertainty12 ,20) and analyse MAR PRO data.12 ,21

    ▸ Missing not at random (MNAR)

    The probability of missing data depends on current and future unobserved scores. PRO scores previously observed are constant but would decline at (or after) drop out, and the process of decline is not observed.18 Data that meet the MNAR assumption are always ‘informative’, that is, missing due to the patient's declining health status, but the extent of decline is not known because it is not observed. Few methods are available for unbiased analysis of MNAR.21

    Part 2: a systematic review of strategies to maximise PRO compliance rates and reduce the potential for bias

    Part 1 of this paper summarised the problem of missing PRO data for the analysis and interpretation of study results. This motivates part 2 of our paper: a systematic review of strategies for all research team members to assist in minimising the problem of missing PRO data.

    Systematic review methods

    Search strategy

    MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL) databases were systematically searched using a search strategy (see online supplementary appendix A) which combined PRO terms with missing data and compliance terms. These databases were chosen as they canvassed the disciplines of interest to our review, and because they indexed key papers already known to the authors. The search strategy was developed by first reviewing literature to identify key search terms. We sought advice from three librarians with expertise in systematic reviews to ensure all relevant Medical Subject Headings (MeSH) were addressed, and conducted several pilot searches to capture targeted papers. The MEDLINE search was restricted to English language articles. Reference lists and citations of included papers retrieved in the database search were screened (by title) for additional relevant sources, using the same eligibility criteria.

    Eligibility criteria

    Papers were included if they provided guidance or recommendations for minimising/preventing missing PRO data in prospective research designs, or for transparent interpretation and reporting of missing PRO data to minimise risk of potential interpretation bias. We excluded non-English articles; conference presentations; research protocols; papers discussing statistical handling of missing PRO data, instrument development, proxy-reporting, patient-reported behaviours (smoking, drug use, etc), non-patient samples and papers reporting general study/trial drop-out rates.

    Study selection

    Two reviewers (RM-B and MJP) independently screened article titles and abstracts using the eligibility criteria. Screening discrepancies were discussed and settled with two senior authors (MB and MTK). Abstracts that appeared to meet the criteria were obtained in full text and assessed against the same criteria. Our search and study selection process complied with Preferred Reported Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines22 (see online supplementary appendix B).

    Extraction and coding of recommendations

    Recommendations were extracted, coded and analysed using framework synthesis methodology (RM-B).23 ,24 An a priori framework was used to organise recommendations into three categories (study design and planning, during active study, reporting), then coded according to the specific recommendation (eg, minimise patient burden). These codes were refined and developed during the process, and organised into three code levels on completion. For example, the major category of ‘minimise patient burden’ was subcategorised into ‘assistance to patients’, ‘questionnaire content’, ‘length of assessments’ and ‘validated questionnaires’. Each subcategory was further categorised for specificity; for example, the third-level categories for ‘length of assessments’ includes ‘fewer assessments’, ‘shorter questionnaire’, ‘use screening questions’, etc. Three reviewers (MTK, MJP, MB) each checked 10% of extractions. Frequencies of each unique recommendation were calculated, and potential drawbacks of each recommendation were described. Two reviewers (MJP, MTK) checked 100% of the final results tables. Disagreements were discussed as a team to achieve consensus.

    Results

    One hundred and seventeen articles (listed in online supplementary appendix C) met the inclusion criteria (figure 1). These arose from oncology, palliative care and other disease-specific and non-disease-specific PRO literature (table 1).

    View this table:
    Table 1

    Characteristics of included sources

    Figure 1
    Figure 1

    PRISMA flow diagram. PRO, patient-reported outcome.

    Design strategies to minimise the problem of missing PRO data

    Recommendations for reducing the problem of missing PRO data through study design are summarised in 12 categories in table 2: PRO assessment schedule: a clinically informative and feasible assessment schedule should be defined, with acceptable assessment time windows and stopping rules; collection of auxiliary or supporting data: collect information to facilitate unbiased interpretation of PRO data in the presence of missing data, such as clinician-rated health status, observational or proxy-reported data; eligibility criteria: include literacy and language requirements, and the need for a valid baseline PRO assessment; feasibility issues: considerations for determining required resources and ensuring the PRO study is feasible; guidance: for trial team members to standardise administration and maximise PRO completion rates; mode of questionnaire administration (MOA): MOA should be feasible and acceptable, and impact on PRO completion rates should be considered; minimise participant burden: employ strategies to ensure PRO assessment is easy and acceptable to participants; PRO measure: PRO measures should be clinically relevant, validated, and acceptable to patients; PROs part of the trial: incorporate PROs into all relevant study documents and ensure the team is committed to the PRO study; quality assurance: prepare databases, study guidance and procedures with ongoing quality assurance in mind; sample: ensure the PRO sample size is representative and sufficient for planned analyses; team involved in design/protocol development: involve a multidisciplinary team, including PRO experts, clinicians, nurses, site coordinators, patients and others.

    View this table:
    Table 2

    Study design and planning strategies to minimise the problem of missing PRO data

    The five most frequently recommended design strategies were: baseline PRO completion as an eligibility criterion (n=28), develop guidance for site staff to standardise the administration of PRO questionnaires(n=27), minimise the length of questionnaires to reduce patient burden (n=18), align PRO assessment time points to clinic visits (n=16) and ensure recruiting sites have sufficient resources to run the PRO study (n=15).

    Implementation strategies to minimise the problem of missing PRO data

    Recommendations for minimising the problem of missing PRO data while the PRO study is active were coded into seven categories in table 3: administration procedures: standardised procedures, particularly for site staff, to maximise PRO compliance; patient education and engagement: education about the value of PROs in the study, and engagement through study updates or incentives; maintaining patient records: contact details and health status should be kept updated; quality assurance: procedures and active communication to monitor compliance and intervene if issues are apparent; site coordinator: appoint an individual responsible for PRO assessment at recruiting sites with appropriate organisational and communication skills; team involved in study implementation: broader trial team must stay engaged and committed to the PRO study, and work together towards its successful completion; and staff training: provide initial and ongoing training about PROs, communication skills, methodology; and formats of such training. The most frequently recommended implementation strategies were: use a PRO completion cover sheet for standardised recording of reasons for missing PRO data (n=39), appoint a site coordinator responsible for PRO assessments (n=33), send reminders about upcoming PRO assessments to site staff (n=30), ensure site staff check completed PRO questionnaires for missed items while the patient is still in the clinic (n=29) and centrally monitor PRO compliance in real-time (n=27).

    View this table:
    Table 3

    Study conduct strategies to minimise the problem of missing PRO data

    Strategies for reporting studies with missing PRO data

    Strategies for reporting studies with missing PRO data are presented in table 4. These addressed a need for clearly reported methodology, including analysis methods; describing the sample, including baseline scores; defining and providing compliance rates; comparing participants with and without missing PRO data; providing reasons for missing data and discussing the impact of missing data on generalisability of findings. The most frequently recommended details to report were: rates of missing PRO data (n=26), reasons/types of missing PRO data (n=15), how missing data were handled for the analysis (n=9), discussion of the potential for bias caused by missing PRO data (n=6), and clinical and demographic characteristics of the sample, including baseline PRO scores (n=5).

    View this table:
    Table 4

    Strategies for reporting studies with missing patient-reported outcome (PRO) data to minimise the potential for biased interpretation of findings

    Discussion

    This paper summarises the problems created by missing PRO data, and highlights the need for all members of the research team to assist in minimising the problem of missing data. Our systematic review identified and synthesised a range of practical strategies for all research team members to maximise PRO compliance and reduce the problem of missing PRO data through design, implementation and reporting. These strategies highlight the need for thoughtful planning and incorporation of PROs into all research documents.25–30 PRO study design should balance the need for sufficient PRO data with the capacity of patients to self-report, and the feasibility and practicality of site staff to collect it at informative time points.31–33 Previous research has demonstrated that involving experienced data collection personnel in PRO study development is crucial to achieving high compliance rates.25 ,34 Strategies for minimising bias caused by missing PRO data involve utilising auxiliary data to inform valid analysis according to the likely missing data mechanism; this must be planned for during study design.

    While the PRO study is active, high-level support of the sponsor and advocacy by the PRO expert on the research steering committee (or similar) is essential to emphasise the importance of PRO data. Given the time-sensitive nature of PRO data, quality assurance strategies are crucial to maintaining high standards, particularly real-time monitoring of PRO completion rates to enable timely intervention if compliance falls below prespecified thresholds.35 Land et al25 found that targeted communication with poorly performing sites led to reductions in rates of missing baseline PROs. Many trial groups have reported success of centralised monitoring systems for maintaining high PRO completion rates.29 ,35–37

    Staff should have access to ongoing training and written guidance, and should understand the importance of PROs.5 ,26 ,32 ,34 ,37–42 The National Cancer Institute of Canada Clinical Trials Group (NCIC CTG) has attributed high PRO completion rates to training the trial team about the importance of avoiding missing PRO data.43 Patient engagement is also crucial. Hellard et al44 found that sending participants’ study updates was the primary reason for high-level participant engagement and retention, and weekly study diary completion rates of 90.7% over 68 weeks. All of these recommendations require intensive resources45 ,46 and research team commitment,47 and highlight the importance of conducting appropriate feasibility checks before activating the study. Research investigators, sponsors and funding bodies have a responsibility to ensure research funds are allocated to quality assurance of PRO studies. Training regarding the importance and efficacy of specific quality assurance strategies may be the catalyst to securing such funding.

    Rouette et al48 found that 86% of clinicians surveyed considered missing data important in interpreting PRO findings, and that clinicians require clear summaries and recommendations for accurate interpretation of trial results. Clear and sufficient information should be reported, so readers can meaningfully interpret the possible impact (bias) of missing PRO data on findings, which is crucial for PROs to impact patient care. This involves reporting descriptions of the study sample, including baseline PRO scores; rates and reasons for missing PRO data; analysis methods, including sensitivity analyses and analysis assumptions, handling of missing data, and discussing the potential impact of missing data on PRO findings. These reporting recommendations are also addressed in the CONsolidated Standards of Reporting Trials (CONSORT) PRO extension, underscoring their importance to transparency of reporting.49

    Systematic reviews have highlighted that methods for handing missing PRO data are often incorrectly or simply not applied,10 ,14 and the extent and handling of missing PRO data is often unreported.8 ,10 ,49–51 These omissions may hinder the reader from being able to interpret the impact of missing data on findings. Journal editors should enforce reporting guidance such as CONSORT-PRO49 in order to promote and maintain a high standard of research evidence. A recent study found that 31% of reviewed RCTs failed to report PRO results despite including PRO endpoints in the trial protocol.52 The authors could not determine reasons why the RCTs failed to report PROs; however, high rates of missing data have discouraged investigators from publishing PRO findings previously.11 This represents a waste of research resources, participants' time and limited research funding as PRO findings left unreported cannot impact patient care.53 Trial registration and publication of research protocols is a motion towards avoiding such examples of publication bias; however, further action towards improving the quality of PRO data is needed, beginning with more comprehensive training about PROs for all research staff. Thus, there is an urgent need for research teams to implement the described strategies to minimise missing PRO data and when missing data are present, to reduce its impact on the quality and dissemination of results.

    Strengths

    The literature on missing PRO data largely comprises statistically technical material that may be inaccessible for non-statisticians. We have summarised the problems created by missing PRO data in a format accessible to anyone involved in designing, conducting or analysing a clinical study. In response to the need for all members of the research team to assist in minimising the problem of missing data, we have provided the first systematic review to collate practical strategies to minimise the problem of missing PRO data. A comprehensive search strategy was used, developed with assistance from field experts and librarians. The review includes recommendations from a substantially large number of sources from various health disciplines. Many were discussion pieces written by highly regarded and experienced PRO experts based on strategies that their trials group or organisations have implemented, with documented improvement in PRO completion rates. This review, therefore, brings together the collective wisdom of experienced opinion leaders in the field. Further, most recommendations are generalisable across disciplines. Patterns and similarities in the recommendations extracted, as well as emerging findings of ongoing work investigating causes of missing PRO data,54 provide evidence of their effectiveness in preventing and addressing the missing PRO data problem.

    Limitations

    As the majority of papers included in our systematic review were discussion or guidance pieces rather than original research reports, we were unable to apply study quality criteria used in traditional systematic reviews to the source papers. However, we did consider potential limitations of each recommendation, which is useful information for researchers considering implementing these strategies. Further, we have cited the frequency of each recommendation. High frequency may indicate widespread use and effectiveness, although we do acknowledge that some less-cited strategies may also be highly effective, and some strategies may only apply to specific disease or research contexts. Gathering empirical evidence as to the degree of effectiveness of the strategies identified in this review would be an interesting direction for future research.

    Despite our efforts in extensively hand-searching reference lists and citing articles, it is possible that relevant sources and/or recommendations were missed. We restricted our database search to MEDLINE and CINAHL databases, and excluded non-English sources. Searching of non-English language databases may have identified additional publications; however, since many themes were identified by numerous sources, we do not believe that this would significantly affect our findings.55 ,56 Coding of recommendations was a subjective process and, as with all qualitative approaches, is subject to interpretation of the analysts; however, rounds of code checking ensured the original meaning of recommendations was retained as far as possible.

    This paper discusses one aspect of PRO data quality: data completeness. Many other factors contribute to high-quality PRO data, such as clinical and psychometric appropriateness of PRO measures (valid, reliable, responsive), compliance with time windows, and ensuring that patients self-complete.57 Likewise, many factors can contribute to invalid interpretation of PRO data, including multiple hypothesis testing57 and clinical versus statistical significance.58 ,59 Some of these issues have been addressed in the context of missing data in this review, but are independently crucial PRO assessment concerns. Readers are directed to the following sources for further guidance on PRO study design,57 protocol development,60 ,61 analysis17 ,57 and reporting49 ,62 of PRO studies.

    Implications

    We recommend that all members of the research team involved in designing, collecting, analysing and reporting PRO data implement the strategies outlined in this review to minimise the problem of missing PRO data. Missing PRO data are preventable in many cases through rigorous study design and methodology. Further guidance on PRO-specific content of trial protocols is required, and is currently under development in the form of a Standard Protocol Items for Clinical Trials (SPIRIT)-PRO extension.63 Significant funding, and staff and participant time is invested in PRO studies. Poorly conducted PRO studies with high rates of preventable missing data yield poor quality evidence. Funding organisations and sponsors should actively promote high-quality PRO research by mandating PRO training for research team members, and publication of PRO findings (adhering to CONSORT PRO extension where applicable) to optimise the value of PRO data and avoid research waste.53 ,63

    However, we acknowledge that in some health settings, missing PRO data are not avoidable due to deteriorating health status of the participants. We have also outlined strategies that may assist statisticians to appropriately handle unavoidable missing PRO data to minimise bias. Again, transparent and complete reporting of missing PRO data and analysis methods, as described in this review, will promote valid interpretation of PRO findings and assist investigators to make better-informed recommendations for patient care, policy and therapeutic labelling.

    Conclusion

    It is essential that all researchers involved in design, conduct, analysis and reporting of PRO data appreciate why missing data is a problem, why in many circumstances statistical methods for handling missing data are not failsafe, and how all members of the research team can assist in minimising the problem of missing PRO data, so that misunderstandings do not become a barrier to achieving the highest possible PRO completion rates. To not do so represents a great waste of research resources and valuable PRO evidence. Careful planning of PRO studies can minimise the risk and problem of missing PRO data. Ongoing quality assurance and team commitment throughout study implementation is also essential, which may be facilitated by involvement of PRO experts and sponsors. Despite the existence of missing PRO data, it is possible to make valid conclusions about the effect of disease and treatment on the patient if missing data are appropriately handled and analysed, and transparently reported.

    Acknowledgments

    The authors gratefully acknowledge the support of the librarians, Rebecca Goldsworthy and Jeremy Cullis (Faculty Liaison Librarians, Sydney Medical School, University of Sydney), and Shawn Hendrikx (Clinical Outreach Librarian, Queen's University) in developing the search strategy.

    References

    1. ↵
      1. Cherny NI,
      2. Sullivan R,
      3. Dafni U, et al
      . A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol 2015;26:1547–73. doi:10.1093/annonc/mdv249
      OpenUrlAbstract/FREE Full Text
    2. ↵
      Food and Drug Administration. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. FDA, 2009. http://www.fda.gov/downloads/Drugs/.../Guidances/UCM193282.pdf (accessed 10 Mar 2014).
    3. ↵
      European Medicines Agency. Oncology Working Party. Draft Reflection Paper on the use of patient reported outcome (PRO) measures in oncology studies. EMA, 2014. http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/06/WC500168852.pdf (accessed 2 Sept 2014).
    4. ↵
      1. Little RJ,
      2. D'Agostino R,
      3. Cohen ML, et al
      . The prevention and treatment of missing data in clinical trials. N Engl J Med 2012;367:1355–60. doi:10.1056/NEJMsr1203730
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Bernhard J,
      2. Cella DF,
      3. Coates AS, et al
      . Missing quality of life data in cancer clinical trials: serious problems and challenges. Stat Med 1998;17:517–32. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<517::AID-SIM799>3.0.CO;2-S
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Staquet M,
      2. Berzon R,
      3. Osoba D, et al
      . Guidelines for reporting results of quality of life assessments in clinical trials. Qual Life Res 1996;5:496–502. doi:10.1007/BF00540022
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Bernhard J,
      2. Gelber RD,
      3. Hürny C
      . Preface. Stat Med 1998;17:511–12. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<511::AID-SIM798>3.0.CO;2-C
      OpenUrl
    8. ↵
      1. Bylicki O,
      2. Gan HK,
      3. Joly F, et al
      . Poor patient-reported outcomes reporting according to CONSORT guidelines in randomized clinical trials evaluating systemic cancer therapy. Ann Oncol 2015;26:231–7. doi:10.1093/annonc/mdu489
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Efficace F,
      2. Fayers P,
      3. Pusic A, et al
      . Quality of patient-reported outcome reporting across cancer randomized controlled trials according to the CONSORT patient-reported outcome extension: a pooled analysis of 557 trials. Cancer 2015;121:3335–42. doi:10.1002/cncr.29489
      OpenUrlCrossRefPubMed
    10. ↵
      1. Fielding S,
      2. Maclennan G,
      3. Cook JA, et al
      . A review of RCTs in four medical journals to assess the use of imputation to overcome missing data in quality of life outcomes. Trials 2008;9:51. doi:10.1186/1745-6215-9-51
      OpenUrlCrossRefPubMed
    11. ↵
      1. Friedlander M,
      2. Mercieca-Bebber RL,
      3. King MT
      . Patient-reported outcomes (PRO) in ovarian cancer clinical trials-lost opportunities and lessons learned. Ann Oncol 2016;27(Suppl 1):i66–71. doi:10.1093/annonc/mdw080
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Fairclough DL,
      2. Peterson HF,
      3. Cella D, et al
      . Comparison of several model-based methods for analysing incomplete quality of life data in cancer clinical trials. Stat Med 1998;17:781–96. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<781::AID-SIM821>3.0.CO;2-O
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Fairclough DL,
      2. Peterson HF,
      3. Chang V
      . Why are missing quality of life data a problem in clinical trials of cancer therapy? Stat Med 1998;17:667–77. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<667::AID-SIM813>3.0.CO;2-6
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Bell ML,
      2. Fiero M,
      3. Horton NJ, et al
      . Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol 2014;14:118. doi:10.1186/1471-2288-14-118
      OpenUrl
    15. ↵
      1. Rubin DB
      . Inference and missing data. Biometrika 1976;63:581–92. doi:10.1093/biomet/63.3.581
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Tabachnick B,
      2. Fidell L
      . Using multivariate statistics. Needham Heights, MA: Allyn & Bacon, 2012.
    17. ↵
      1. Bell ML,
      2. Fairclough DL
      . Practical and statistical issues in missing data for longitudinal patient-reported outcomes. Stat Methods Med Res 2014;23:440–59. doi:10.1177/0962280213476378
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Curran D,
      2. Bacchi M,
      3. Schmitz SFH, et al
      . Identifying the types of missingness in quality of life data from clinical trials. Stat Med 1998;17:739–56. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<739::AID-SIM818>3.0.CO;2-M
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Dong Y,
      2. Peng CY
      . Principled missing data methods for researchers. Springerplus 2013;2:222. doi:10.1186/2193-1801-2-222
      OpenUrlCrossRefPubMed
    20. ↵
      1. Curran D,
      2. Molenberghs G,
      3. Fayers PM, et al
      . Incomplete quality of life data in randomized trials: missing forms. Stat Med 1998;17:697–709. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<697::AID-SIM815>3.0.CO;2-Y
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Fielding S,
      2. Fayers PM,
      3. Ramsay CR
      . Investigating the missing data mechanism in quality of life outcomes: a comparison of approaches. Health Qual Life Outcomes 2009;7:57. doi:10.1186/1477-7525-7-57
      OpenUrlCrossRefPubMed
    22. ↵
      1. Moher D,
      2. Liberati A,
      3. Tetzlaff J, et al
      . Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 2009;6:e1000097. doi:10.1371/journal.pmed.1000097
      OpenUrlCrossRefPubMed
    23. ↵
      1. Barnett-Page E,
      2. Thomas J
      . Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol 2009;9:59. doi:10.1186/1471-2288-9-59
      OpenUrlCrossRefPubMed
    24. ↵
      1. Gale NK,
      2. Heath G,
      3. Cameron E, et al
      . Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117. doi:10.1186/1471-2288-13-117
      OpenUrlCrossRefPubMed
    25. ↵
      1. Land SR,
      2. Ritter MW,
      3. Costantino JP, et al
      . Compliance with patient-reported outcomes in multicenter clinical trials: methodologic and practical approaches. J Clin Oncol 2007;25:5113–20. doi:10.1200/JCO.2007.12.1749
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Cella DF,
      2. Lloyd SR
      . Data collection strategies for patient-reported information. Qual Manag Health Care 1994;2:28–35. doi:10.1097/00019514-199402040-00006
      OpenUrlPubMed
    27. ↵
      1. Fallowfield LJ,
      2. Harper P
      . Health-related quality of life in patients undergoing drug therapy for advanced non-small-cell lung cancer. Lung Cancer 2005;48:365–77. doi:10.1016/j.lungcan.2004.11.018
      OpenUrlPubMed
    28. ↵
      1. Little RJ,
      2. Cohen ML,
      3. Dickersin K, et al
      . The design and conduct of clinical trials to limit missing data. Stat Med 2012;31: 3433–43. doi:10.1002/sim.5519
      OpenUrlCrossRefPubMed
    29. ↵
      1. Walker M,
      2. Brown J,
      3. Brown K, et al
      . Practical problems with the collection and interpretation of serial quality of life assessments in patients with malignant glioma. J Neurooncol 2003;63:179–86. doi:10.1023/A:1023900802254
      OpenUrlCrossRefPubMed
    30. ↵
      1. Taphoorn MJ,
      2. Sizoo EM,
      3. Bottomley A
      . Review on quality of life issues in patients with primary brain tumors. Oncologist 2010;15:618–26. doi:10.1634/theoncologist.2009-0291
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Aaronson NK
      . Quality of life research in cancer clinical trials: a need for common rules and language. Oncology (Williston Park, NY) 1990;4:59–66.
      OpenUrlPubMed
    32. ↵
      1. Young T,
      2. de Haes D,
      3. Curran D, et al
      . Guidelines for assessing Quality of Life in EORTC clinical trials. Brussels: EORTC QOL Group, 2002.
    33. ↵
      1. Fallowfield L
      . Compliance issues in quality of life assessment: experiences of two Cancer Research Campaign sponsored groups. Stat Med 1998;17:541–6. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<541::AID-SIM801>3.0.CO;2-#
      OpenUrlCrossRefPubMed
    34. ↵
      1. Larkin ME,
      2. Lorenzi GM,
      3. Bayless M, et al
      . Evolution of the study coordinator role: the 28-year experience in Diabetes Control and Complications Trial/Epidemiology of Diabetes Interventions and Complications (DCCT/EDIC). Clin Trials 2012;9:418–25. doi:10.1177/1740774512449532
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Bernhard J,
      2. Gusset H,
      3. Hürny C
      . Practical issues in quality of life assessment in multicentre trials conducted by the Swiss Group for Clinical Cancer Research. Stat Med 1998;17:633–9. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<633::AID-SIM810>3.0.CO;2-U
      OpenUrlCrossRefPubMed
    36. ↵
      1. Ganz PA,
      2. Day R,
      3. Costantino J
      . Compliance with quality of life data collection in the National Surgical Adjuvant Breast and Bowel Project (NSABP) Breast Cancer Prevention Trial. Stat Med 1998;17:613–22. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<613::AID-SIM808>3.0.CO;2-9
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Wisniewski SR,
      2. Leon AC,
      3. Otto MW, et al
      . Prevention of missing data in clinical research studies. Biol Psychiatry 2006;59:997–1000. doi:10.1016/j.biopsych.2006.01.017
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Bernhard J,
      2. Peterson HF,
      3. Coates AS, et al
      . Quality of life assessment in International Breast Cancer Study Group (IBCSG) trials: practical issues and factors associated with missing data. Stat Med 1998;17:587–601. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<587::AID-SIM806>3.0.CO;2-#
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. Hahn EA,
      2. Webster KA,
      3. Cella D, et al
      . Missing data in quality of life research in Eastern Cooperative Oncology Group (ECOG) clinical trials: problems and solutions. Stat Med 1998;17:547–59. doi:10.1002/(SICI)1097-0258(19980315/15)17:5/7<547::AID-SIM802>3.0.CO;2-F
      OpenUrlCrossRefPubMed
    40. ↵
      1. Cella DF
      . Methods and problems in measuring quality of life. Support Care Cancer 1995;3:11–22. doi:10.1007/BF00343916
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Wilcox S,
      2. Shumaker SA,
      3. Bowen DJ, et al
      . Promoting adherence and retention to clinical trials in special populations: a women's health initiative workshop. Control Clin Trials 2001;22:279–89. doi:10.1016/S0197-2456(00)00130-6
      OpenUrlCrossRefPubMed
    42. ↵
      1. Young T,
      2. Maher J
      . Collecting quality of life data in EORTC clinical trials—what happens in practice? Psychooncology 1999;8:260–3. doi:10.1002/(SICI)1099-1611(199905/06)8:3<260::AID-PON383>3.0.CO;2-Z
      OpenUrlCrossRefPubMed
    43. ↵
      1. Osoba D,
      2. Dancey J,
      3. Zee B, et al
      . Health-related quality-of-life studies of the National Cancer Institute of Canada Clinical Trials Group. J Natl Cancer Inst.1996; Monographs 2: 107–111.
    44. ↵
      1. Hellard ME,
      2. Sinclair MI,
      3. Forbes AB, et al
      . Methods used to maintain a high level of participant involvement in a clinical trial. J Epidemiol Community Health 2001;55:348–51. doi:10.1136/jech.55.5.348
      OpenUrlAbstract/FREE Full Text
    45. ↵
      1. Revicki D,
      2. Fairclough D
      . Preventing missing data. In: Fayers P, Hays R, eds. Assessing quality of life in clinical trials: analysis and interpretation. 2nd edn. Oxford University Press, 2005:209–222.
    46. ↵
      1. Scott RB,
      2. Farmer E,
      3. Smiton A, et al
      . Methodology of neuropsychological research in multicentre randomized clinical trials: a model derived from the International Subarachnoid Aneurysm Trial. Clin Trials 2004;1:31–9. doi:10.1191/1740774504cn008xx
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Hayden KA,
      2. Moinpour CM,
      3. Metch B, et al
      . Pitfalls in quality-of-life assessment: lessons from a Southwest Oncology Group breast cancer clinical trial. Oncol Nurs Forum 1993;20:1415–19.
      OpenUrlPubMed
    48. ↵
      1. Rouette J,
      2. Blazeby J,
      3. King M, et al
      . Integrating health-related quality of life findings from randomized clinical trials into practice: an international study of oncologists’ perspectives. Qual Life Res 2015;24:1317–25. doi:10.1007/s11136-014-0871-9
      OpenUrlCrossRefPubMed
    49. ↵
      1. Calvert M,
      2. Blazeby J,
      3. Altman DG, et al
      . Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA 2013;309:814–22. doi:10.1001/jama.2013.879
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Brundage M,
      2. Bass B,
      3. Davidson J, et al
      . Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers. Qual Life Res 2011;20:653–64. doi:10.1007/s11136-010-9793-3
      OpenUrlCrossRefPubMedWeb of Science
    51. ↵
      1. Mercieca-Bebber RL,
      2. Perreca A,
      3. King M, et al
      . Patient-reported outcomes in head and neck and thyroid cancer randomised controlled trials: a systematic review of completeness of reporting and impact on interpretation. Eur J Cancer 2016;56:144–61. doi:10.1016/j.ejca.2015.12.025
      OpenUrl
    52. ↵
      1. Schandelmaier S,
      2. Conen K,
      3. von Elm E, et al
      . Planning and reporting of quality-of-life outcomes in cancer trials. Ann Oncol 2015;26:1966–73. doi:10.1093/annonc/mdv283
      OpenUrlAbstract/FREE Full Text
    53. ↵
      1. Glasziou P,
      2. Altman DG,
      3. Bossuyt P, et al
      . Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014;383:267–76. doi:10.1016/S0140-6736(13)62228-X
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Palmer M,
      2. Mercieca-Bebber R,
      3. King M, et al
      . A framework for classifying root causes of missing data for patient reported outcomes in prospective studies. Qual Life Res 2015;24:a2016.
      OpenUrl
    55. ↵
      1. Juni P,
      2. Holenstein F,
      3. Sterne J, et al
      . Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol 2002;31:115–23. doi:10.1093/ije/31.1.115
      OpenUrlAbstract/FREE Full Text
    56. ↵
      1. Moher D,
      2. Pham B,
      3. Klassen TP, et al
      . What contributions do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000;53:964–72. doi:10.1016/S0895-4356(00)00188-8
      OpenUrlCrossRefPubMedWeb of Science
    57. ↵
      1. Fairclough D
      . Design and analysis of quality of life studies in clinical trials. New York: Chapman & Hall, 2010.
    58. ↵
      1. King MT
      . A point of minimal important difference (MID): a critique of terminology and methods. Expert Rev Pharmacoecon Outcomes Res 2011;11:171–84. doi:10.1586/erp.11.9
      OpenUrlCrossRefPubMed
    59. ↵
      1. Wyrwich KW,
      2. Nienaber NA,
      3. Tierney WM, et al
      . Linking clinical relevance and statistical significance in evaluating intra-individual changes in health-related quality of life. Med Care 1999;37:469–78. doi:10.1097/00005650-199905000-00006
      OpenUrlCrossRefPubMedWeb of Science
    60. ↵
      1. Calvert M,
      2. Kyte D,
      3. Duffy H, et al
      . Patient reported outcome (PRO) assessment in clinical trials: a systematic review of guidance for trial protocol writers. PLoS ONE 2014;9:e110216. doi:10.1371/journal.pone.0110216
      OpenUrlCrossRefPubMed
    61. ↵
      1. Kyte D,
      2. Duffy H,
      3. Fletcher B, et al
      . Systematic evaluation of the patient reported outcome (PRO) content of clinical trial protocols. PLoS ONE 2014;9:e110229. doi:10.1371/journal.pone.0110229
      OpenUrl
    62. ↵
      1. Brundage M,
      2. Blazeby J,
      3. Revicki D, et al
      . Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards. Qual Life Res 2013;22:1161–75. doi:10.1007/s11136-012-0252-1
      OpenUrlCrossRefPubMedWeb of Science
    63. ↵
      1. Calvert M,
      2. Kyte D,
      3. von Hildebrand M, et al
      . Putting patients at the heart of health-care research. Lancet 2015;385: 1073–4. doi:10.1016/S0140-6736(15)60599-2
      OpenUrl
      1. Tang ST,
      2. McCorkle R
      . Use of family proxies in quality of life research for cancer patients at the end of life: a literature review. Cancer Invest 2002;20:1086–104. doi:10.1081/CNV-120005928
      OpenUrlCrossRefPubMedWeb of Science
      1. Gheorghe A,
      2. Roberts T,
      3. Hemming K, et al
      . Evaluating the generalisability of trial results: introducing a centre- and trial-level generalisability index. Pharmacoeconomics 2015;33:1195–214. doi:10.1007/s40273-015-0298-3
      OpenUrl
      1. Rutherford C,
      2. Costa D,
      3. Mercieca-Bebber R, et al
      . Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis. Qual Life Res 2015;3:3.
      OpenUrl
      1. Cantrell MA,
      2. Lupinacci P
      . Methodological issues in online data collection. J Adv Nurs 2007;60:544–9. doi:10.1111/j.1365-2648.2007.04448.x
      OpenUrlCrossRefPubMedWeb of Science
      1. Edwards P,
      2. Roberts I,
      3. Clarke M, et al
      . Methods to increase response rates to postal questionnaires. Cochrane Database Syst Rev 2007;(2):MR000008. doi:10.1002/14651858.MR000008.pub3
      1. Osoba D,
      2. Bezjak A,
      3. Brundage M, et al
      . Evaluating health-related quality of life in cancer clinical trials: the National Cancer Institute of Canada Clinical Trials Group experience. Value Health 2007;10(Suppl 2):S138–45. doi:10.1111/j.1524-4733.2007.00278.x
      OpenUrlCrossRefPubMedWeb of Science
      1. Osoba D,
      2. Bezjak A,
      3. Brundage M, et al
      . Analysis and interpretation of health-related quality-of-life data from clinical trials: basic approach of the National Cancer Institute of Canada Clinical Trials Group. Eur J Cancer 2005;41:280–7. doi:10.1016/j.ejca.2004.10.017
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Contributors RM-B, MB, MC, MRS and MTK were involved in study concept. RM-B, MB, MC and MTK were involved in study design. RM-B, MJP, MB, MC and MTK were involved in data acquisition and analysis. RM-B was involved in manuscript preparation. RM-B, MB, MC and MTK were involved in manuscript editing. RM-B, MJP, MB, MC, MRS and MTK were involved in manuscript review.

    • Funding RM-B is supported by Sydney Catalyst, courtesy of the Cancer Institute New South Wales. MTK is supported by the Australian Government through Cancer Australia.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The authors have published all relevant data collected as part of this study; however, readers are invited to contact the corresponding author if further information is desired.