Recommendations for including or reviewing patient reported outcome endpoints in grant applications
BMJ 2021; 373 doi: https://doi.org/10.1136/bmj.n1367 (Published 30 June 2021) Cite this as: BMJ 2021;373:n1367
- Claire Snyder, professor of medicine, oncology, and health policy and management1,
- Alexandra Gilbert, honorary consultant in clinical oncology2,
- David Moher, senior scientist3,
- Derek Kyte, lecturer in health research methods4,
- Ellie Daniels, senior scientific director5,
- Madeleine King, Cancer Australia chair in quality of life6,
- Melanie Calvert, professor of outcomes methodology 4 7,
- Ronald C Chen, chair8,
- Michael Brundage, professor of oncology and public health sciences9
- on behalf of the PROTEUS consortium
- 1Johns Hopkins School of Medicine and Bloomberg School of Public Health, Baltimore, MD, USA
- 2Leeds Cancer Centre, Leeds, UK
- 3Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
- 4Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- 5Cancer Prevention and Control Research, American Cancer Society, Atlanta, GA, USA
- 6University of Sydney, School of Psychology, Sydney, NSW, Australia
- 7Birmingham Health Partners Centre for Regulatory Science and Innovation, NIHR Biomedical Research Centre, NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR Applied Research Collaboration West Midlands, University of Birmingham, Birmingham, UK
- 8Department of Radiation Oncology, University of Kansas Medical Center, Kansas City, KS, USA
- 9Queen’s University, Kingston, ON, Canada
- Correspondence to: C Snyder csnyder{at}jhu.edu
- Accepted 26 May 2021
Patients, clinicians, regulators, policy makers, and clinical guideline developers value information regarding the impact of disease and treatment from the perspective of patients. Thus, patient reported outcome (PRO) assessments that collect this information are a critical aspect of research studies.12345678 The PRO results from research studies can only be used if they are measured appropriately and reported clearly. However, a recent review of 160 international clinical trials with PRO endpoints found frequent suboptimal reporting, and over a third of trials failed to report PRO findings at all. 9
Several methodological tools have been developed to improve the design, analysis, reporting, and interpretation of PROs in research studies.101112131415 These tools were developed using rigorous methods, including engaging patients and other stakeholders, to provide guidance on designing the PRO aspects of research studies, collecting and analysing the data, and interpreting and reporting the results. The PROTEUS consortium (patient reported outcomes tools: engaging users and stakeholders) was formed to optimise the use of PROs in research studies and clinical practice, in part by promoting the use of these and other PRO resources.16 It builds on the work of other initiatives, such as those to develop core outcome sets17 and inform selection of PRO measures. 18 A PROTEUS consortium meeting in June 2019 (Baltimore, MD), including both grant applicants and funders, identified publication of recommendations regarding the key PRO elements that should be included in grant applications as an important strategic initiative.
Four general methodological areas require specific consideration for the PRO components of proposed research studies: rationale, study procedures, measure selection, and analytical approach. Detailed guidance regarding these PRO methods are available from the resources described above. Specifically, the 2013 SPIRIT (standard protocol items: recommendations for interventional trials) guidance identified the minimum elements required in clinical trial protocols, generally19; the SPIRIT-PRO extension provides specific recommendations for the PRO aspects of clinical trials, including the rationale and study procedures.10 The International Society for Quality of Life Research (ISOQOL) published minimum standards for PRO measures to use in patient centred and comparative effectiveness research,11 consistent with other guidance documents.18 The SISAQOL consortium (setting international standards in analysing patient reported outcomes and quality of life endpoints data) has made preliminary recommendations on analytical approaches, with further work in progress.12
The recommendations in these documents were developed via rigorous, stakeholder engaged, formal consensus processes. However, given the space constraints in most grant applications, not all the recommended information from these three documents can be included. The PROTEUS consortium undertook an effort to identify the key information to include in grant applications. By following these recommendations, investigators can demonstrate, and reviewers evaluate, preparedness to conduct the PRO aspects of the study rigorously, and ensure that adequate resources have been budgeted. By ensuring preparedness at the grant application phase, investigators should be more likely to conduct the study successfully and report the PRO findings effectively.
This project’s aim was to identify the key information from existing guidelines that should be included in grant applications to ensure rigorous methods for assessing patients’ perspectives using PROs. The authors—a subgroup of PROTEUS consortium members—volunteered to develop these recommendations on behalf of the consortium.
Summary points
With the increasing emphasis on patient centred research and the associated increased use of patient reported outcomes in research studies, guidance is needed on how grant applicants can demonstrate, and how grant reviewers can evaluate, the research team’s knowledge and ability to conduct high quality research with patient reported outcomes
Based on existing international guidelines developed through rigorous, stakeholder engaged, formal consensus processes, this article gives practical recommendations on what information on patient reported outcomes should be included in grant applications, along with example text to demonstrate how to deal with each recommendation
The recommendations consider the space constraints of grant applications, and prioritise information that should always be included, information to add when a patient reported outcome is the primary endpoint or when space allows, and information that is helpful to include when space is not a limitation
Methods
The technical appendix provides a full description of the four step, informal consensus development process used by the authors to develop these recommendations. The authors have a range of career and disciplinary backgrounds. They include clinicians, researchers, grant applicants, and grant reviewers with expertise in PROs, clinical trials, or both. Several authors were also involved in developing the SPIRIT-PRO extension (MB, MC, DK, MK), ISOQOL minimum standards for selecting a PRO measure for use in research studies (CS, MB), and the SISAQOL analysis guidance (MC, MK); ED serves as the senior scientific director for a major funding organisation.
Briefly, the authors first identified 40 items (37 from SPIRIT-PRO, three from SISAQOL) to be considered for recommendation. Secondly, the candidate items were rated by each author using a four point scale:
Always include—assume there is only enough space in the grant application for one paragraph worth of content related to PROs (although this content might be described throughout the grant application)
Always include when the PRO is the primary endpoint or if a second paragraph of PRO content can be included in the grant application (although this content might be described throughout the grant application)
Helpful information if space allows
Not necessary even if space allows.
Thirdly, the candidate recommendations were ranked on the basis of average rating—for discussion purposes only, and not to be standalone determinants of the recommendations. Example text was also suggested to demonstrate how recommendations preliminarily categorised as (1) and (2) in the list above could be dealt with by investigators when writing grant applications. Finally, the author team reviewed the preliminary categorisations based on the rating exercise and discussed possible changes to the item categorisation and refinements to the example text. The final paper was circulated to the PROTEUS consortium members before submission.
Results
The technical appendix presents specific results from the first three steps of the recommendation-development process. The final recommendations for information to always include, or to include if the PRO is a primary endpoint or if a second paragraph of PRO content can be included, are presented in box 1 and box 2, respectively, along with example text. Based on the refinements and the combination of some items, six topics should always be covered: rationale, research question, outcomes of interest, time points for assessment, data collection plan, and analysis methods (box 1). An additional 10 topics should be addressed if the PRO is a primary endpoint or if additional space is available (box 2). These topics provide more detail regarding the PRO study’s background and rationale, data collection and management, and analysis. Box 3 lists 15 PRO topics that might also be helpful to include in grant applications if space allows.
Topics related to patient reported outcomes (PRO) that should always be covered in grant applications, with example text*
1) Describe the rationale for PRO assessment:
“It is anticipated that the majority of patients in this trial will have asymptomatic metastatic disease (detected by a rise in prostate-specific antigen [PSA] or routine imaging). As such, the quality-of-life (QOL) outcomes for each treatment will be secondary outcomes reflective of the negative impact on QOL for either Treatment A or Treatment B, and the trial findings can be placed in context of the disease-related outcome benefits. The results will be used for . . . (examples: inform clinical practice guidelines, health technology assessment, regulatory application, describe treatment impact).”
2) State the PRO specific research question(s):
“The objective of the QOL substudy is to compare health-related QOL for four specific domains in men treated with Treatment A versus Treatment B in the post androgen receptor (AR)-targeted therapy setting.”
3) Specify the PRO concepts or domains used to evaluate the research question(s) (eg, overall health related quality of life, specific domain, specific symptom), and the questionnaire(s) selected to assess them:
“QOL will be assessed with the Functional Assessment of Cancer Therapy-Prostate (FACT-P) [add reference], which consists of the FACT-G (general), a 27-item self-report questionnaire that measures general QOL in cancer patients, and a 12-item prostate cancer-specific subscale. The primary QOL outcome is the FACT-P treatment outcome index (TOI), which combines the physical and functional wellbeing items of the FACT-G with the 12 prostate-specific items. The null hypothesis is that Treatment A will have no negative impact on QOL compared to Treatment B for the treatment outcome index derived score.”
4) Describe the time points for PRO assessment:
“At baseline and every 6 weeks (maximum 6 cycles), to be done prior to the clinical assessment.”
5) Include a data collection plan outlining the permitted mode(s) of administration (eg, paper, telephone, electronic, other) and setting (eg, clinic, home, other):
“Data will be collected at clinic visits using paper records completed in the clinic prior to the clinic visit.”
6) State the PRO analysis method(s), in relation to the objective(s). State the broad PRO objectives, specifying if they are exploratory/descriptive or aim to evaluate treatment efficacy/clinical benefit. If they are to evaluate treatment efficacy/clinical benefit, state specific hypotheses (including relevant PRO concepts or domains) and include whether the between-group comparison tests for superiority, equivalence, or non-inferiority. If the broad PRO objectives include within-patient or within-treatment group comparisons, clearly state the assumption (that is, improvement, worsening, stable state, overall effect), specific objective (eg, proportion of responders, time to PRO event, magnitude of improvement or worsening), and principal time point of interest:
“The overall objective is to compare mean FACT-P Treatment Outcome Index scores between groups. The primary analyses will use GLM [generalised linear model] regression to test the hypothesis that TOI scores are, on average, higher in patients receiving the intervention compared to standard of care over the duration of the study (24 weeks). The FACT-P sub-scales will also be calculated and compared between arms in exploratory analyses. To determine clinical significance of any observed TOI differences between arms, a 7 point change will be considered clinically meaningful [add reference]. Also as an exploratory analysis, the proportion of patients improved at 12 weeks will be calculated (that is, the individual TOI score improved compared to baseline by 7 points [add reference] or more) and compared between arms using unadjusted χ2 comparisons.”
*The topics might be included in a dedicated section or could be described throughout the grant application.
Topics that should be included in grant applications if a patient reported outcome (PRO) is a primary endpoint or if a second paragraph of PRO content can be included, with example text*
Background and rationale for PRO assessment
1) Summarise PRO findings in relevant studies:
“The quality of life associated with various treatments in metastatic castrate resistant prostate cancer (mCRPC) in patients undergoing therapy such as docetaxel, abiraterone and enzalutamide has previously been described [add references]. However, the quality of life in patients receiving Treatment A has not been well studied or described. Treatment B has been shown to impact on patients’ quality of life in a number of domains (physical functioning, emotional functioning, and specific symptoms such as fatigue, gastrointestinal and other systems) [add references].”
Data collection and management
2) Justify the PRO instrument selected and provide or cite evidence of PRO instrument measurement properties and patient acceptability or burden, ideally in the population of interest:
“The FACT-P [Functional Assessment of Cancer Therapy-Prostate] was chosen because Treatment B has been shown to impact a number of QOL [quality of life] domains including physical functioning, emotional functioning, and numerous specific symptoms, and the FACT-P is validated and has been used successfully in multiple studies in this patient population [add reference].”
3) If PROs will not be collected from the entire study sample, provide a rationale and describe the PRO specific eligibility criteria (eg, PRO substudy, language or reading requirements, or pre-randomisation completion of PRO assessment):
“The sample size for the primary trial endpoint is 780 patients. Since this sample size would result in an over-powered PRO analysis, only the first XXX patients enrolled will participate in the PRO substudy.”
4) When the study context requires someone other than a study participant to answer on his or her behalf (a proxy reported outcome), state and justify the use of a proxy respondent:
[In general, proxy assessments are discouraged; however, there might be circumstances where proxy reporting is necessary. If proxies are going to be used and the PRO measure has been validated for proxy assessment:] “The use of proxy reports is discouraged; however, proxies may be used for study participants who are cognitively impaired and unable to complete the assessment themselves. To be eligible to provide proxy data about a participant, the proxy must live with the participant, be willing to provide data at scheduled assessments, and have sufficient proficiency in English.”
5) Specify PRO data collection and management strategies to minimise missing data:
“To minimise missing data, we will train staff regarding procedures for collecting PROs before clinical assessment, checking the completed forms for missing items, and submitting the data. A written manual will be provided for reference. Compliance rates (proportion of submitted forms of those expected) will be monitored in real-time, allowing for mitigation if compliance falls below that expected in specific participating centres.”
6) State whether PRO data will be monitored during the study to inform the clinical care of individual study participants:
[If not monitored:] “To maintain blinding, patients’ clinicians will not have access to the PRO data to inform clinical care.”
[If monitored:] “The research team will review completed PRO questionnaires. If, through this review, the research team becomes concerned for the wellbeing of the participant, they will discuss these concerns with the participant directly and may also consult with the PI [principal investigator] and/or treating clinician if the concerns involve the patient’s safety.”
Analysis
7) When a PRO is the primary endpoint, state the required sample size (and how it was determined) and recruitment target (accounting for expected loss to follow-up):
[If the PRO had been a primary endpoint, text such as the following could have been included:] “A sample size of XXX will provide 90% power to test the hypothesis that mean TOI scores are at least 7 points greater for patients randomised to Treatment B compared to Treatment A with a significance level of 5% based on an assumed SD [standard deviation] of XX in this patient population. Seven points is considered clinically meaningful [add reference]. The PRO sample size will be adjusted upwards to allow for a 15% non-compliance rate, resulting in a final sample size of YYY patients (YYY÷2 in each arm).”
8) Outline the methods for handling missing items or entire assessments (eg, approach to imputation and sensitivity analyses):
“Compliance (received v expected forms with appropriate windows) will be described. GLM [generalised linear model] regression . . . will, in part, account for missing data. If missing data exceeds 10% of expected measures, sensitivity analyses will be conducted.”
9) Specify whether more than one language version will be used:
“All validated languages of the FACT-P will be used.”
10) Include PRO analysis plans for addressing multiplicity or type I (α) error:
“Because we have identified a single PRO domain, the TOI [treatment outcome index], for our primary analysis comparing Treatment A and Treatment B, there is no need to adjust the alpha, and P<0.05 will be used; P values for all other domains will be descriptive.”
*This content is in addition to the content described in box 1. These additional topics might be included in a dedicated section or could be described throughout the grant application.
Recommended information on patient reported outcomes (PRO) to include in grant applications if space allows
Background and rationale for PRO assessment
Describe instrument’s domains, number of items, recall period, and scaling and scoring (eg, range and direction of scores indicating good or poor outcome)
Provide or cite evidence of instrument interpretation guidelines, if available, ideally in the population of interest
Specify the individual(s) responsible for the PRO content of the protocol
State whether the measure will be used in accordance with any user manual and specify and justify deviations if planned
Provide or cite evidence of the validity of proxy assessment, if available and relevant
Data collection and management
Justify if the initial PRO assessment is not before randomisation, where applicable
Specify whether PRO collection is prior to clinical assessments
If using multiple questionnaires, specify whether order of administration will be standardised
Provide a rationale for the PRO assessment time points and specify time windows for PRO assessment
State whether translated versions have been developed using currently recommended methods
Describe the process of PRO assessment for participants who discontinue or deviate from the assigned intervention protocol, where applicable
If PRO data will be monitored to inform clinical care, describe how this will be managed in a standardised way
Describe how PRO data monitoring for clinical care will be explained to participants (eg, in the participant information sheet and consent form)
Analysis
If sample size is not established on the basis of the PRO endpoint, then discuss the power of the principal PRO analyses
State how missing data will be described
Discussion
This article provides recommendations for both grant writers and grant reviewers (including researchers and patient-partners) regarding the PRO information that should be included in grant applications. As the assessment of PROs becomes more common, in part because of the increasing emphasis on patient centred research, grant applicants need to be able to demonstrate (for grant reviewers to evaluate) the knowledge of the study team and their ability to undertake scientifically rigorous PRO research. At the same time, recommendations need to take into account the strict space constraints common in grant applications.
Using guidance documents developed through rigorous, stakeholder engaged methods, we identified the key topics for inclusion in grant applications. This prioritisation exercise was required because space constraints for many grant applications do not allow coverage of, for example, all of the information recommended by SPIRIT-PRO. Therefore, our goal was to identify the subset of existing recommendations relevant for grant applications using a multistep approach. The cut-off points in categorising the items were arbitrary, and qualitative review of the recommendations informed the final decisions. To provide additional practical guidance, we also included example text demonstrating how each of the recommended topics could be included in a grant application.
This paper, however, is not a guide for how to design a PRO study. Although box 1 and box 2 include example text regarding key PRO study components, the approaches described are not exhaustive. For example, the example text for dealing with multiplicity (box 2, point 10) describes selecting one PRO domain for hypothesis testing. Other approaches for dealing with this issue could be to adjust the P values to take account of multiple testing and retain a nominal overall P value, or to de-emphasise reporting P values in favour of reporting confidence intervals. In another example, box 1 (point 4) presents example text outlining the time points for PRO assessment, which should always be included in grant applications. However, the process for determining which are the most informative time points for a specific study (eg, consulting with clinical experts, patient partners, and trials units) is not described here. Additional details regarding the time points for PRO assessment should be included in the study protocol, in accordance with SPIRIT-PRO, with a table, for example. The same applies for many of the other recommendations, which would require more detail in the study protocol or statistical analysis plan, both of which typically do not have space constraints. Sufficient PRO expertise within the study team is required to implement these recommendations in practice.
The recommendations reported here are primarily directed at grant applicants and grant reviewers, and we would refer individuals to the foundational ISOQOL minimum standards,11 SPIRIT-PRO,10 and SISAQOL12 guidance for other contexts. Other guidance documents are available on how to report the results of PRO research studies1314 and on how clinicians should evaluate the PRO literature.15 By following the above recommendations at the formative, grant application phase, research teams can successfully report their results meaningfully at the completion of the study. Furthermore, many of the recommendations included here are not specific to PROs and would apply to many other research study endpoints. By beginning with the end in mind, these recommendations aim to help both grant applicants and grant reviewers ensure the rigor and relevance of PRO research studies.
Acknowledgments
The views expressed in this article are those of the authors and not necessarily those of the NIHR, Department of Health and Social Care, Regulatory Horizons Council or Department for Business, Energy and Industrial Strategy.
Footnotes
Contributors: CS led the recommendation development process described here and serves as the guarantor of the article. She is the principal investigator of the PROTEUS Consortium. MB is the PROTEUS consortium co-principal investigator and partnered with CS in leading this project. The PROTEUS consortium recommended development of this grant writing guidance. All authors participated in the recommendation development process as described above and reviewed and approved the final version of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: Funded by the Patient-Centered Outcomes Research Institute and Genentech.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Patient-Centered Outcomes Research Institute and Genentech for the submitted work; CS and MB receive support for their leadership of the PROTEUS consortium; MC and MK receive honorariums for participating on the PROTEUS steering committee; MK has a subcontract from Johns Hopkins for PROTEUS related work; no other authors have funding directly related to this work; CS and MB report a grant from Pfizer for a separate PROTEUS project, for which MC also receives an honorarium; CS and MB have had previous funding from Genentech outside of the submitted work; RCC reports personal fees from Abbvie, Accuray, Myovant, Bayer, and Blue Earth Diagnostics outside the submitted work; DK reports grants from the UK National Institute for Health Research (NIHR), Innovate UK, NIHR Birmingham Biomedical Research Centre, NIHR SRMRC at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, and Kidney Research UK, and personal fees from Merck and GlaxoSmithKline (GSK) outside the submitted work; ED is employed by the American Cancer Society; AG and DM have nothing to disclose; MC reports personal fees from Astellas, from Takeda, Glaukos, Merck, Daiichi Sankyo, and GSK, and grants from Health Data Research UK, Wellcome Trust, Alan Turing Institute, and Research England, UK Research and Innovation outside the submitted work; MC is an NIHR senior investigator and receives funding from the NIHR Birmingham Biomedical Research Centre, NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR ARC West Midlands at the University of Birmingham and University Hospitals Birmingham National Health Service Foundation Trust, Health Data Research UK, Innovate UK (part of UK Research and Innovation), Macmillan Cancer Support, and UCB Pharma.
Provenance and peer review: Not commissioned; externally peer reviewed.