1 Introduction

Current methods for safety reporting and risk assessment still rely heavily on healthcare professionals (HCPs). A way to improve the quantity and/or quality of safety information is to encourage patient-reported outcomes of adverse events (PRO-AEs), which are more patient focused and may have less formal data collection processes that do not rely on input from HCPs. Some HCPs such as pharmacists, however, might facilitate PRO-AE collection [1, 2]. HCPs will also retain a critical role in assessing the causality between adverse events (AEs) and drugs, especially for individual cases of severe and fatal AEs.

Because of the varied nature of patient populations (see Fig. 1), a range of different data collection tools, analytical approaches and methodologies may need to be deployed to meet different PRO-AE requirements. A classification based on whether or not the relevant patient population is prespecified (rather than just pre- or post-approval) provides a rational basis for further subdividing the safety populations [3].

Fig. 1
figure 1

Classification of PRO-AEs. PRO-AE patient-reported outcome of adverse event. a Prespecified populations will have different analytics to non-prespecified populations, with the latter being lower in the quantitative hierarchy. Analytics used lower in this hierarchy can also be utilised for populations higher up

Full size image

For instance, the dataset is more structured and the patient population is better defined in post-approval, prespecified populations (e.g. phase 4 clinical trials and prospective observational studies) than in post-approval, non-prespecified patient populations. However, even within the non-prespecified group there are variations—for example, some patient support websites use structured data templates to collect PRO-AEs, whereas most websites have only unstructured free text.

There are also distinctions between clinical trials, where the denominator is clear and there are specific goals around balancing safety and efficacy [4, 5], and safety surveillance systems that may have no clear denominator representing the total number of patients. There are also issues around either defining a cohort or opening up reporting to any patient, either voluntary or solicited.

Patient-reported outcome (PRO) data can be captured in a structured way using a suitable instrument, which is a specific reporting tool that through completion generates a structured dataset. This could be a questionnaire or diary along with supporting information such as training materials, scoring, etc.

The amount and nature of missing data, under-reporting and bias are substantially affected by the specific data collection approach taken. In the regulatory context, the approach to collecting PROs for AEs may differ depending on whether the aim is a comparative tolerability claim (i.e. a product having a more favourable tolerability profile compared with another) or just general documentation of AEs.

Tools used to capture PROs have become established in areas such as assessing the quality of care delivered to patients by providers and health-related quality of life (HRQoL) measures used in clinical trials. In these fields, PROs are based on methodologies that have been well validated. In the UK National Health Service (NHS), patient-reported outcome measures (PROMs) have been used to measure patient outcomes for some planned surgical interventions, such as hip and knee replacements, varicose veins and hernia [6], but the ability to collect PRO-AEs within this framework is currently relatively limited.

The use of PRO instruments for capturing AE data is not so well developed. Most progress has been made with the collection of pre-approval PRO-AEs in clinical trials, although there is also significant previous literature on voluntary reporting [3]. Some PRO questionnaires given to patients in trials cover potential adverse effects, and for certain conditions such as HIV, there has been systematic collection of symptoms from patients using checklists. The National Cancer Institute (NCI) in the USA is also developing a methodology for collecting PRO-AEs in oncology trials based on a patient-reported outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). In the epidemiological context, AE outcome measures can be continuous variables or have binary outcomes.

This guidance will consider if methodologies used to collect pre-approval PRO-AEs could be modified for post-launch studies with prespecified populations. The successful use of PRO tools has suggested to some that monitoring of AEs reported by patients (as well as their care-givers and relatives, where appropriate) outside of the framework of clinical trials should be routinely considered in everyday practice, because it more closely represents the patients’ self-evaluation of HRQoL [7]. However, some signs or symptoms may be due to the disease rather than the drug, and it is important to distinguish these.

The use of PRO-AE instruments outside of clinical trials is also less well implemented. Whilst some countries have previously developed web-based systems to enable consumers to report post-approval suspected adverse reactions (ARs) directly to the regulatory authority [e.g. Food and Drug Administration (FDA) MedWatch, Medicines and Healthcare Products Regulatory Agency (MHRA) Yellow Card scheme, Australia TGA Blue Card scheme, Health Canada Med Effect] in an attempt to address under-reporting by HCPs, this approach has had only limited success in capturing safety data [8, 9]. Recent EU legislation means that there will be increased consumer reporting of this type across member states.

Similarly, there are currently only a few non-regulatory approaches aiming to capture PRO-AE data, including PRO-AE-enabled patient support websites [10]. Whilst popular with patients, these sites are primarily intended for the purposes of patient support and education rather than as PRO-AE instruments. The potential to mine AE data from unstructured text derived from online health forums is feasible, but needs further development before wider utilisation [11].

Patient support programmes (PSPs), which are developed by companies to assist patients and/or HCPs in better managing diseases and treatments, are already subject to AE reporting requirements. PSPs, however, are typically not specifically designed to collect PRO-AE data post launch and may represent missed opportunities to collect improved post-marketing safety data.

There is evidence [7] to suggest that collecting symptom data directly from patients can be beneficial, leading to an improvement in the accuracy and efficiency of symptomatic AE data gathering [12–14]. It is also clear that this is a growth area, particularly in the unregulated social network environment, and monitoring of this data is a concern for health authorities.

However, for industry, patients, academics and regulators to make best use of this information, guidelines are needed. The definitions, standards, methodologies and utility of PRO-AEs set out in this guidance will support and promote the use of PRO-AE instruments across the drug life cycle. This PRO-AE guidance may also be applied to other related situations, for example, the monitoring of medical devices used by patients or potentially the data generated by patient-worn devices (e.g. for the reporting of falls).

2 Definition of a PRO-AE

It is important to clearly define what is meant by PRO-AEs and to determine whether these are different from other forms of consumer reports. There are established definitions used by the European Medicines Agency (EMA) and the US FDA for PROs in general and for AEs [15–17]. A recommended definition for PRO-AEs based on these resources, and used in this document, is as follows:

A PRO-AE is any untoward medical occurrence, whether or not considered treatment/intervention related, that is reported or transmitted directly by the patient without interpretation by a clinician or anyone else. PRO-AEs may be collected by both structured and unstructured reports.

This definition includes information from patients directly reporting their own experiences (e.g. symptoms) and from patients transmitting data that they have received (e.g. laboratory test results). Patient-reported symptoms may be related to the drug or due to other factors, such as the underlying disease, that represent AEs rather than adverse drug reactions (ADRs). Although it can be argued that patients may not always be able to distinguish between symptoms, signs, safety events and ADRs, this should not matter unduly, provided that AEs and ADRs are distinguished and appropriately interpreted.

There is some debate about whether PROs should be covered by a narrower definition related only to the use of validated instruments, which includes just a subset of all patient self-reported data. For the purposes of this guidance, PROs encompass the full range of self-reporting, as the guidance proposes that an acceptable overarching framework can be created, and that approaches that are currently less validated may actually collect more clinically applicable information as newer analytic methods evolve.

Hence, PRO-AEs (whether structured or unstructured, and prespecified or non-prespecified) include signs and symptoms that are less severe and reflect tolerability, more severe ARs that reflect safety issues, as well as signs and symptoms that may be a precursor of more severe ARs (e.g. febrile angina preceding the onset of aplasia and systemic infection). In addition, abnormal laboratory values that could be a precursor for a potentially severe AR (e.g. neutropenia or liver injury) may be considered as patient-transmitted outcomes. Abnormal biological values may provide useful supporting information, although there could be heterogeneous reporting by patients.

There may be a limitation in adequately reporting some symptoms for which patients may not be aware, e.g. cognitive dysfunction or loss of memory, although in some cases a patient’s relative, carer or HCP may identify these to the patient, who subsequently reports them.

3 Scope

This guidance covers PRO-AEs from both the pre- and post-approval environments, as shown earlier in Fig. 1. It proposes definitions, methodologies, data collection tools, datasets and analytic approaches for PRO-AE reporting that are rigorous, transferable and consistent.

However, because the methods, tools and analytics for pre-approval PRO-AEs are already relatively well documented [18, 19], the main focus of this guidance will be on post-approval PRO-AEs from both prespecified and non-prespecified populations. As the adoption of PRO-AEs in clinical trials is well advanced, the transferability of knowledge gained in this area will also be examined.

The main purpose for collecting post-approval AE data is to enable companies and regulators to assess the ongoing safety profile of a product, under real-use conditions, using signal detection techniques. However, under-reporting, incomplete data, and reporting bias currently limit the utility of signal detection, and alternative methods of data collection such as PRO-AEs need to be explored.

The goal of analysing PRO-AE data is to add to the knowledge already provided by HCPs, when available, increasing the likelihood and speed of detection of signals. PRO-AEs can also be used complementary to traditional data on efficacy and safety for benefit–risk analysis, risk minimisation evaluation, and communication of safety issues to patients. The collection of symptom-based PROs is also important in many large post-approval studies to better define (in terms of frequency, description and evolution) symptoms that were previously reported in randomised clinical trials.

Analysis of non-prespecified populations gives access to much larger populations than can be achieved even from observational studies, as well as reports and perspectives on symptoms that are important to patients, but which are often not reported to HCPs or remain unreported by patients because of the sensitive nature of those symptoms. Thus, the collection and analysis of PRO-AEs may increase the ability of finding new safety concerns (especially rare safety issues that require large numbers of exposures to detect) and also broadens the types of data collected (events of concern specifically to the patient).

Although open questions may be used to elicit safety information in the latter population, there may be a significant risk of selection and reporting bias. Qualitative analysis can be difficult and might potentially lead to false signals if the population is heterogeneous and all relevant factors are not identified and analysed. Therefore, careful consideration must be given to both the questionnaire design and any subsequent analytics to reflect this (Sects. 5, 6). The same considerations apply to data collected from online patient communities.

4 Rationale for Wider Use of PRO-AEs

4.1 Problems with Current AE Data

Current safety reporting and risk assessment processes for both prespecified and non-prespecified populations are heavily dependent on input from HCPs and introduce a number of limitations, as shown in Table 1, although they still represent the current gold standard.

Table 1 Problems with current AE data in prespecified and non-prespecified populations
Full size table

Despite these shortcomings, there may still be a reluctance to adopt an approach that also includes PRO-AEs because of:

  • Regulatory constraints;

  • Concerns about feasibility;

  • Scepticism regarding the validity and reliability of using patient-reported AE data;

  • Higher data volumes needing review that might obscure or reduce focus on key safety concerns;

  • Added administrative requirements and cost;

  • Limitations of the available questionnaires/methods;

  • Industry concerns about the potentially high volume of relatively ‘minor’ AEs reported by patients, and the cost and practicality associated with processing them;

  • Privacy issues, including control of patient information collected and stored electronically.

However, these objections are not insurmountable, as shown in Table 2, and have been successfully dealt with in some clinical PRO safety systems, such as the NCI approach.

Table 2 Potential solutions to objections against use of PRO-AE instruments
Full size table

4.2 When Could PRO-AE Data Be Collected?

PRO-AE data could potentially be collected at all stages of drug development as well as post launch (see Fig. 2). Currently, the majority of PRO-AEs are collected in phases 2 and 3, often supporting more traditionally collected safety information.

Fig. 2
figure 2

PRO-AEs are applicable at all stages of the medicinal product life cycle. PRO-AE patient-reported outcome of adverse event

Full size image

A useful framework for subclassifying PROs would be to consider their impact on measuring benefit–risk balance at each stage along the drug development life cycle. For instance, it has been reported that PRO-AE data from pre-approval clinical trials have:

  • Better described the patient’s underlying health and functional status;

  • Led to the detection of potentially serious AE symptoms earlier than clinician reporting [19].

Similar advantages to collecting PRO-AEs from post-approval populations might be anticipated. A distinction can be drawn between PRO-AEs in specified populations (either pre- or post-launch, with both utilising similar methodologies) and those in patient populations not prespecified, which demand different methodologies.

Some of the PRO-AE measures that are well documented in the pre-approval space could also be adapted for non-prespecified, post-approval studies. However, issues with the reporting time frame, completeness of data, lack of medical confirmation, and the ability to accurately define the numerator and denominator at risk (i.e. number of subjects experiencing events as a proportion of total number of subjects) mean that different or additional approaches may be more appropriate for the post-approval, non-prespecified group.

Post-approval PRO measurements could therefore be used in:

  • Screening to detect signals in the general population and in observational cohorts;

  • Controlled trials for more targeted assessments;

  • Other PRO-AE-based approaches in post-authorisation studies, for instance, the proactive follow-up of patients after occurrence of an AE of special interest (which may include a medical confirmation of the detected events through the event-treating physician).

4.3 Benefits and Challenges of PRO-AE Data

Integrating the patient perspective into drug safety reporting not only improves the accuracy of the data collected but may also enhance the patient-centeredness of clinical research. In fields such as oncology, where symptoms are common and can substantially impair patients’ functioning and quality of life, information about patients’ experiences of AEs is essential for multiple stakeholders, including patients, drug developers, regulators, and payers. The benefits of PRO-AEs are summarised in Table 3.

Table 3 Benefits of PRO-AEs to different stakeholders
Full size table

Further advantages of PROs more generally, from a benefit–risk perspective, are shown in Table 4.

Table 4 Further potential benefits of PROs in general
Full size table

There are also potential public health downsides to patient adverse outcomes being disseminated without being put into an appropriate benefit–risk context, e.g. causing unnecessary alarm or leading to discontinuation of effective therapies. Safe and effective public communication of benefit–risk should form part of any organised collection of PRO-AEs, in both prespecified and non-prespecified populations.

However, how such communication takes place for unplanned, unstructured, spontaneously posted, web-based PRO-AEs is more challenging. It can be reasonably argued that meaningful analysis of such information offers the opportunity for objective communication of the actual risks, perhaps through regulators or other trusted third parties, thus representing a significant improvement on the current situation.

Other perceived disadvantages of PRO-AEs include costs, the possibility of swamping safety signal with noise, exacerbation of the placebo effect at pre-approval phases, and the possibility that the PRO-AE data is not independent of the AEs collected by physicians in pre- and post-approval phases, although most of these issues can be resolved with appropriate methodology. In fact, the two data sources might be highly correlated and therefore of low added value, although still of confirmatory benefit. However, it is rarely possible to match up patient information on reports in the safety database, usually because limited identifying information is provided for patients from non-standardised data collection schemes. There is a significant risk of having duplicates in the database for cases received from multiple sources.

This guidance does not attempt to discuss the merits of the newer forms of web-based communication of such information compared with the traditional processes, as this issue has a much wider scope than PRO-AEs and transcends the whole topic of better public communication of benefit–risk balance.

4.4 Evidence Supporting the Use of PRO-AE Data

There is now considerable evidence from clinical trials that patient and clinician reports of AEs, particularly those based on subjective symptoms during cancer treatment, can provide diverse yet complementary data. For example, it has been shown that including information gathered directly from study participants via PRO measures improved the predictive accuracy of clinician-reported CTCAEs [19, 30]. Although clinicians were able to predict survival well, patient reporting of symptomatic toxicities better reflected the patient’s underlying health state and functional status.

Patient reporting also appears to detect potentially serious AEs or their symptoms earlier than clinician reporting. This is well demonstrated by the timing and cumulative incidence of patient versus clinician reporting of severe diarrhoea in the irinotecan/fluorouracil/leucovorin arm of the NCI intergroup trial N9741 [31], in which many life-threatening gastrointestinal serious AEs were detected [32], as well as subjectively reported AEs in musculoskeletal and central nervous systems.

Evidence suggests that patients are best placed to report their subjective experiences, whereas HCPs contextualise that experience in terms of the disease [19]. Both views are valid and contribute towards improving the understanding of the treatment of AEs if recognised as such by regulators. Industry is thus incentivised to develop mechanisms that allow both datasets to be fully utilised [19].

For drugs already on the market, there is some evidence of increasing consumer activity in the reporting of AEs to regulators. However, the fastest rate of growth in potential consumer-reported AEs appears to be through non-regulatory, PRO-enabled patient websites [3].

5 Developing PRO Instruments for AEs

The FDA has produced guidance [16] and the EMA [17] has published a reflection paper on PRO measures in support of label claims. The FDA guidance states that PRO instruments can be used to measure important safety concerns if those concerns represent symptoms or signs that are best captured from the patient perspective.

A challenge remains of how to integrate PRO-AEs into a benefit–risk assessment, e.g. post-approval. Usually, PRO-AEs will be collected without PRO-FEs (favourable events). If there is lack of perceived efficacy, should patients report this as an AE? It is possible that a few years after marketing, the AE profile of the drug may be defined more accurately but without similar insight into the efficacy/effectiveness profile.

Hence, although the FDA guidance infers that PRO instruments should aim to measure the adverse consequences of treatment separately from the effectiveness of treatment, more recent methodologies would support efficacy (or effectiveness) data being collected from the same dataset as safety, and may in fact improve the utility of the information [33]. Although the context for the FDA and EMA documents is to develop PRO instruments to support labelling, there are some methods in these guidances that may apply to non-labelling objectives.

PRO-AE instruments, like PRO instruments in general, need to be validated and ‘fit for purpose’, i.e. practical for use by patients, to encourage reporting, and interpretable by a number of stakeholders (prescribers, regulators, industry, policymakers and, not least, patients themselves). The following subsections address:

  1. 1.

    A process for developing PRO-AE instruments;

  2. 2.

    Creating a suitable taxonomy;

  3. 3.

    Selecting appropriate technologies;

  4. 4.

    The availability of alternative data sources;

  5. 5.

    The relationship between PRO-AEs and AE data from other sources.

5.1 Process for Developing PRO-AE Instruments

A suitable framework for developing PRO-AE instruments should be used. An example is given here, which was adapted by the Patient-Reported Outcomes Safety Event Reporting (PROSPER) Consortium in a series of iterative discussions and reviews from an approach presented by the FDA in their guidance document [16]. The details associated with each stage are presented in Table 5, which may be iterative. Optional method-specific steps that may be needed are shown in Table 6.

Table 5 Example approach for developing a PRO-AE instrument
Full size table
Table 6 Additional steps that may be required in the example approach
Full size table

A PRO initiative by the National Quality Forum in the USA has suggested that characteristics for a good PRO instrument for AEs and other usages include actionability, meaningfulness to both patient and HCP, facilitation of shared decision making between patient and HCP, and implementability [33].

Some instruments designed to capture adverse effects systematically are available, and have been shown to improve care [34, 35]. Nurses, as the professionals closest to patients, may be best placed to lead the profiling of patient problems. Ensuring that single pertinent questions capture clinically important symptoms efficiently requires cognitive input from both HCPs and patients [36].

5.2 Creating a Suitable Taxonomy

One issue with the development of PRO tools, and especially those used for collecting AE data, is the creation of a taxonomy by an iterative process. In one study, terms were aligned with the existing Medical Dictionary for Regulatory Activities (MedDRA) and refined with input from patients, including comprehension testing within which patient-reported symptoms were categorised and defined [7]. Once achieved, patient-reported symptoms and AEs were generally in agreement with those reported by clinicians and nurses. The PRO tool used in this study was a modified version of the CTCAE using 13 terms familiar to cancer patients. Cirillo et al. [7] have shown that a modified version of the dictionary used in trials can be utilised in the post-marketing setting by patients undergoing chemotherapy.

The development of a patient-friendly dictionary is an important consideration when developing PRO-AE tools to encourage reporting of PRO-AEs. An alternative approach being investigated by the World Health Organization (WHO)/Uppsala Monitoring Centre (UMC) consists of an anatomical interface to help patients identify appropriate AR terms. The electronic interface uses the image of a body, and clicking on a particular area drills down to present AR terms associated with that body area (WHO/UMC, unpublished communication).

5.3 Selecting Appropriate Technologies

In clinical trials, most PRO-AE reporting systems have been paper based. However, the NCI has developed a web-based platform for CTCAEs, as well as standard paper channels. The equivalency between web-based and other traditional channels is currently being tested. This initiative was undertaken to develop a system that would be used by patients to report adverse symptoms in cancer trials; generate useful data for investigators, regulators, clinicians, academic groups and patients; and be compatible with existing AE reporting systems [28].

For marketed drugs, there have been web-based systems allowing patients to report suspected ARs directly to the regulators in the UK, Denmark, Netherlands, US, Sweden, etc. Recent EU pharmacovigilance legislation has included a requirement for direct patient-reporting mechanisms to regulatory authorities.

The WHO/UMC is also developing a web-based system for patient self-reporting of AEs (WHO/UMC, unpublished communication). In contrast, some non-regulatory websites are not exclusively used to collect PRO-AEs, but are designed to be consumer friendly.

When developing PRO-AE instruments, there is a need to consider the most appropriate technology. However, deploying PRO-AE instruments through the internet or through smart technologies such as phones and tablets does require consideration to be given to:

  1. 1.

    Access to technology;

  2. 2.

    Data privacy and storage issues;

  3. 3.

    Appropriateness for the population;

  4. 4.

    Data transmission;

  5. 5.

    Cost;

  6. 6.

    Technical awareness of patients—since the dataset will be biased if only technically aware patients report;

  7. 7.

    Patient knowledge of the ability to report and patient willingness to report;

  8. 8.

    Data privacy and protection.

5.4 Availability of Alternative Data Sources

Data from existing patient sources including electronic patient records are routinely collected in primary care and may be suitable for AE screening, such as patient safety indicators, and to validate novel PRO-AE instruments.

The current FDA Sentinel Initiative is aiming to develop a national electronic safety monitoring system by augmenting existing safety monitoring systems. This has been achieved by leveraging multiple sources of electronic data through partnering with data holders and use of a common data model for healthcare systems, insurance companies and industry, with potential coverage of up to a third of the US population. Although currently not in Sentinel, large datasets such as this would be enhanced by systematically collected PRO-AE data.

The advantages of this approach include enhancing the active post-marketing monitoring of medical product safety to effectively and rapidly examine common outcomes (e.g. myocardial infarction, fractures) in large, real-world populations, with improved access to subgroups and special populations. Additionally, validated quantitative approaches have been developed for signal refinement using sequential monitoring of single time points with adjustments to minimise confounding, near real-time monitoring and an accessible library of tools/resources.

5.5 Relationship Between PRO-AEs and AE Data from Other Sources

There are various models that can be employed and the approach should be prespecified [19]:

  1. 1.

    Independent reporting, in which AE data gathered from the patient and the clinician are collected, analysed, and reported completely separately from each other. A subsequent comparison of the commonalities and differences between the patient-reported and clinician-reported data may provide valuable insights.

  2. 2.

    Merged reporting, in which AE data gathered from the patient and the clinician are collected separately but then merged and analysed together. However, this is a challenging goal, so in many situations, it is preferable to keep the two data pools of clinician- and patient-collected AEs separate to aid meaningful interpretation.

  3. 3.

    Collaborative reporting, in which AE data gathered from the patient is then provided to clinicians to inform their reporting. Specific trials might be needed to test the utility and validity of this concept.

The choice of approach depends on whether two specific pools of information, from patients or from medically confirmed sources, can be combined or whether, because they provide discrete data with different utility, they must be separately stored and analysed. Another factor is the impact of feedback of PRO-AE information on stakeholder behaviour (including prescribers and patients).

A formal comparison of these three approaches is planned as a part of the NCI’s PRO-CTCAE initiative [28]. In total, there are 124 questions that assess different attributes (e.g. presence, frequency, severity, interference) of 78 symptoms that are represented in both the CTCAE (version 4) and MedDRA AE lexicons. Using cognitive testing, these 124 questions have been extensively evaluated by cancer patients, and have been found to be comprehensible and to measure the symptom of interest. An electronic system provides a mechanism for scheduled periodic surveillance and collection of patient-reported symptom data, using the web or a touch-tone telephone. US validation study complete and reported at international meetings, with robust psychometric properties shown for most items. Linguistic adaptation to multiple languages is ongoing, and there are multiple ongoing trials assessing the feasibility of this approach [28].

6 Data Capture and Appropriate Datasets

As already shown in Fig. 1, it is envisaged that PRO-AE instruments will be used to collect different datasets at different stages of drug development. These data in turn will be used to measure different concepts and domains.

Because of this, an important part of PROSPER’s remit is to investigate what PRO-AE datasets might be appropriate when addressing different concepts and domains. Table 7 identifies some of the common AE data collection tools currently in use. European regulatory authorities are also developing standard, structured forms specifically for web-based reporting of suspected ARs by patients as well as HCPs.

Table 7 Currently available datasets for adverse events data capture
Full size table

However, tools such as CIOMS I, MedWatch 3500 and ICH E2B may not be suitable in their full form for use as PRO-AE instruments. These may require editing, with a focus on the PRO concepts/domains being measured, down to a core dataset suitable for PRO-AEs in non-prespecified patient populations. There may also be a need to add some further terms to optimise PROs for patient safety and benefit–risk analysis, e.g. severity as well as seriousness.

An attempt has been made to identify the minimum number of fields in the CIOMS I and MedWatch forms [see Appendix: electronic supplementary material (ESM) 1 and 2] that could provide valuable PRO-AE data. We have also identified those fields in the ICH E2B dataset that would provide the ‘gold standard’ in terms of PRO-AE data (see Appendix: ESM 3). The general aim should be to have enough fields completed by patients to allow easy linkage to individual case study reports (ICSRs) from health professionals, so that these reports may be used together as well as separately.

However, another approach would be to identify the PRO concept or end point that is being assessed and identify the minimum fields needed to determine this value. This may necessitate identifying new fields, e.g. the question “Has a report been submitted by your doctor or HCP?” could result in yes/no/I don’t know options being offered, and the “yes” option would mean that additional information may be required, such as HCP name.

Once a mechanism has been put in place to capture PRO-AEs, a key consideration is how to raise patient awareness to actively provide unsolicited PRO-AEs.

Guidance should also include a framework to define how data are managed, including any data privacy issues:

  • When data are collected, what are the rules of engagement?

  • What will be collected?

  • What will be received (by pharmaceutical companies/regulators)?

  • How will it be validated? The WHO/UMC has investigated this issue, and within its organisation, uses an instrument to score the validity of a report [38, 39].

  • How will duplicates be detected? For example, the ‘hit-miss model’.

Who performs these tasks is of less importance as long as the framework is clear about what needs to be done. However, if a pharmaceutical company is provided with the raw database, then this would trigger reporting responsibilities under current regulations. For the use of similar healthcare databases, personal details are of course already removed (i.e. anonymised databases) for reasons of confidentiality.

7 Analysis of PRO-AE Data

As already discussed, a useful way of classifying PRO-AEs is to relate them to the main population types under investigation, i.e. prespecified and non-prespecified. Further stratification can be provided by considering whether the AE data collected are structured or unstructured, and whether or not the patient selection method is more or less restrictive (see Fig. 1). Analyses of PRO-AEs from prespecified and non-prespecified patient populations need different methodologies, including definition of a core safety dataset to permit useful signal detection. Where possible, measuring baseline symptoms gives a useful additional perspective.

Patient-reported safety information that better reflects the ‘real world’, i.e. with a non-prespecified population and non-structured datasets, has increased medical relevance. However, this must be balanced against quantitative validity (see Fig. 1), at least against the current agreed standards of data from randomised, blinded and controlled trials.

This section considers how analytical robustness of patient-reported safety outcomes can be preserved within these more real-world settings, as well as more controlled settings such as pre-approval clinical trials.

7.1 Scope of the Evaluation

7.1.1 Prespecified Populations

In early clinical trials, the safety evaluation is exploratory in nature and is primarily used to delineate the toxicity boundaries of the dose-response curve (Table 8). However, direct PRO-AE measurements could be used at this stage to screen for unexpected reactions and, if their incidence and severity were confirmed in phase 3 trials, used to support filing/registration claims.

Table 8 Analysis of safety data from different stages and different populations
Full size table

Although there are inherent limitations with pre-approval safety evaluations, such as small sample size, narrow eligibility criteria and limited duration of follow-up, their ability to detect adverse symptom events among study participants improves with the use of clear dataset definitions, methodology for collection, and agreed analytic approaches.

PRO-AE measurements could also be used for post-approval signal detection or for targeted assessments in phase 4 controlled trials.

As discussed in Sect. 5.5, safety data collected via PRO-AE instruments can potentially be analysed independently from or in combination with AE data from other sources. These data can also be used to inform the benefit–risk balance of a product by including in the analysis efficacy or effectiveness data, which may also include PROs.

7.1.2 Non-prespecified Populations

Post-approval safety data are useful for general screening to detect low-frequency safety signals through the exploitation of a larger user population, even from non-prespecified populations. In addition, other PRO-AE-based PASS study designs may benefit from proactive patient follow-up for occurrence of AEs of special interest.

Consequently, the main question is, can PRO-AE data be amalgamated with data from other sources? Low-frequency safety signals may potentially have a significant impact on the benefit–risk profile of a product.

7.2 Choice of Variables and Data Collection

In both the pre- and post-approval phases, the occurrence of serious AEs and treatment discontinuations due to AEs are particularly important to identify. If PRO-AE tools are utilised, a common AE dictionary that allows patient-reported data to be summarised and collated is important (Table 9). This might necessitate the development of a set of terms that are understood by the patient and can be mapped to higher level ‘medical’ terms, similar to the PRO-CTCAE dictionary developed by the NCI for oncology trials [28]. In addition, a severity scoring system for pre-agreed PRO-AEs can be developed. For example, in the case of a symptom like rash, questions will focus on its location on the body.

Table 9 Data collection considerations
Full size table

7.2.1 Prespecified Populations

In pre-approval clinical trials, data from objective laboratory tests, clinical variables (blood pressure, pulse, respiratory rate) and investigations (e.g. radiology) usually form a major part of the safety data. These types of data are reported directly and their transmission by patients would not normally be expected as part of a study. However, clinical AEs (diseases, signs, and symptoms) would be suitable for PRO-AE methods, although of these, a strong emphasis would be placed on patient-reported symptoms.

7.2.2 Non-prespecified Populations

A rapidly increasing quantity of patient-reported data is available on the internet. Unstructured data can present a particular challenge for extracting useful information, since the data are usually incomplete. PRO-AE-enabled websites allow a more structured approach, where the dataset can be designed pre-emptively, and so are far likelier to provide a higher quality of data.

7.3 Defining the Evaluable Population

7.3.1 Prespecified Populations

For pre- and post-approval clinical trials, the evaluable population for PRO-AE comparisons is usually defined as those subjects who received at least one dose of the investigational drug, as well as any comparator drug/placebo groups (Table 10). Safety variables should be collected as comprehensively as possible from these participants, including type of AE, severity, onset, and duration. Additional safety evaluations may be needed in specific subpopulations, such as females, the elderly, the severely ill, or those who have a common concomitant treatment.

Table 10 Population groups
Full size table

All relevant safety variables will be evaluated, and the broad approach should be indicated in the protocol. All AEs should be reported, whether or not they are considered related to treatment. All available data in the study population should be accounted for in the evaluation. Definitions of measurement units and reference ranges of laboratory variables and other investigations should be made with care; if different units or different reference ranges appear in the same trial (e.g. if more than one laboratory is involved), then measurements should be appropriately standardised to allow a unified evaluation. Use of a toxicity grading scale should be prespecified and justified. Definition of PRO symptom severity scales, e.g. for pain, nausea, etc., can also be standardised between studies.

The incidence of an AE is usually expressed in the form of a proportion relating number of subjects experiencing events to the number of subjects at risk.

7.3.2 Non-prespecified Populations

Where PRO-AEs are being used in the post-approval phase with non-prespecified populations, the safety variables should be collected as comprehensively as possible from these subjects, including type of AE (using appropriate terms), severity, onset, and duration. Evaluable subjects are those for whom a retrospectively defined set of data elements is available. A suggested minimum dataset is provided in the Appendix (see the ESM), to which severity, onset and duration could also be added.

7.4 Statistical Evaluation Techniques

7.4.1 Prespecified Populations

Because, prior to approval, the range of possible adverse effects is very large, the assessment of unexpected PRO-AEs must be built into all protocols. As a result, the safety data (whether collected in the traditional manner or via a PRO-AE tool) are best addressed by applying descriptive statistical methods to the data, supplemented by calculation of confidence intervals wherever this aids interpretation. It is also valuable to make use of graphical presentations in which patterns of AEs are displayed both within treatment groups and within subjects. For example, presenting, in ascending order of risk, a series of AEs for the comparator and active drug together with their relative hazard ratio and confidence intervals is a good visual technique.

The calculation of p values can be useful to identify a specific difference or as a flagging device applied to a large number of safety variables to highlight differences worthy of further attention.

If hypothesis tests are used, statistical adjustments for multiplicity to quantify the Type I error are appropriate, but the Type II error is usually of more concern [40]. Care should be taken when interpreting putative statistically significant findings when there is no multiplicity adjustment.

In the majority of trials, investigators are seeking to establish that there are no clinically unacceptable differences in safety compared with either a comparator drug or placebo. As is the case for non-inferiority or equivalence evaluation of efficacy, the use of confidence intervals is preferred to hypothesis testing in this situation. In this way, the considerable imprecision often arising from low frequencies of occurrence is clearly demonstrated [40].

7.4.2 Non-prespecified Populations

There are a number of available approaches to aid the interpretation of PRO data, which could be adapted to PRO-AE data. They cover a spectrum of standard types of analyses for these types of data, including descriptive statistics, data mining/disproportionality, and multivariate analyses (Table 11).

Table 11 Statistical methods suitable for post-approval, non-prespecified datasets
Full size table

Other approaches include comparing a PROMs score with known clinical parameters, such as days in hospital and illness severity [41], the proportion of patients whose PROMs scores improve or worsen after intervention [42], and cross-instrument calibration using advanced statistical analyses such as item response theory (IRT) [43]. Although not specific to PRO-AE evaluation, such techniques might assist the quantification and assessment of patient-reported benefit–risk profiles. For longitudinal data, comparison of patient-level AUCs, or cumulative incidence of worst severity scores may be performed.

The test for statistical significance will depend on the approach used for the data analysis. This may include p values, standard deviations, and/or confidence intervals.

The use of statistics based on disproportionality is also possible in non-prespecified populations with non-structured datasets, but more innovative approaches are needed for this group (see next section on data mining).

7.5 Data Mining Techniques

The use of data mining for signal detection during clinical trials is a relatively new concept and allows researchers to perform ad hoc and close-to-real-time data analysis, reducing time in the interchange between researchers, clinicians and statisticians [45].

Traditionally, analysis has been carried out by a systematic manual review of every report sent by physicians to pharmacovigilance experts. These reports are registered in pharmacovigilance database systems.

More recently, automated signal detection methods have been developed to supplement qualitative clinical methods (see Table 12 and ESM 4). While these automated methods cannot replace expert clinical reviewers, they can assist with the difficult task of screening huge numbers of drug–event combinations in databases for potential signals. Through commonly used methods that are based on an underlying model of statistical association, databases are scrutinised for a significant occurrence of disproportionalities or dependencies between drug–event pairs.

Table 12 The use of data mining for different populations
Full size table

The WHO/UMC and the FDA are currently using automated detection algorithms based upon Bayesian analysis to achieve signal generation. These methods have a major role in prioritising signals by disproportionality, which are then clinically reviewed. The validity of the approach has been demonstrated in several papers and is now routinely practised [46]. Though the optimum algorithms are not established, there is much work being done in this area through the OMOP project [47–50].

Work has also been commenced on natural language processing to allow analysis of narratives and related terms [51].

MedDRA is the unified standard terminology currently used for recording and reporting adverse drug event data in most countries, but it may be necessary to extend/adapt it to include certain patient-reported terms being mapped to already accepted Preferred Terms (PTs)/Lower Level Terms (LLTs). In the meantime, verbatim terms reported by the patient can be reported directly, or grouped together into recognised symptom PTs.

Methods based on data warehousing and statistical analysis techniques provide various algorithms to identify trends or clusters of events, which may increase the accuracy and reliability of interpretation of safety data. The basic components are already available to companies for performing data mining of these safety databases for pharmacovigilance.

Statistical analysis system (SAS) tools are available for analytic processing, particularly to create complex analyses and reports. Utilising a suite of other SAS solutions enables companies to perform proactive pharmacovigilance. This suite may include data integration and tools to support clinical trials and drug development, which provides a standard, compliant environment for storage, retrieval, analysis, reporting and signal detection.

The signal detection process used by the WHO/UMC for analysis of VigiBase Centre consists of a combination of automated knowledge discovery methods [52], triage (prioritisation) algorithms and clinical review [53]. The knowledge discovery methods highlight drug–AR pairs with unexpectedly large numbers of reports relative to the average reporting ratios in the database. Triage algorithms use a combination of quantitative and qualitative information to focus attention on the key issues for follow-up [53]. Reports related to drug–AR pairs picked out by the triage algorithms are sent to an expert review panel, with pattern discovery methods often useful for profiling groups of reports and suggesting alternative explanations for increased reporting.

Hypotheses of suspected ARs first highlighted in automated knowledge discovery, which remain after clinical review, are then communicated to industry/regulators and published as appropriate [54, 55]. However, the risk of distortion from undiscovered data quality problems and the difficulty of obtaining complete, detailed information on reported AR incidents mean that signals of suspected ARs often remain tentative, even after clinical review [56].

A knowledge discovery process has been applied to the WHO/UMC VigiBase to resolve some of these shortcomings by examining the entire AR signal detection process [56]. The steps covered were data collection, cleaning and preparation; reduction and projection; data analysis and interpretation; and finally dissemination, incorporation into existing structures and action based on discovered knowledge. A duplicate detection method (the hit-miss model) that can reliably identify pairs of unexpectedly similar reports and a new measure for highlighting suspected drug–drug interaction have also been validated on the WHO/UMC database, which could be applied to compare PRO databases with overlapping medically confirmed spontaneous reports [56].

Examples of data mining techniques and related analytic approaches including predictive modelling, clustering or database segmentation, link analysis, and deviation detection are shown in ESM 4 in the Appendix (adapted from [57]).

8 Data Protection

Collection of identifiable patient data needs to meet the various international, regional (e.g. EU) and national data protection legal requirements, as appropriate. For example, real-world collection of identifiable patient data on the web, particularly when solicited as structured reports in the post-approval setting, must be done in a manner consistent with the relevant data protection legislation. The problem is complicated by the fact that the legislation would normally be enforceable in the country where the web collection tool originates, and also where any data is stored for safe harbour. Patients, of course, may be able to access and input their own data from other countries, and it is unclear to what extent these other countries’ data protection laws, if different from the enforceable country, need to be met.

9 Conclusions

The patient perspective is an essential component of drug safety profiles. Within this document, the PROSPER Consortium sets out an innovative framework for improving safety data during both clinical development and post-marketing phases by integrating the patient’s voice and perspective.

This comprehensive guidance suggests methods for including PRO-AEs as part of any structured questionnaire design, which may be used in both specified and non-prespecified populations. There is a valid distinction between the assessment and data mining of grouped, safety data, giving aggregate patterns of PRO-AE information, which do not necessarily require medical confirmation on an individual basis, as opposed to the assessment of single cases for which medical confirmation remains important.

The guidance proposes a minimum core dataset for use with unstructured data sources, details of which are provided in the Appendix (see the ESM). Various approaches to analytic methodology are considered, with the method of choice dependent on the specific patient-reported data being collected. For larger, non-prespecified populations and unstructured data, the quantitative methodology will require the use of more novel and emerging data mining approaches.

Patient-centeredness and patient safety are emerging as core elements of any responsive health system. As patients become more activated in their health and healthcare, they support the maxim of “Nothing about me, without me” and expect others involved in their care to engage in shared decision making, so that what matters to the patient is always included.

As stated at the outset of this document, the potential clinical utility and value of real-world information obtained from real patients taking real drugs is great [58]. Technological advances that were not available even a decade ago now offer unprecedented access for multidirectional communication between patients, clinicians, the private sector, industry and regulatory authorities.

We believe that the current guidance will help ensure that the voice of the patient will be heard, so that the safety profile and hence benefit–risk balance of new and existing medicines is better defined.

10 Glossary

Adverse event (AE): Any untoward medical occurrence in a patient or clinical trial subject administered a medicinal product, which does not necessarily have to have a causal relationship with this treatment [Dir 2001/20/EC 76 Art 2(m)]. An AE can therefore be any unfavourable and unintended sign (e.g. an abnormal laboratory finding), symptom, or disease temporally associated with the use of a medicinal product, whether or not considered related to the medicinal product [15].

Adverse reaction (AR): A response to a medicinal product that is noxious and unintended [DIR 2001/83/EC Art 1(11)]. ‘Response’ in this context means that a causal relationship between a medicinal product and an AE is at least a reasonable possibility.

Common Terminology Criteria for Adverse Events (CTCAE): A dictionary developed by the US NCI designed for use in clinical trials to aid clinicians in detecting and documenting AEs commonly encountered in oncology treatment.

Concept: The specific goal of measurement, i.e. the thing that is to be measured by the PRO instrument.

Data mining: The process of extracting meaningful patterns from large datasets using automated computational and statistical tools and techniques.

Disproportionality: Disproportionality analysis is an important tool for identifying new signals/patterns in spontaneous AE report databases. There are a variety of disproportionality measures (DPMs) used in safety signalling. DPMs are based on the ratio of the number of observed (O) cases and the number of expected (E) cases (i.e. O/E).

Domain: In a PRO instrument, a domain is a discrete concept within a multi-domain concept. All of the items within a single domain contribute to the measurement of the domain concept. The domain could be, but is not necessarily, considered a specific end point within the overall conceptual model.

Harm: Adverse impact on an individual’s health status.

Healthcare professional (HCP): Any qualified professional involved in the delivery of healthcare to the patient, whether or not involved in prescribing. Specifically, this term includes doctors, prescribers other than doctors, pharmacists, nurses, dieticians, and physiotherapists.

Health-related quality of life (HRQoL): A multi-domain concept that represents the patient’s overall perception of the impact of an illness, its treatment, and their overall state of health on their quality of life. This concept is used in FDA PRO effectiveness guidance (FDA, 2009).

Instrument: A structured means to capture PRO data, e.g. a questionnaire or diary along with supporting information such as training materials, scoring, etc. In the context of this guidance, this could be a specific reporting tool that through completion generates a structured dataset.

Item: An individual question within a PRO instrument that is evaluated by the patient.

Non-prespecified population: A group of patients where the number and type of patients included has not been pre-agreed.

Non-regulated consumer website: A website that is not set up with an aim to collect PRO-AEs, and where data posted will therefore be unstructured, and safety information posted is, at best, reviewed and collected.

Non-structured data: Data recorded without a data model.

Patient-reported outcome (PRO): The concept of any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else. For the purposes of this guidance, PROs are defined as encompassing the full range of self-reporting, ranging from those data collected via a structured and validated questionnaire to those collected in a free-text manner via spontaneous patient submissions (or in some cases from caregivers or surrogates).

Patient-reported outcome of adverse event (PRO-AE): A PRO-AE is any untoward medical occurrence, whether or not considered treatment/intervention related, which is reported or transmitted directly by the patient without interpretation by a clinician or anyone else. PRO-AEs may be collected by both structured and unstructured reports.

Patient-reported outcome measures (PROMs): An instrument, scale or single-item measure used to assess the PRO concept as perceived by the patient, obtained by directly asking the patient to self-report. The term has also been used in the UK-NHS, in the context of patient satisfaction, symptoms and functional status measures in relation to selected surgical procedures. In the context of epidemiology, AE outcome measures can be continuous variables or have binary outcomes.

Patient-reported outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE): A specific subset from the US NCI CTCAE project taxonomy; work is in progress.

Patient support programme (PSP): Sometimes also called ‘patient assistance programmes’, these may comprise non-interventional observational study designs or medical enquiry/patient assistance services. Safety reporting processes should be clearly defined by the market authorisation holder before the programme commences, to ensure that serious and non-serious AR reports are captured, as appropriate, and that causality assessments and regulatory reporting are performed, where required.

Post-approval/post-authorisation: Occurring after a medicinal product has received regulatory approval (post-licensing).

Pre-approval/pre-authorisation: Occurring prior to a medicinal product receiving regulatory approval.

Prespecified population: Enrolment of a specific patient group in conformance with a pre-agreed number and patient type or profile. This profile may be restricted (with inclusion/exclusion criteria) or unrestricted (no specific inclusion/exclusion criteria, but the size and/or geography of the group could still be limited).

PRO-AE-enabled patient support website: A specific website (whether company sponsored or not) that uses structured data collection in a scientifically robust way and is designed for patient education, interaction and self-reporting of PRO-AEs via a structured data collection instrument.

PRO-enabled: Data collection processes/systems and technologies that are adapted for the collection of patient-reported outcomes.

Registry: A defined group of patients, whether having a specific exposure (e.g. taking one or more specific treatments) or having a specific disease (i.e. a disease registry). A registry may be prespecified (pre-agreed dataset) or non-prespecified (open enrolment).

Regulated consumer website: A website that is set up with an aim to collect PRO-AEs, and where data posted may be either structured or unstructured, but safety information posted is regularly reviewed and collected.

Risk: Defined by the UMC as “the probability of harm being caused; the probability (chance, odds) of an occurrence”. The EU defines “risks related to use of a medicinal product” as “any risk relating to the quality, safety or efficacy of the medicinal product as regards patients’ health or public health; and any risk of undesirable effects on the environment” [DIR 2001/83/EC Art 1(28)].

Social network site: Web-based services that allow individuals to construct a public or semi-public profile within a bounded system; articulate to a list of other users with whom they share a connection; and view and traverse their list of connections and those made by others within the system. Non-medical social network sites are not specifically designed for a treatment or disease, but are where PRO-AEs may be reported incidentally, often in an unstructured way (e.g. Facebook or Twitter).

Structured data: An organised dataset based on a data model, in contrast to unstructured data that consists of open text fields.

Validation: The process of evaluating a PRO instrument’s ability to measure a specific concept or collection of concepts accurately.

Voluntary enrolment: Where patients are able to enrol themselves, whether prespecified (usually in the post-approval setting) or non-prespecified.