Article Text
PDF
Abstract
Background Substantial amounts of public funds are invested in health research worldwide. Publicly funded randomised controlled trials (RCTs) often recruit participants at a slower than anticipated rate. Many trials fail to reach their planned sample size within the envisaged trial timescale and trial funding envelope.
Objectives To review the consent, recruitment and retention rates for single and multicentre randomised control trials funded and published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme.
Data sources and study selection HTA reports of individually randomised single or multicentre RCTs published from the start of 2004 to the end of April 2016 were reviewed.
Data extraction Information was extracted, relating to the trial characteristics, sample size, recruitment and retention by two independent reviewers.
Main outcome measures Target sample size and whether it was achieved; recruitment rates (number of participants recruited per centre per month) and retention rates (randomised participants retained and assessed with valid primary outcome data).
Results This review identified 151 individually RCTs from 787 NIHR HTA reports. The final recruitment target sample size was achieved in 56% (85/151) of the RCTs and more than 80% of the final target sample size was achieved for 79% of the RCTs (119/151). The median recruitment rate (participants per centre per month) was found to be 0.92 (IQR 0.43–2.79) and the median retention rate (proportion of participants with valid primary outcome data at follow-up) was estimated at 89% (IQR 79–97%).
Conclusions There is considerable variation in the consent, recruitment and retention rates in publicly funded RCTs. Investigators should bear this in mind at the planning stage of their study and not be overly optimistic about their recruitment projections.
- recruitment
- Retention
- Randomised Controlled Trials
- Health Technology Assessment
- Publicly funded
- review
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Substantial amounts of public funds are spent on healthcare research and randomised controlled trials (RCTs) and this is potentially wasted if a trial fails to recruit to time and target sample size. Trialists and funders have highlighted recruitment and retention as a key issue for the conduct of RCTs.
This study reports the recruitment and retention rates for 151 single and multicentre randomised control trials funded by the UK's National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme and published in the HTA Journal between 2004 and 2016.
There is considerable variation in the consent, recruitment and retention rates in publicly funded RCTs.
Crude recruitment rates, assuming all centres were recruiting for the same time period were calculated; as such the recruitment rate estimates may be an underestimation of the true recruitment rate. The study was restricted to publicly funded RCTs published as reports in the HTA Journal and not commercially sponsored trials.
Recruitment to trials is complex and the complete picture cannot be untangled in a simple review.
Introduction
Substantial amounts of public funds are invested in medical research worldwide with an estimate of US$100 billion in 2012.1 In 2014/2015, the National Institute for Health Research (NIHR) in England spent £237.6 million across a broad range of research programmes and initiatives to ensure that patients and the public benefit from the most cost-effective, up-to-date health interventions and treatments.2 A substantial proportion of this research expenditure was invested in Randomised Controlled Trials (RCTs) to assess the clinical and cost-effectiveness of new health technologies. RCTs are widely regarded as the most powerful research design for evaluating new health technologies and decision makers, such as the UK's National Institute for Health and Care Excellence (NICE), are increasingly using the results of RCTs to guide practice and policy.
A frequently reported problem with publicly funded RCTs is that the recruitment of participants is often slower or more difficult than expected. Many trials fail to reach their planned sample size within the originally envisaged trial timescale and trial funding envelope. A review of a cohort of 122 trials funded by the UK Medical Research Council (MRC) and the NIHR Health Technology Assessment Programme (HTA), between 1994 and 2002 found that less than a third (31%) of the trials achieved their original patient recruitment target; 55/122 (45.1%) achieved < 80% of their original target and half (53%) were awarded an extension.3 This situation has improved marginally over time, with a recent review of 73 HTA/MRC funded studies recruiting between 2002 and 2008,4 finding that 55% (40/73) of the trials achieved their original patient recruitment target; 16/73 (22%) achieved <80% of their original target and 45% (33/73) were awarded an extension.4
A HTA commissioned review recommended further research is required, particularly in relation to: problems being experienced and solutions employed in current RCTs; the optimum structure, staffing and organisation for the conduct of large and small trials; and the factors which influence the participation5 in RCTs. Several Cochrane systematic reviews have suggested strategies to improve the recruitment6 ,7 and retention8 of participants to RCTs. These recruitment strategies include: telephone reminders; requiring potential participants to opt-out of being contacted by the trial team regarding participation; and open (unblinded) designs. Other HTA commissioned reviews have shown that participant and clinician preferences, for the intervention, can affect trial recruitment9 and that payment to healthcare professionals for patient recruitment to trials acts as a limited incentive.10 However despite the growing literature summarising the barriers and facilitators to recruitment to RCTs only 55% of trials are recruiting to within 80% of the target. A recent survey among the directors of the clinical trials units registered with the UK NIHR Clinical Research Network identified priorities for research into the methodology of trials. The top three priorities were improving recruitment, choice of outcomes and improving retention.11
The Consolidated Standards of Reporting Trials (CONSORT) Statement, first published in 1996,12 ,13 and revised in 200114 and 2010,15 is a set of standards for publication of results of RCTS in medical journals. They are for the article itself and the article abstract.15 The CONSORT statement includes details of the number of eligible patients; number of patients randomised; number of recruiting centres and recruitment time period (start and finish time of recruitment). A review of publicly funded RCTs was carried out to evaluate how well recruitment and retention figures are reported; how successful RCTs are in reaching their target sample size and retaining participants and to assess recruitment rates.
Methods
Trial identification
Reports of individually RCTs published in the NIHR HTA Journal from January 2004 to April 2016 were reviewed. The HTA Journal publishes research on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. Reports are published in the HTA Journal if (1) they have resulted from work for the NIHR HTA Programme and (2) they are of a sufficiently high scientific quality as assessed by the external reviewers and journal editors (http://www.journalslibrary.nihr.ac.uk/hta/about-the-journal). Trial reports published in the HTA Journal were chosen as they are of high quality and provide detailed trial and recruitment information including the number of centres and recruitment period.
A pilot review of 30 trials reported in five major journals: British Medical Journal (BMJ), The Lancet, New England Journal of Medicine, The Journal of the American Medical Association (JAMA) and Annals of Internal Medicine as well as six trials reported in the HTA Journal, found that there was sufficient information to calculate the recruitment rate for only 23 out of the 30 trials reported in the journals. This information was available for all six trials reported in the HTA Journal. The pilot identified six trials published in the HTA journal over a 7 months period and with a basic extrapolation over a 12 year publication period it was deemed this would provide a large and manageable number of trials for review. Limiting the review to publicly funded trials published in the HTA Journal identified trials from medicine, surgery and therapy as well as from a range of disease areas.
The HTA Journal reports were obtained from the NIHR Journals Library website (http://www.journalslibrary.nihr.ac.uk/hta—date last accessed 10 August 2016) along with any previously published trial paper, protocol paper or trial protocol, where available. For trials that had a published International Standardised Randomised Controlled Trial Number (ISRCTN) number this was used to check the ISRCTN register of clinical trials for any additional information, a trial website or any previously unobtainable trial reports (cf. http://www.isrctn.com/). The trial report published in the HTA Journal was used as the main resource where there were discrepancies in reporting. The titles and abstracts of all reports published in the HTA journal from January 2004 to April 2016 were checked for relevance. January 2004 was chosen as a start date for the review because there were relatively few reports of RCTs in the first 7 years of the HTA Journal from 1997 to 2003 (13 RCTs out of 208 reports).
Inclusion/exclusion criteria
Trials included in the review were single and multicentre RCTs that were either fully or partially randomised and recruitment to the trial had finished. Where trials had reported early termination, either prior to completion of recruitment or following recruitment but prior to completion of follow-up, the trials were retained in the review and the reason for the termination was noted. Trials that were nested parallel trials as part of another RCT were included as were trial reports of two or more parallel RCTs. Trials that were excluded include: cluster randomised trials as these have separate specific recruitment issues;16 adaptive designs and pilot trials. HTA reports of a pilot trial that went on to a full trial were retained and the results from the full trial were extracted and included in the review. Trials of influenza vaccination were excluded as these recruit patients over a short period of time, usually 1–3 months, and so have an exceptionally high recruitment rate.
Data extraction
Once HTA reports had been selected for inclusion, information was extracted, using a standardised data extraction form. For each trial the following general trial information was extracted: the trial design, the clinical area, use of blinding, intervention type, type of control, number of arms, single or multicentre and number of centres, recruitment setting and the number and timing of follow-up visits. Data relating to the sample size and recruitment rate was extracted including: the target and actual sample size, the overall and centre-specific recruitment period and CONSORT information on the numbers screened, consented, randomised and analysed with the primary outcome.17 Where available more detailed trial information was recorded including: use of a pilot, whether there was support from a trials unit, geographical region, recruitment strategy and country where the trial took place. Data extraction was carried out by a team of reviewers. Each article was independently reviewed by a second member of the review team. Any uncertainties were resolved by discussion.
The standard of reporting of trial information was good but for some variables the details were not always available. There was limited information about whether trials had any form of pilot phase or had involvement from a clinical trials unit. These features were recorded as absent where they were not mentioned.
The primary outcome for the review was considered to be the recruitment rate for each trial. To calculate this accurately the centre-specific recruitment periods within the trials were extracted. However, this was generally poorly reported.
Analysis
The recruitment rate was defined as the number of participants recruited and randomised per centre per month. This was summarised and compared using median rates and the IQR due to the skewness of the distribution of the data.18 For secondary outcomes, the percentage of eligible patients randomised and randomised patients assessed and analysed with the primary outcome (retention), were expressed in terms of the median and IQR. Comparisons were made between categories of different characteristics using appropriate non-parametric tests; Wilcoxon rank sum (for characteristics with two levels), Kruskal-Wallis (three or more nominal levels) and non-parametric test of trend (three or more ordered levels)19). Additionally, associations between trial characteristics and recruitment rates were investigated individually using Wilcoxon rank sum tests to compare trials on the presence and absence of certain characteristics. Analysis was carried out on a complete case basis so where the characteristic information, the recruitment or retention data were missing these were excluded. Data were collected in excel and transferred to the R statistical software for analysis.20
The recruitment period was calculated as the time between dates of recruitment start and recruitment completion. If only months were reported the recruitment period was estimated as the time between the 1st of the first month and the end of the final month unless explicitly stated otherwise. If the date of the first recruit was reported instead of the date of start of recruitment then the 1st of the month of the first recruit was taken as the start of recruitment. Start of screening was used to calculate the recruitment period where the start of recruitment was not reported. In cases where information on the start and end of the recruitment period was not explicitly reported this was estimated from subtracting the length of the follow-up period from the length of study period where this was suitably reported.
The recruitment rate was calculated in two distinct ways. First, to calculate the overall recruitment rate, the total number of patients recruited was divided by the maximum number of sites recruiting, then divided by total number of months that the trial recruited for. In reality the opening of trial sites is likely to be staggered. For the majority of trials most sites do not recruit for the entire recruitment period. For this reason this estimate of the overall recruitment rate for multicentre trials is likely to be an underestimate. To account for the differences in start-up times for sites and the corresponding site-specific recruitment periods, where available, the centre-specific recruitment periods were extracted. These were averaged over the number of sites to give an average centre-specific recruitment period. An average recruitment rate was calculated as the total number of patients recruited, divided by the maximum number of sites and then divided by the average number of months recruiting.
Results
In total 778 reports were published between January 2004 and April 2016 in the HTA Journal and 596 of these were excluded following screening of all titles and abstracts. The search produced 191 trial reports of randomised trials of which a further 40 were excluded for various reasons (18 cluster RCTs; 15 pilot/feasibility studies; 3 influenza vaccination trials and four excluded for other reasons). The 15 pilot/feasibility studies were standalone/external trials at the outset and were not definitive trials that were changed to a pilot study as a result of poor recruitment. In total, 151 individually RCTs were included in the review and analysed as shown in figure 1.
Flow diagram of search process for a review of trial reports published in the Health Technology Assessment Journal between 2004 and end of April 2016. RCT, randomised controlled trials.
Trial characteristics
The characteristics of the 151 trials included in the review are summarised in table 1. The majority of trials were two armed, parallel group, multicentre trials. Trials were identified from a variety of different clinical areas with 18% (27/151) of trials in mental health, including neurosciences, psychiatry and psychology, and 13% (19/151) of trials of musculoskeletal conditions, including orthopaedics, rheumatology and back pain. Trials were most commonly set in hospitals (54% (82/151)) and 91% (137/151) took place solely in the UK. Drug trials were as common as therapy trials, both occurred more frequently than surgery trials. 30% (46/151) of trials used an intervention that was not easily categorised and there was a far greater number of trials that used an active control (80% (121/151)) compared with a placebo (20% (30/151)).
Characteristics of the trials included in the review
The trial characteristics relating to recruitment and sample size are summarised in table 2. The final recruitment/sample size target ranged from 44 participants to 28 000 and final total actual/achieved recruitment ranged from 19 participants to 24 510. Two trials targeted and achieved recruitment of 20 000 or more participants; one of these was a trial of trauma patients21 and the other was a cervical screening trial.22 Overall 56% (85/151) of trials recruited to their final recruitment target and 79% (119/151) managed to recruit to within 80% of the recruitment target. For 34% (52/151) of trials the original sample size target was revised (downward in 79% (41/52)). Eight single-centre trials were identified. Five trials recruited in more than 100 centres; the maximum number of centres was 274. The majority of trials had a final follow-up visit at 18 months or less postrandomisation and the longest reported final follow-up was 10 years postrandomisation.
Recruitment and sample size characteristics of the trials included in the review
CONSORT and recruitment data
The data completeness in relation to CONSORT and recruitment information is summarised in table 3. Out of the 151 trials identified 95 (63%) demonstrated complete compliance with the CONSORT statement and reported each of the number: screened, eligible, declined consent, recruited and assessed in their primary outcome. The number of participants recruited, randomised and assessed for the primary outcome, used to measure retention, was available for all 151 trials. To calculate the recruitment rate 144 out of 151 trials reported the maximum length of the recruitment period, from first centre opening to completion of recruitment, and 106 reported the total number of centres that recruited at least one participant. Centre-specific recruitment information, used to calculate an average recruitment period per centre, could only be extracted from 34 of the 111 trials (25%). The overall recruitment rate, based on the maximum recruitment length, was calculated for 142 out of 151 RCTs.
Data completeness in relation to CONSORT guidelines and recruitment information
Recruitment and retention rates
From the 142/151 (94%) trials with sufficient information the median recruitment rate was found to be 0.92 patients recruited per centre per month. This ranged from 0.04 to 57.75 patients per centre per month, with 80th and 90th percentiles of 4.4 and 10.1 patients recruited per centre per month, respectively. The two studies found to have the largest recruitment rates were single-centre studies,23 ,24 recruited from obstetrics and gynaecological populations (figure 2). The eight single-centre studies produced five of the nine23–31 largest recruitment rates ranging from 16 to 58 patients per centre per month. Taking the multicentre studies on their own the median recruitment was 0.86 patients recruited per centre per month with a range from 0.04 to 30.11 patients recruited per centre per month. A median of 70% (IQR 51–87%) of eligible patients were consented and randomised and a median 89% (IQR 79–97%) of randomised patients had valid primary outcome data for analysis (table 4).
Overall recruitment and retention rates
Recruitment Rates by clinical area for the 151 Health Technology Assessment trials considered.
Tables 5 and 6 summarise the trial recruitment and retention rates by various trial characteristics (setting, number of arms, control type, original and final recruitment targets, total number recruited and time of follow-up). There is some statistical evidence of an association between trial setting, final recruitment target and the total number recruited although there is no clear pattern to these associations.
Association between recruitment rate (number of patients/centre/month) and trial characteristics
Association between the trial retention rate (% of randomised participants with valid primary outcome data for analysis) and trial characteristics
Table 7 compares the results of the current review, in terms of successful recruitment to target sample size, with two previous reviews.3 ,4 It should be noted that there is some overlap in the trials included in our review and Sully et al; so we have included a column with the non-overlapping time interval for the 2009–2016 data. Table 7 shows that reaching 100% of the original sample size target is lower in 2009–2016 than previous periods/reviews; with only 50% (45/90) achieving the original sample size target. The original sample size target was revised in 39% (35/90) of trials; and this revision was downwards for the majority of trials, 71% (25/35).
Comparison of current review with results of two previous reviews in terms of successful recruitment to target sample size and extensions to recruitment
Discussion and conclusions
This study provides a comprehensive review of the recruitment and retention data of a cohort of 151 RCTs funded and published by the UK NIHR HTA Programme from 2004 to 2016. This review found that the final recruitment target sample size was achieved in 56% (85/151) of the RCTs; the median recruitment rate (participants per centre per month) was 0.92 (IQR 0.43–2.79) and the median retention rate 89% (IQR 79–97%).
This review found that 56% of publicly funded RCTs achieve their target sample size, a similar figure to that found in the most recent review of 55%, by Sully et al which covered the period of 2002 to 2008.4 However, there is still a suggestion that recruitment success is improving slightly compared with the previous review covering the period of 1994 to 2002.3 Even though the recruitment picture is improving there is certainly still room for improvement with more than half of the 151 publicly funded RCTs not recruiting to target which in some cases was revised down during the course of the trial. These findings are congruent with the concerns of clinical trials unit directors.11
There is a possible relationship between planned sample size and recruitment rate with recruitment rate increasing as the target sample size increases. Sample sizes are inflated for expected attrition or non-response and this is commonly set at 10–20%.18 ,32 The estimate of average retention was 89% suggesting that the current inflation of sample sizes for attrition is reasonable. Overall retention is not as big an issue as recruitment in terms of fulfilling a sample size for a primary outcome. These findings slightly contrast with the concerns of clinical trials unit directors'.11 The retention figure, however, will be affected by the number of trials with short-term outcomes and the use of survival analysis methods with time to event outcomes, where missing outcomes are typically censored at the time of any loss to follow-up and but included in the analysis.
This study has several limitations. Data extraction was carried out by two independent reviewers. Reviewers conferred to try to ensure consistency in the interpretation of data extraction items; but it is possible that errors have occurred. Crude recruitment rates, assuming all centres were recruiting for the same time period were calculated; as such the recruitment rate estimates may be an underestimation of the true recruitment rate. The study was restricted to publicly funded RCTs published as reports in the HTA Journal and not commercially sponsored trials. There is a possibility of publication bias as this study is restricted to trials that have had their results published in the NIHR HTA Journal; as not all funded trials are actually published. However the possibility of publication bias is small as a review of projects funded by the NIHR HTA Programme, between 2002 and 2011, found that 98% (274/280) published in the programme's journal.33 The HTA's expectation (in line with their contract) is that all HTA Programme funded studies publish in the NIHR Journals Library, even when they have had to close early because of, for example, poor recruitment. The HTA will ask investigators to include a section/chapter on the challenges faced and lessons learnt that will then inform other researchers who might be considering similar research.
There are limitations in drawing conclusions from this data, not least in the accuracy of the recruitment rates calculated, potential confounders not accounted for and in some cases underlying factors that cannot be measured in the data. Recruitment to trials is complex and the complete picture cannot be untangled in a simple review. This review does, however, provide some pointers to factors that might need to be considered when estimating recruitment periods for RCTs and could be used in models of recruitment projection. Recent qualitative research has also highlighted that realistic estimation of recruitment rates is complex and that early planning and pilot and feasibility work to help project trial recruitment is important.34
In practice, recruitment rates will vary, depending on whether the target population is acute, where opportunistic recruitment will target incident cases, or chronic, where database recruitment can effectively target prevalent cases.34 ,35 It will also vary according to whether the intervention is therapeutic or preventive36 and the base incidence and prevalence rate of the condition.
Based on this review for most publicly funded trials the recruitment rate is likely to be between 1 and 2 participants per centre per week (4–10 a month). There is considerable variation in the consent, recruitment and retention rates in publicly funded RCTs. Investigators should bear this in mind at the planning stage of their study and not be overly optimistic about their recruitment projections.
References
Footnotes
Contributors SJW is the guarantor of the study, had full access to all the data in the study, and is responsible for the integrity of the data and the accuracy of the data analysis. SJW contributed to the study conception and design, acquisition of data, analysis and interpretation of data, and writing of the report. JR contributed towards the selection of relevant data and extraction of data, as well as the drafting of the paper. CK contributed to the study design and the selection and extraction of the data. IBdAH-C contributed towards the selection, extraction and analysis of the data, as well as the drafting of the paper and graphics within it. OB contributed to the selection and extraction of the data, as well as the drafting of the paper. RMJ contributed to the selection and extraction of the data, as well as the drafting of the paper. MS contributed to the selection and extraction of the data, as well as the drafting of the paper. DH contributed to the study conception and design, the acquisition of data, the interpretation of data, as well as the drafting of the paper. SAJ contributed to the selection and extraction of the data, as well as the drafting of the paper. BN contributed to the selection and extraction of the data, as well as the drafting of the paper. LF contributed to the selection and extraction of the data, as well as the drafting of the paper.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare (1) no financial support for the submitted work from anyone other than their employer; (2) no financial relationships with commercial entities that might have an interest in the submitted work; (3) no spouses, partners or children with relationships with commercial entities that might have an interest in the submitted work; (4) No non-financial interests that may be relevant to the submitted work.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data set available on request from the corresponding author at s.j.walters@sheffield.ac.uk