Review
Registered trials report less beneficial treatment effects than unregistered ones: a meta-epidemiological study in orthodontics

https://doi.org/10.1016/j.jclinepi.2018.04.017Get rights and content

Abstract

Objectives

Clinical trial registration is widely recommended because it allows tracking of trials that helps ensure full and unbiased reporting of their results. The aim of the present overview was to provide empirical evidence on bias associated with trial registration via a meta-epidemiological approach.

Study Design and Settings

Six databases were searched in September 2017 for randomized clinical trials and systematic reviews thereof assessing the effects of orthodontic clinical interventions. After duplicate study selection and data extraction, statistical analysis included a two-step meta-epidemiological approach within- and across-included meta-analyses with a Paule-Mandel random-effects model to calculate differences in standardized mean differences (ΔSMD) between registered and unregistered trials and their 95% confidence intervals (CI), followed by subgroup and sensitivity analyses.

Results

A total of 16 meta-analyses with 83 trials and 4,988 patients collectively were finally included, which indicated that registered trials reported less beneficial treatment effects than unregistered trials (ΔSMD = −0.36; 95% CI = −0.60, −0.12). Although some small-study effects were identified, sensitivity analyses according to precision and risk of bias indicated robustness.

Conclusion

Signs of bias from lack of trial protocol registration were found with nonregistered trials reporting more beneficial intervention effects than registered ones. Caution is warranted by the interpretation of nonregistered randomized trials or systematic reviews thereof.

Introduction

Randomized clinical trials are regarded as the gold standard in comparative efficacy research and form the basis for translating research evidence to clinical practice [1]. Among their advantages, methodological transparency is crucial and entails registration of the trial design protocol in a public domain prior to trial initiation to improve accountability in the conduct and reporting of research [2]. Trial protocols can be used post hoc to compare the original plan with subsequent procedures and analyses [3], thereby potentially reducing the risk of data dredging. A priori trial registration can additionally safeguard against bias-related phenomena such as delayed publication or nonpublication of trials, selective reporting of outcomes, manipulation of the analysis plan, and counting covert duplicate publications within systematic reviews as separate trials [3], [4], [5], [6].

Insights on trial characteristics systematically associated with treatment effects can be gleamed through meta-epidemiological studies. This is a subgenre of the big family of overviews of reviews, wherein data from a collection of meta-analyses is integrated and classified according to a specific study-level trait to empirically assess its influence on treatment effects [7].

A recent comprehensive meta-epidemiological study reported that nonregistered or retrospectively registered trials tend to show inflated treatment effects [8]. Although the effect was not statistically significant, this is confirmed by cross-sectional overviews of randomized trials in cardiology [9] or general medicine [10] that report weak associations between positive findings and trial registration, although the effects were not always consistent. It is also important to note that these analyses were based on qualitative evaluations of trial results by using either a P value cutoff or the trialists' interpretation of the results in text, both of which can be problematic.

The aim of this meta-epidemiological study is to provide empirical evidence of possible differences in the results of registered and nonregistered randomized clinical trials in orthodontics as a sign of bias, including its direction and magnitude.

Section snippets

Protocol, eligibility criteria, and registration

The protocol for this overview of reviews and trials was made a priori based loosely on the format of a systematic review, registered in PROSPERO (CRD42017072043), and all post hoc changes to the protocol were appropriately noted. According to the criteria set a priori, eligibility included parallel randomized clinical trials (or meta-analyses thereof) of human patients on any experimental intervention compared to a conventional or control group with any binary/continuous outcome in

Study selection

The literature search yielded a total of 1,241 hits electronically and 18 hits manually as of September 2, 2017; 216 of which, proceeded to full text assessment after eliminating duplicates and ineligible studies by title or abstract (Fig. 1 Supplementary Table 3a-3g). Finally, a total of 48 trials on similar comparisons were identified as eligible for inclusion in the present meta-epidemiological study. These were included in 16 separate meta-analyses on seven different topics, with some trial

Principal findings

The present overview summarizes empirical evidence up to September 2017 about the effects of trial registration on the results of orthodontic randomized clinical trials. Empirical evidence from 16 meta-analyses and 4,988 patients indicated that registered trials report considerably less beneficial treatment effects compared to nonregistered trials, which can be interpreted as signs of bias of moderate magnitude. The fact that bias from lack of registration was found to be of moderate magnitude

Acknowledgments

The authors thank Sophy K Barber (Leeds Dental Institute), Peter Miles (Caloundra, Queensland), Mueez Shedam (Rangoonwala Dental College and Research Centre), and Yanhua Xu (Kunming Medical University School) for providing details of their studies.

References (40)

  • K. Dwan et al.

    Comparison of protocols and registry entries to published reports for randomised controlled trials

    Cochrane Database Syst Rev

    (2011)
  • M.R. Tramèr et al.

    Impact of covert duplicate publication on meta-analysis: a case study

    BMJ

    (1997)
  • F. Song et al.

    Publication and related biases

    Health Technol Assess

    (2000)
  • A. Dechartres et al.

    Association between trial registration and treatment effect estimates: a meta-epidemiological study

    BMC Med

    (2016)
  • C. Emdin et al.

    Association of cardiovascular trial registration with positive study findings: epidemiological Study of Randomized Trials (ESORT)

    JAMA Intern Med

    (2014)
  • A. Odutayo et al.

    Association between trial registration and positive study findings: cross sectional study (Epidemiological Study of Randomized Trials-ESORT)

    BMJ

    (2017)
  • E. Lesaffre et al.

    Reported methodological quality of split-mouth studies

    J Clin Periodontol

    (2007)
  • D. Koletsi et al.

    Does published orthodontic research account for clustering effects during statistical data analysis?

    Eur J Orthod

    (2012)
  • O. Ciani et al.

    Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study

    BMJ

    (2013)
  • Cited by (20)

    • Making ERP research more transparent: Guidelines for preregistration

      2021, International Journal of Psychophysiology
      Citation Excerpt :

      Results showed that 57% of the studies published before 2000 showed a significant benefit of the intervention, as opposed to only 8% of trials published after 2000.7 Similar results were reported in a (preregistered) meta-analysis of meta-analyses of orthodontics and dentofacial orthopedics studies: registered trials reported less favorable intervention effects compared to unregistered trials (Papageorgiou et al., 2018). Preregistration can also help identify whether funding sources are correlated with study outcome, potentially uncovering questionable practices due to (undisclosed) conflicts of interest.

    • Development of a core outcome set for use in routine orthodontic clinical trials

      2020, American Journal of Orthodontics and Dentofacial Orthopedics
    View all citing articles on Scopus

    Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

    Data availability: The datasets and commands of this study are openly available through Zenodo (http://doi.org/10.5281/zenodo.1186318).

    Conflicts of interest: The authors declare no conflicts of interest to disclose.

    View full text