Intended for healthcare professionals

  1. News & Views
  2. Bias in meta-analysis...
  3. Bias in meta-analysis detected by a simple, graphical test
Letters

Bias in meta-analysis detected by a simple, graphical test

BMJ 1998; 316 doi: https://doi.org/10.1136/bmj.316.7129.469 (Published 07 February 1998) Cite this as: BMJ 1998;316:469

Asymmetry detected in funnel plot was probably due to true heterogeneity

  1. Andreas E Stuck, Chiefa,
  2. Laurence Z Rubenstein, Professor of geriatric medicineb,
  3. Darryl Wieland, Professorc
  1. a Department of Geriatrics and Rehabilitation, Zieglerspital, Berne, Switzerland
  2. b Education and Clinical Center, UCLA-Sepulveda VA Medical Center, Los Angeles, CA, USA
  3. c Department of Medicine, Division of Geriatrics, University of South Carolina School of Medicine, Columbia, SC, USA
  4. d Department of Clinical Epidemiology, Leiden University Hospital, 2300 RC Leiden, Netherlands
  5. e Department of Public Health and Community Medicine, A27, University of Sydney, NSW 2006, Australia
  6. f Department of Social and Preventive Medicine, University of Queensland, Medical School, Herston, QLD 4006, Australia
  7. g Unit of Healthcare Epidemiology, Institute of Health Sciences, Oxford University, Oxford OX3 7LF
  8. h Diabetes Research Laboratories, Radcliffe Infirmary, Oxford OX2 6HE
  9. i Department of Social Medicine, University of Bristol, Bristol BS8 2PR
  10. j Department of Social and Preventive Medicine, University of Berne, Switzerland
  11. k Academic Section of Geriatric Medicine, Royal Infirmary, Glasgow G4 0SF
  12. l NHS Centre for Reviews and Dissemination, University of York, York YO1 5DD

    Editor—Egger et al report that they “found bias in 38% of meta-analyses published in four leading journals.”1 This is misleading, at least insofar as our meta-analysis of inpatient geriatric consultations is concerned.2

    Firstly, the bias observed in our meta-analysis was not a retrospective detection of bias, as one might infer from Egger et al's statements. We knew that there was evidence of heterogeneity for the pooled effect estimates of geriatric consultation programmes and reported this finding.2 Secondly, the asymmetry detected in the funnel plot of the meta-analysis of inpatient consultation programmes was probably due not to bias (distortion of true effect) but to true heterogeneity (true difference of effects between trials). We took the presence of heterogeneity as an opportunity to examine whether we could identify the programme elements that might have resulted in the observed effect differences between geriatric consultation programmes. Using a multivariate logistic regression approach, we found that both geriatric assessment programmes in which the consultant controlled the implementation of the recommendations and those that included long term follow up resulted in better outcomes than did programmes in which this was not the case.

    Thus, the meta-analytical methods of testing heterogeneity or drawing funnel plots should not be considered absolute criteria for separating good from bad meta-analyses. Meta-analyses reporting effect estimates that may contain bias should continue to be published in leading medical journals, as long as the possibility of heterogeneity is stated and potential underlying reasons for heterogeneity are addressed. This is especially true for meta-analyses of complex interventions. Although they are methodologically difficult to deal with, variations in effect estimates give us the opportunity to disentangle the black box of complex interventions, such as of geriatric assessment, and identify what the necessary ingredients of these programmes are.3

    A third issue concerns …

    View Full Text

    Log in

    Log in through your institution

    Subscribe

    * For online subscription

    Buy this article

    Back to top