Should meta-analysts search Embase in addition to Medline?

Abstract

It is widely accepted that meta-analysts should search multiple databases. The selection of databases is ideally based on the potential contribution of each database to the project or on the potential for bias if a database is excluded, as supported by research evidence. We explore whether searching Embase yields additional trials that influence a meta-analysis. We identified meta-analyses that searched Medline and Embase. A random-effects weighted mean method was used to estimate the intervention effect in articles indexed only in Embase compared with those indexed elsewhere. On average, Embase-unique trials yielded significantly smaller estimates by 29% (ratio of odds ratio [ROR] 0.71, 95% confidence interval [CI] 0.56–0.90) but influenced the pooled estimate by an average of only 6% (ROR 0.94, 95% CI 0.88–0.99). Searching Medline but not Embase risks biasing a meta-analysis by finding studies that show larger estimates, but their prevalence seems low enough that the risk may be slight, provided the rest of the search is comprehensive.

Introduction

Comprehensive searching is considered standard practice when conducting meta-analyses (MAs) of randomized controlled trials (RCTs) [1], [2], [3]. This entails searching multiple classes of sources (bibliographic databases, reference lists, trials registries, hand searching, etc.) and all relevant sources within each class. There are many databases that could be searched in the course of a meta-analysis. For instance, Dialog, an information aggregator, lists more than 20 bibliographic databases and five full-text databases in its Medicine collection [4]. The typical systematic reviewer searches a small subset of the available databases, even in an extensive search (the median number searched in our sample was 3, and the maximum was 7). The available time and financial resources impose some limits on what can reasonably be undertaken [5]. The professional expertise of a research librarian can guide the choice, but ideally the selection of databases should be informed by the evidence regarding the contribution of the source to the validity of the results of the research to be undertaken. Indeed, Eldredge introduces evidence-based librarianship as that which uses the best available evidence based upon library science research to arrive at sound decisions about solving practical problems in librarianship [5].

There are two prominent bibliographic databases that are relevant for most meta-analyses: Embase and Medline. One of the most difficult choices for reviewers with limited resources is whether or not to search both. Numerous studies have examined the performance of Embase and Medline as tools to identify RCTs [6], [7], [8], [9], [10], [11], and these studies have established that no single database indexes all RCTs. Both contain bibliographic records with citation information and in many cases abstracts of the articles. Both are professionally indexed using a controlled vocabulary. Both are available through several database re-sellers who apply their own search interfaces. Access costs vary according to the licensing arrangement in effect and may be borne by the researcher or paid through an institutional or consortium subscription. Medline is produced by the US National Library of Medline. It includes over 10 million citations since 1966, and more than over 3600 journals are indexed. One version of Medline is available free of charge over Internet through the National Library of Medicine's PubMed service. Embase is produced by Elsevier Science. It includes over 3 million citations since 1980, and more than 3600 journals are indexed.

Even though there are strong recommendations to search multiple databases, there are some disincentives to doing so: Additional searches increase the time and costs of searching and often yield irrelevant trials or result in no additional trials [12]. This may help to explain why over half of meta-analyses report searching only one electronic database, usually Medline [13], [14].

It is not clear if limiting the databases searched introduces bias into the results of meta-analyses. Bias could arise, for example, from differential coverage of languages [15] or from North American emphasis [16]. In part, bias may depend on the subject of the meta-analysis; for instance, Watson and Richardson found a significant number of RCTs in a psychiatric topic in the PsycINFO database [10]. Regardless of the source of bias, the test is not simply whether reports of RCTs exist in one database and not another, but whether the articles influence the estimate of effect size in a meta-analysis. To have such an influence requires that the trials meet all inclusion criteria for a meta-analysis and differ systematically in effect size from those identified through other sources. Before including studies in the analysis, meta-analysts impose additional criteria, considering factors such as the similarity across trials in terms of participant populations and outcome measures.

In this study, we attempt to determine whether Embase plays a useful role in identifying material that would meet all inclusion criteria for an analysis and would not otherwise be available through a search of the Medline database. That is, does the incremental contribution of Embase influence the findings of a meta-analysis such that the decision not to search Embase would potentially introduce bias into the results? We searched for Embase-unique trials (EMU); that is, those that are indexed in Embase but not in Medline and are also included in the estimate of effect in a meta-analysis of RCTs.