Examining the readiness of best evidence in medical education guides for integration into educational practice: A meta-synthesis

We considered quality in relation to how each Guide fulfilled the 25 elements of the STORIES Statement. All reviews fulfilled the following four criteria: contained a structured summary, provided rationale for conducting the review, summarized details of the included studies, and presented findings in light of stated objectives. The least satisfied criteria (n = 8 Guides) pertained to research teams contacting external individuals familiar with the topic. Below we highlight several key criteria. Detailed information for each criteria is available in Supplement 1: https://​figshare.​com/​s/​590bcc368c760928​0aec.

We considered accessibility in terms of readers’ ability to access, either in hard-copy or online, the full-text of a Guide and its supporting materials. Detailed online accessibility information for each article is available in Supplement 1 (Available at: https://​figshare.​com/​s/​590bcc368c760928​0aec.) Additionally, we examined the steps authors took (e.g., inclusion of practice points) to make Guides cognitively accessible for a variety of readers, including busy clinician-educators who may not be engaged in HPE research as a primary duty.

The full-texts of all Guides are available by subscription in Medical Teacher. Of the included Guides, 26 were freely accessible from the BEME website. These free versions tended to be the author’s final draft, meaning that readers may encounter documents with author ‘track changes’ and tables and figures missing or located at the end of the manuscript. On the Medical Teacher website, 11 reviews were freely available as of 24 March 2018, and several of them overlapped with those featured on the BEME website. As of 24 March 2018, 11 Guides were not publicly accessible, all of which were published between 2015–2017, thereby comprising the majority of recently published Guides. For most Guides, components of the methods and results, such as search strategies, data abstraction tools, and results tables, were made available on the BEME and/or Medical Teacher websites with neither location containing all the materials.

Guides included or made available supplemental materials to increase the cognitive accessibility for readers at differing levels of engagement with HPE research. For example, all Guides featured publicly accessible abstracts. Currently, Medical Teacher requires structured abstracts under 200 words. Reviews published from 2008–2017 followed this convention. However, reviews published prior to 2008 were significantly longer and included robust method sections and, in many cases, ‘headline results’ or ‘highlight points’ set off with bullet points. These brief points connected readers with key findings for practice, making the material more readily accessible. For example, one review included the practice point ‘educational feedback is the most important feature of simulation-based medical education [32].’ Additionally, within the body of 35 Guides, authors included practice points. Practice points, generally featured in the Guide’s introduction section, are 4–6 bulleted points, offset in a box, designed to provide readers a brief snapshot of major findings. For example one review noted, ‘It is key that portfolio implementation is well-designed and sustained, with high-level organizational support, to ensure uptake’ [47] and another that ‘The Education for Sustainable Healthcare framework can guide curricula and teaching development [15].’

External to the Guides themselves, 34 authors included a ‘BEME Spotlight.’ A BEME spotlight is a two-page structured summary of the Guide’s main conclusions and recommendations. To host these materials, and others, each Guide has a webpage on the BEME website (https://​www.​bemecollaboratio​n.​org/​Published+Review​s/​). Page contents vary by Guide, but generally included links to the Guide’s full-text (if publicly available), the review protocol, an executive summary, PowerPoint slides from conference presentations, conference posters, and the Medical Teacher website. For example, the BEME site for O’Dunn-Orto’s Guide on the musculoskeletal exam [37] features links to its abstract, a publicly-accessible MS Word version of the full-text, a conference poster, a PowerPoint presentation, and the associated Medical Teacher article. While the BEME and Medical Teacher websites indicate via links and/or references that these materials exist, if a practitioner were to access a Guide via PDF, there is no indication in the PDF document that these supplemental materials exist.

Relevance is the ‘degree to which something is related or useful to what is happening or being talked about [58]’. Because relevance has a personal dimension, and because a given teaching approach may be relevant to one educator and not to another, we did not judge each Guide’s relevance, per se. Instead, we report on the elements provided that might inform readers in making their own relevance judgments based on their individual practice situations. Elements related to relevance included: implications for researchers and educators, inclusion of stakeholders, inclusion of educational theories and/or models, and currency.

Practitioners infrequently apply generic evidence to their educational practice without first considering its fit with their local context [60]. Clearly stated research questions can help practitioners to quickly determine a review’s focus and determine its relevance for their practice. We found that the Guides, with one exception, presented their research questions; however, the minority called out the PICO elements. Common in clinical research and a requirement for reviews following the PRISMA Guidelines [57], PICO elements provide a structured representation of the research question, which may facilitate a reader’s ability to quickly identify its relevance to their context. While we recognize that the ‘kaleidoscopic nature’ of HPE research [61] may give rise to a variety of types of research questions and the use of diverse methodologies, the majority of Guides were systematic reviews, a synthesis type for which PICO questions are quite appropriate. Recently, the BEME Collaboration has promoted the use of a variety of types of knowledge synthesis and provides resources to help researchers explore and adopt the knowledge synthesis methodology most appropriate for their research question [11].

Most Guides reported the details of how they executed literature searching and determined the inclusion and exclusion of studies. By reporting these details, authors provided transparency in their methods thus allowing readers to judge the strengths and weaknesses of the review. Including these details also allows future researchers to fully replicate and update the Guide, thereby avoiding inefficiencies and waste [62].

In fields such as public health and clinical medicine, practitioners report that the inability to access full-texts of information in a timely and cost-effective manner is a major barrier to uptake of evidence into practice [63]. Fortunately, the full-text of almost 75% of the included Guides was publicly accessible in some form. However, more recent reviews require subscription privileges or payment of $54.00 for 24-hour access. For many educational practitioners, this is a barrier that can potentially impede integrating current evidence into practice. We suggest review authors exercise their right, as described by Medical Teacher’s publisher, Taylor & Francis, to deposit a copy of their original manuscript (prior to peer review) to a preprint server [64]. Preprint servers provide practitioners with barrier-free, immediate access to Guide content. Additionally, preprint servers are indexed by search engines, such as Google Scholar, making them findable by a broad audience of educators. Lastly, recent research has reported that articles with an associated preprint actually have higher Altmetric scores and citation rates than those without [65].

The creation of targeted derivative products of the Guides, such as Practice Points and BEME Spotlights, aligns well with HPE practitioners’ requests for straightforward syntheses and brief presentations of empirical evidence [66]. In other disciplines, similar approaches have been linked with increased understanding of evidence and improvements in knowledge uptake and application to practice [67‐69]. Therefore, these materials should be made maximally visible and readily accessible. Going forward, Medical Teacher editors might consider embedding links to these materials on their website or directly referencing them within the review’s full-text. Additionally, the editors might follow Academic Medicine’s example and assemble these valuable resources as an e‑book [70].

In HPE, the transferability of educational approaches and innovations from one institution to another and between learners has been questioned [71]. To address this issue, researchers have suggested that theory-informed approaches enable practitioners to understand why things work and under what conditions, thus making it possible to transform ‘off the shelf’ approaches into approaches appropriate for their own educational contexts [72‐74]. Therefore, it is recommended that descriptions of educational interventions report on the principles, theories, and/or philosophies upon which those interventions are based. This idea is quite similar to the advice of Varpio et al., who noted, ‘While an educational innovation’s techniques may seem to be surface structures, they are realizations of deeper fundamental principles. The fundamental principles are themselves realizations of the innovation’s foundational philosophy. When techniques and/or principles are modified to a context, it is important to analyze if the modifications continue to uphold the innovation’s philosophy [74].’ Unfortunately, a minority of Guides reported these elements. We believe this is an area ripe for improvement, which could, in turn, facilitate greater transferability.

BEME guidelines state that a Guide should be updated within 3 years of publication [75]. Currently, few Guides meet this criterion suggesting some findings may no longer be relevant. BEME suggests that authors, in addition to publishing a formal update, post a supplement to the original review that highlights important studies published since the completion of the original review. We identified a single supplementary update [23]. It is the Guide author’s prerogative to update their work (or not); however, BEME reserves the right to permit other researchers to update a Guide. In the future, BEME might consider partnering authors with graduate programs in HPE to enlist students help with increasing the timeliness of Guide updates. Nevertheless, we should also acknowledge that the timing of necessary updates may vary, depending on how much new information is available and whether that new information makes a meaningful difference. Some topics may have no new studies accrue for many years, while others may need very frequent updates [76‐78].

We focused on a subset of HPE reviews that are highly curated and well-supported. Therefore, these findings may not generalize across HPE reviews. Future research should consider expanding our approach to examine HPE reviews more broadly. We framed our analysis of the Guides using the STORIES Statement. However, future researchers may want to more specifically focus on evaluating the overall quality of the Guides using another targeted tool, such as the AMSTAR Checklist.

Notwithstanding these limitations, we believe this meta-synthesis has a number of important strengths. To our knowledge, it is the first analysis of BEME Guides, which are created for use in educational practice. In addition, we undertook a systematic search and independently engaged in extracting data from all Guides. Lastly, we engaged in this research with a diverse team of stakeholders who have varied expertise and backgrounds. We hope this fact increases this meta-synthesis’ relevance to a variety of stakeholders. Ultimately, we are optimistic that this analysis will help those who write and publish future Guides, and similar publications, continue to evolve such that knowledge syntheses can more fully catalyze the translation of evidence into educational practice.