Transferability of principles of evidence based medicine to improve educational quality: systematic review and case study of an online course in primary health care

Aims

As the developers of an online degree course for health professionals, we aimed to:

• Evaluate the use of an evidence based medicine framework in an educational development setting

• Develop robust quality standards for the delivery of an online postgraduate course in primary health care

• Draw general lessons about the transferability of the principles of evidence based medicine to educational practice.

Research team

We are a multidisciplinary, research oriented academic team comprising four general practitioners (one with a strong interest in information technology), a social scientist, a psychologist, and an educationalist; we work closely with an academic nursing unit. We had previously taught in undergraduate medicine, postgraduate short courses, and work based training, but we were new to the online environment.

Educational context

We established a part time MSc degree in primary health care at University College London in 1999. The course is entirely online except for an initial one week summer school. It caters for a diverse student group of general practitioners, public health physicians, community nurses, pharmacists, and managers drawn from the United Kingdom and mainland Europe, most of whom sign up to the course to achieve goals such as developing new services, establishing local research programmes, or developing and evaluating teaching and training initiatives.

When we embarked on this project in 1997, University College London had a policy of discouraging distance learning because of concerns about quality. Hence, we found ourselves a test case for wider issues concerning the credibility and feasibility of online learning at our institution.

Methods

The study comprised three overlapping phases: secondary research, primary research, and synthesis, as shown in figures 1 and 2.

Fig 1

Methods used in preparation of the quality framework

• Download figure
• Open in new tab
• Download powerpoint

Secondary research phase

We did a systematic review of the literature on online education. Following the sequence in box 1, we framed focused questions and tried to select research designs, search strategies and data sources appropriate to each. Although we initially constructed these questions in terms of how the course affected student performance, our final list of questions was as follows:

• What is a high quality online learning experience for postgraduate students of primary health care?

• How can we provide that experience consistently and efficiently?

• How can we reliably demonstrate the quality of our course to internal critics and external evaluators?

• How can we best support, train, and supervise our staff?

• How will we know when we are failing?

• How can we improve our performance year on year?

We applied a formal search strategy to online databases (notably ERIC (Educational Resources Information Centre) and PsycInfo). We also searched books, grey literature (especially dissertations and internal reports), and key journals by hand. We gained additional insights from attending conferences and courses (including online Open University courses), joining academic email lists, and making direct contact with experts in the field. Through these, we encountered many examples of existing online programmes, which we considered as case studies.

We examined literature on the development of audit and quality assurance programmes from industry and the service sector (for example, ISO 9000, Investors in People, Royal College of General Practitioners Quality Practice Award) and identified official guidelines on distance education produced by the Quality Assurance Agency in the United Kingdom and comparable publications from other countries.

We applied standard critical appraisal checklists to guidelines.9 For qualitative research papers (where difficulties in appraisal are well described10), we used a range of checklists 7 11^–13 to guide in-depth discussion of published studies and prompt contact with authors where necessary. These strategies and sources are described fully on bmj.com (appendix 1).

Fig 2

Methods used to synthesise the quality framework

• Download figure
• Open in new tab
• Download powerpoint

Results of systematic review

We found only one randomised controlled trial examining what works in online education in our subject. This was a small trial on the effect of online postgraduate programmes in primary health care.1 The full results of our search are available on bmj.com Of around 300 primary research articles and 700 reviews and editorials, we rejected around 95% as irrelevant or methodologically poor. Many original research papers had not been peer reviewed (some had been published exclusively on the internet), and most were limited to technical details or superficial case description. The studies of undergraduate medical education were the only ones whose sampling frames, interventions, and outcomes could be meaningfully compared in a summary table, and we have published a systematic review of these studies.14

Of the 15 guidelines for online education, around half were relevant and potentially transferable, but validity was hard to assess. The recommendations from the UK Quality Assurance Agency generally seemed sensible, but the evidence base was not clear and there was little advice on dissemination, implementation, or local adaptation (see appendix 2 on bmj.com for details). Several US guidelines seemed more robust and flexible, but most of these still took an institutional focus and the practical lessons for people developing courses were unclear.

Formulating a quality framework

Combining our diverse secondary and primary sources to produce a clear vision for quality, a succinct set of standards, and a set of measurable success criteria for our own course was difficult and complex. It required repeated discussion and revisiting of concepts. Our primary research often provided rich case examples that enabled us to make sense of (or challenge) the published recommendations. Critical incidents proved particularly useful as triggers for action. Examples of all these and the final version of the quality framework are given on bmj.com (appendices 3 and 4).

Despite the plethora of papers and guidelines on online education, we found no simple recipes for developing evidence based quality standards in our educational project. We repeatedly found that reflection on practical experience (rather than, say, the application of critical appraisal checklists) enabled us to test the validity and transferability of published evidence to our course.

Applicability of evidence based medicine

We believe there are four key differences between evidence based education and conventional evidence based medicine. Firstly, many questions relating to clinical practice fall into a simple and logical taxonomy (such as, prevalence, prognosis, or therapy). The different types of question have a corresponding preferred research design (survey, cohort study, randomised controlled trial, etc) with accepted criteria for assessing validity (the critical appraisal checklist13). Educational questions have a more complex taxonomy, a less direct link with particular preferred study designs, and no universally accepted criteria for assessing validity. 15 16

Secondly, most of the definitive research questions generated for this project were qualitative—that is, they began with exploratory, open ended stems such as how or what. In the early stages of the project, we constructed questions in the format used in evidence based medicine (population, intervention, comparison, and outcome)—for example, “What proportion of students will pass an exam if all the teaching is online compared with the proportion that would pass if taught conventionally?” Implicit in this question is a behaviourist model of learning, in which the students are viewed as a population sample; the online course as an intervention; conventional education as the comparison; and student performance as the outcome. The validity of such assumptions is highly questionable, especially when (as in many postgraduate and continuing professional development courses) the goals of the course are humanistic rather than behaviourist (professional development, motivation, support, confidence) and (as with most part time adult learners) there is wide variation between students in terms of background, personal goals, life commitments, learning styles, ongoing circumstances, and so on. 17 18

Thirdly, we found the online educational literature difficult to access and navigate. This is unlikely to be wholly due to our lack of technical familiarity with the databases, since the user interface for the ERIC and Psyclit databases is identical to that for Medline. Some key search terms (e-learning, computer mediated communication) have multiple synonyms, and others (quality, performance) have multiple meanings. Given the diverse nature of qualitative research, search filters intended to select out such studies are in reality neither sensitive nor specific.10

Fourthly, and perhaps most importantly, we found that educational development requires practical wisdom and not merely research evidence. Although the theoretical knowledge we gained about online learning from published guidelines often scored well on objective measures of quality, it served to confuse as much as inform us. But the practical knowledge that we gleaned from conferences, academic mailing lists, expert contacts, Open University courses, and our portfolio of case examples from education, health care, and industry was invaluable in converting evidence into action.

Conclusions

Hammersley has accused the evidence based medicine movement of “making false and dangerous promises” for the transferability of its methods to education. In particular, he claims, it does not address how research evidence should be combined with other kinds of evidence in making practical judgments in educational development.8 We agree that the educational community must take care not to climb uncritically on the evidence based medicine bandwagon in the politically fashionable drive towards a focused and scientific approach. We propose an alternative decision making sequence (box 2) that better reflects the reality of evidence based education.

In conclusion, the linear and formulaic link between evidence and practice implicit in evidence based medicine proved inadequate for the complexities of educational research. Conceptual models designed for multifaceted problems, which may be more appropriate, include cognitive restructuring theory,19 complexity (non-linearity) theory,20 activity theory (the relation between course developers, contexts, and tools),21 and the sharing of tacit knowledge in informal communities of practice.22