Evaluating informatics applications—clinical decision support systems literature review

https://doi.org/10.1016/S1386-5056(01)00183-6Get rights and content

Abstract

This paper reviews clinical decision support systems (CDSS) literature, with a focus on evaluation. The literature indicates a general consensus that clinical decision support systems are thought to have the potential to improve care. Evidence is more equivocal for guidelines and for systems to aid physicians with diagnosis. There also is general consensus that a variety of systems are little used despite demonstrated or potential benefits. In the evaluation literature, the main emphasis is on how clinical performance changes. Most studies use an experimental or randomized controlled clinical trials design (RCT) to assess system performance or to focus on changes in clinical performance that could affect patient care. Few studies involve field tests of a CDSS and almost none use a naturalistic design in routine clinical settings with real patients. In addition, there is little theoretical discussion, although papers are permeated by a rationalist perspective that excludes contextual issues related to how and why systems are used. The studies mostly concern physicians rather than other clinicians. Further, CDSS evaluation studies appear to be insulated from evaluations of other informatics applications. Consequently, there is a lack of information useful for understanding why CDSSs may or may not be effective, resulting in making less informed decisions about these technologies and, by extension, other medical informatics applications.

Introduction

Systems to aid in medical decision making were introduced over 25 years ago. Relatively few are in general use in clinical settings. Despite their potential usefulness, the lack of widespread clinical acceptance long has been of concern among researchers and medical informaticians [1], [2], [3].

This paper reviews literature that focuses on evaluation of clinical decision support systems (CDSS). The paper discusses the following key findings: The main emphasis is on changes in clinical performance that could affect patient care. Many evaluations of CDSSs use designs based on laboratory experiment or randomized controlled clinical trials (RCTs) to establish how well the systems or physicians perform under controlled conditions. Other approaches to evaluation, such as ethnographic field studies, simulation, usability testing, cognitive studies, record and playback techniques, and sociotechnical analyses rarely appear in this literature. As was the case over ten years ago, few systems have been evaluated using naturalistic designs to study actual routine CDSS use in clinical settings. Consequently, the CDSS evaluation literature focuses on performance or specific changes in clinical practice patterns under pre-defined conditions, but seems lacking in studies employing methodologies that could indicate reasons for why clinicians may or may not use CDSSs or change their practice behaviors. Further, there is little reference in the CDSS literature to a theoretical basis for understanding the many issues that arise in developing and implementing CDSSs. In addition, the studies concern physicians to the near exclusion of other clinicians or potential users. Lastly, the literature seems not to be informed by studies of other medical computer applications, such as hospital information systems (HISs), computer based patient records (CPRs), physician order entry (POE), or ancillary care systems. These studies could provide useful insights into issues that likely would be relevant to acceptance and use of CDSSs.

Section snippets

Literature review methods

An automated literature search was done using Medline with the assistance of a librarian. This search identified papers classified as about a: (1) decision support system; (2) clinical decision support system; (3) expert system; and (4) decision aid. ‘CDSS’ has a variety of definitions. Any system that was considered a CDSS by the authors and catalogers of the papers reviewed was considered so for purposes of this review. This decision was made, instead of using an a priori definition of CDSS,

Usefulness of CDSSS

The literature indicates a general consensus that clinical decision support systems are thought to have the potential to improve care, or at least to change physicians’ behavior [5]. Reminders [6], [7], [8], [9], [10]. alerts [11], [12], [13], [14], [15], [16], [17], treatment plans [6], and patient education [6], have been reported as effective in changing practice behaviors. Evidence of positive effect is more equivocal for guidelines [18], [19], [20], [21]. Some studies suggest that

Evaluations of CDSS

Appendix A profiles all the evaluation studies of CDSSs found in the literature search. There are 27 studies reported in 35 papers. Papers reporting related studies are counted as one study each, though they are listed separately. Two of the 35 papers [48] are substantially the same, and, therefore, listed as one entry in the table.

A review of the studies in Appendix A suggests several notable tendencies:

  • 1.

    As Appendix A shows, most studies are of specific changes in clinical performance that

Conclusions

Despite calls for alternatives, or recommendations to select designs congruent with system development stage and different evaluation questions [49], [50], [65], [67], RCTs remain the standard for evaluation approaches for CDSSs [85], [86], making evaluation traditions for CDSSs similar to those for other computer information systems, whether or not they may be intended for use in health care. Most commonly, systems, whether medical or not, have been evaluated according to selected outcomes

Acknowledgements

I am grateful to Dr Richard Spivack of the US National Institute of Standards and Technology for his invaluable assistance in the automated literature search.

References (88)

  • B. Kaplan

    The computer prescription: medical computing, public policy, and views of history

    Sci. Technol. Human Values

    (1995)
  • A.G. Randolph et al.

    Users’ guides to the medical literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system

    J. Am. Med. Assoc.

    (1999)
  • E.A. Balas, S.A. Boren, Clinical trials and information interventions. In: E.S. Berner (Ed.), Clinical decision support...
  • E.A. Balas et al.

    The clinical value of computerized information services

    Arch. Fam. Med.

    (1996)
  • D.A. Davis et al.

    Changing physician performance: A systematic review of the effect of continuing medical education strategies

    J. Am. Med. Assoc.

    (1995)
  • D.L. Hunt et al.

    Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review

    J. Am. Med. Assoc.

    (1998)
  • R.D. Zielstorff, M.J. Teich, M.D. Paterno, M. Segal, P-CAPE: A high-level tool for entering and processing clinical...
  • S. Shea et al.

    A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting

    J. Am. Med. Inform. Assoc.

    (1996)
  • D.W. Bates et al.

    Effect of computerized physician order entry and a team intervention on prevention of serious medication errors

    J. Am. Med. Assoc.

    (1998)
  • D.W. Bates et al.

    The impact of computerized physician order entry on medication error prevention

    J. Am. Med. Inform. Assoc.

    (1999)
  • A.K. Jha

    Identifying adverse drug events

    J. Am. Med. Inform. Assoc.

    (1998)
  • G.J. Kuperman et al.

    Improving response to critical laboratory results with automation: results of a randomized controlled trial

    J. Am. Med. Inform. Assoc.

    (1999)
  • M. Monane et al.

    Improving prescribing patterns for the elderly through an online drug utilization review intervention: a system linking the physician, pharmacist, and computer

    J. Am. Med. Assoc.

    (1998)
  • R.A. Raschke et al.

    A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital

    J. Am. Med. Assoc.

    (1998)
  • H.R. Warner, Jr., S. Miller, K. Jennings et al., Clinical event management using push technology—implementation and...
  • T.H. Lee et al.

    Translating good advice into better practice

    J. Am. Med. Assoc.

    (1997)
  • J.M. Overhage et al.

    Computer reminders to implement preventive care guidelines for hospitalized patients

    Arch. Intern. Med.

    (1996)
  • D.L. Schriger et al.

    Implementation of clinical guidelines using a computer charting system: effect on the initial care of health care workers exposed to body fluids

    J. Am. Med. Assoc.

    (1997)
  • R.G. Woolf et al.

    Potential benefits, limitations and harms of clinical guidelines

    BMJ

    (1999)
  • I.E. Buchan

    Local clinical guidelines: description and evaluation of a participative method for their development and implementation

    Fam. Practice

    (1996)
  • D.C. Classen

    Clinical decision support systems to improve clinical practice and quality of care

    J. Am. Med. Assoc.

    (1998)
  • D. Cook et al.

    The trials and tribulations of clinical practice guidelines

    J. Am. Med. Assoc.

    (1999)
  • D.K. Litzelman et al.

    Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols

    J. Gen. Intern. Med.

    (1993)
  • M.M. Morgan, J. Goodson, G.O. Barnett, Long-term changes in compliance with clinical guidelines through computer-based...
  • P.J. Greco et al.

    Changing physicians’ practices

    New Engl. J. Med.

    (1993)
  • C.L. McDonald

    Protocol-based computer reminders, the quality of care, and the non-perfectability of man

    New Engl. J. Med.

    (1975)
  • J. Bouaud, B. Séroussi, C.-C. Antoine et al., Hypertextual navigation operationalizaing generic clinical practice...
  • M.D. Cabana et al.

    Why don't physicians follow clinical practice guidelines?: a framework for improvement

    J. Am. Med. Assoc.

    (1999)
  • D.A. Davis et al.

    Translating guidelines into practice: A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines

    Can. Med. Assoc. J.

    (1997)
  • D.K. Litzelman et al.

    Physicians’ reasons for failing to comply with computerized preventive care guidelines

    J. Gen. Intern. Med.

    (1996)
  • C.P. Friedman et al.

    Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems

    J. Am. Med. Assoc.

    (1999)
  • J.P. Kassirer

    A report card on computer-assisted diagnosis—the grade: C

    New Engl. J. Med.

    (1994)
  • F.M. Wolf, C.P. Friedman, A.S. Elstein et al., Changes in diagnostic decision-making after a computerized decision...
  • V.L. Patel

    Cognition and technology in health education research

    Can. J. Public Health

    (1996)
  • Cited by (283)

    View all citing articles on Scopus
    View full text