Evaluating informatics applications—clinical decision support systems literature review
Introduction
Systems to aid in medical decision making were introduced over 25 years ago. Relatively few are in general use in clinical settings. Despite their potential usefulness, the lack of widespread clinical acceptance long has been of concern among researchers and medical informaticians [1], [2], [3].
This paper reviews literature that focuses on evaluation of clinical decision support systems (CDSS). The paper discusses the following key findings: The main emphasis is on changes in clinical performance that could affect patient care. Many evaluations of CDSSs use designs based on laboratory experiment or randomized controlled clinical trials (RCTs) to establish how well the systems or physicians perform under controlled conditions. Other approaches to evaluation, such as ethnographic field studies, simulation, usability testing, cognitive studies, record and playback techniques, and sociotechnical analyses rarely appear in this literature. As was the case over ten years ago, few systems have been evaluated using naturalistic designs to study actual routine CDSS use in clinical settings. Consequently, the CDSS evaluation literature focuses on performance or specific changes in clinical practice patterns under pre-defined conditions, but seems lacking in studies employing methodologies that could indicate reasons for why clinicians may or may not use CDSSs or change their practice behaviors. Further, there is little reference in the CDSS literature to a theoretical basis for understanding the many issues that arise in developing and implementing CDSSs. In addition, the studies concern physicians to the near exclusion of other clinicians or potential users. Lastly, the literature seems not to be informed by studies of other medical computer applications, such as hospital information systems (HISs), computer based patient records (CPRs), physician order entry (POE), or ancillary care systems. These studies could provide useful insights into issues that likely would be relevant to acceptance and use of CDSSs.
Section snippets
Literature review methods
An automated literature search was done using Medline with the assistance of a librarian. This search identified papers classified as about a: (1) decision support system; (2) clinical decision support system; (3) expert system; and (4) decision aid. ‘CDSS’ has a variety of definitions. Any system that was considered a CDSS by the authors and catalogers of the papers reviewed was considered so for purposes of this review. This decision was made, instead of using an a priori definition of CDSS,
Usefulness of CDSSS
The literature indicates a general consensus that clinical decision support systems are thought to have the potential to improve care, or at least to change physicians’ behavior [5]. Reminders [6], [7], [8], [9], [10]. alerts [11], [12], [13], [14], [15], [16], [17], treatment plans [6], and patient education [6], have been reported as effective in changing practice behaviors. Evidence of positive effect is more equivocal for guidelines [18], [19], [20], [21]. Some studies suggest that
Evaluations of CDSS
Appendix A profiles all the evaluation studies of CDSSs found in the literature search. There are 27 studies reported in 35 papers. Papers reporting related studies are counted as one study each, though they are listed separately. Two of the 35 papers [48] are substantially the same, and, therefore, listed as one entry in the table.
A review of the studies in Appendix A suggests several notable tendencies:
- 1.
As Appendix A shows, most studies are of specific changes in clinical performance that
Conclusions
Despite calls for alternatives, or recommendations to select designs congruent with system development stage and different evaluation questions [49], [50], [65], [67], RCTs remain the standard for evaluation approaches for CDSSs [85], [86], making evaluation traditions for CDSSs similar to those for other computer information systems, whether or not they may be intended for use in health care. Most commonly, systems, whether medical or not, have been evaluated according to selected outcomes
Acknowledgements
I am grateful to Dr Richard Spivack of the US National Institute of Standards and Technology for his invaluable assistance in the automated literature search.
References (88)
- et al.
Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations
Lancet
(1993) - et al.
Computerized decision support based on a clinical practice guideline improves compliance with care standards
Am. J. Med.
(1997) - et al.
A diagnostic support system in general practice: is it feasible?
Int. J. Med. Inform.
(1997) - et al.
Electronic communication and collaboration in a health care practice
Artif. Intell. Med.
(1998) - et al.
Methodology for evaluation of knowledge-based systems in medicine
Artif. Intell. Med. A
(1994) - et al.
The VATAM guidelines
Int. J. Med. Inform.
(1999) Evaluating medical expert systems
Soc. Sci. Med.
(1987)Decision support systems in diabetes: a systems perspective
Comput. Methods Programs Biomed.
(1998)- J.G. Anderson, S.J. (Eds.), Use and impact of computers in clinical medicine. Springer; New York,...
The medical computing ‘lag’: perceptions of barriers to the application of computers to medicine
Int. J. Technol. Assess. Health Care
(1987)
The computer prescription: medical computing, public policy, and views of history
Sci. Technol. Human Values
Users’ guides to the medical literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system
J. Am. Med. Assoc.
The clinical value of computerized information services
Arch. Fam. Med.
Changing physician performance: A systematic review of the effect of continuing medical education strategies
J. Am. Med. Assoc.
Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review
J. Am. Med. Assoc.
A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting
J. Am. Med. Inform. Assoc.
Effect of computerized physician order entry and a team intervention on prevention of serious medication errors
J. Am. Med. Assoc.
The impact of computerized physician order entry on medication error prevention
J. Am. Med. Inform. Assoc.
Identifying adverse drug events
J. Am. Med. Inform. Assoc.
Improving response to critical laboratory results with automation: results of a randomized controlled trial
J. Am. Med. Inform. Assoc.
Improving prescribing patterns for the elderly through an online drug utilization review intervention: a system linking the physician, pharmacist, and computer
J. Am. Med. Assoc.
A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital
J. Am. Med. Assoc.
Translating good advice into better practice
J. Am. Med. Assoc.
Computer reminders to implement preventive care guidelines for hospitalized patients
Arch. Intern. Med.
Implementation of clinical guidelines using a computer charting system: effect on the initial care of health care workers exposed to body fluids
J. Am. Med. Assoc.
Potential benefits, limitations and harms of clinical guidelines
BMJ
Local clinical guidelines: description and evaluation of a participative method for their development and implementation
Fam. Practice
Clinical decision support systems to improve clinical practice and quality of care
J. Am. Med. Assoc.
The trials and tribulations of clinical practice guidelines
J. Am. Med. Assoc.
Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols
J. Gen. Intern. Med.
Changing physicians’ practices
New Engl. J. Med.
Protocol-based computer reminders, the quality of care, and the non-perfectability of man
New Engl. J. Med.
Why don't physicians follow clinical practice guidelines?: a framework for improvement
J. Am. Med. Assoc.
Translating guidelines into practice: A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines
Can. Med. Assoc. J.
Physicians’ reasons for failing to comply with computerized preventive care guidelines
J. Gen. Intern. Med.
Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems
J. Am. Med. Assoc.
A report card on computer-assisted diagnosis—the grade: C
New Engl. J. Med.
Cognition and technology in health education research
Can. J. Public Health
Cited by (283)
Clinical decision support system in emergency telephone triage: A scoping review of technical design, implementation and evaluation
2024, International Journal of Medical InformaticsClinical Decision Support Tools in the Electronic Medical Record
2024, Kidney International ReportsRecommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics: Second Revision
2023, International Journal of Medical InformaticsREVISITING HEALTH INFORMATION TECHNOLOGY ETHICAL, LEGAL, and SOCIAL ISSUES and EVALUATION: TELEHEALTH/TELEMEDICINE and COVID-19
2020, International Journal of Medical InformaticsA new approach to integrating patient-generated data with expert knowledge for personalized goal setting: A pilot study
2020, International Journal of Medical Informatics