Many experts argue that addressing the burgeoning problems of the health care system requires a fundamental redesign, a transformation in which existing modalities are replaced by new paradigms for care delivery.1,2 As the Institute of Medicine has stated, simply “trying harder” to make the existing system work better will not be enough.3 However, while the need for fundamental redesign is generally agreed upon, less agreement exists for how those changes should be made and what that new health care system should look like. Expanding the use of information technology has emerged as one of the most widely supported aspects to health care system redesign.4–8

Health information technologies designed to improve clinical decision making are particularly attractive for their potential to address the growing information overload clinicians face and to provide a platform for integrating evidence-based medicine into care delivery.9,10 However, despite the theoretical and intuitive benefits of such technologies, the existing literature has demonstrated mixed empirical results.11,12 Additional work in understanding adoption of computerized decision support is critical. In this issue of the Journal of General Internal Medicine, studies by Agostini et al. and Graber and Mathew, respectively, examine decision support systems focused on treatment and diagnosis.13,14 Together, they illustrate many of the challenges involved with using and evaluating computerized decision support systems.

The study by Agostini et al. provides insight into the dynamics of using decision support systems for clinical care and illustrates how quality improvement approaches can be applied to adopting health information technology. The authors qualitatively evaluated physician perceptions of an electronic reminder related to prescribing sedative hypnotics for older hospitalized patients. The reminder was integrated into a health information system that allowed clinical orders to be entered through a computer. It was triggered when a physician attempted to order diphenhydramine or diazepam for insomnia in patients aged 65 or older. The authors used semistructured interviews to evaluate perceptions of the benefits and limitations of the reminder in a cohort of 36 housestaff physicians who ordered a sedative hypnotic despite receiving the reminder.

The results suggest that a complex set of factors underlie physician use of computerized reminders. Despite the decision to not follow the recommended action, physician respondents had both positive and negative perceptions of the reminder. These diverse perceptions were technology-specific (positive perception of integrating computers into clinical care), user interface-related (time needed to read reminder), professional (threats to physician autonomy), and health sciences-related (educational value/information content). These findings provide insight into the dynamics of adopting clinical decision support systems. Health information technologies are tools whose value is influenced by how clinicians modify their work practices to use them and by how organizational change is enacted when they are adopted. Social science theory and a growing body of empirical research suggest that successfully adopting computerized decision support depends not only on the technology but also on social, political, organizational, and practice-related factors.15–18 Elucidating these contextual factors may facilitate successful adoption, as well as better system design.

A second contribution of the study is its integration of quality improvement principles into decision support system adoption. In an earlier pre-post study, the authors showed that 1 year after the reminder was implemented orders for sedative hypnotics in older patient decreased by 3 absolute percentage points (18% vs 15%). In this study, the authors follow up on those positive findings by examining physicians who did not follow the recommendations of the reminder. Through this approach, the authors are laying the groundwork for future improvements in the computerized reminder. Such iterative approaches may facilitate successful use of health information technology systems by allowing for continued performance improvement. Though adoption of new technologies is often assumed to be a single event marked by a distinct before and after, it is in fact a multistage process that involves the routinization of the technology after it is implemented.19 During routinization, the technology becomes fully integrated into the work practices of an individual or into the culture of an organization. Ongoing, postimplementation evaluations of decision support systems such as the study by Agostini et al. may facilitate the type of deep integration needed to embed information technology into the social and technical fabric of healthcare.

Prior research has suggested that studies examining decision support systems may be biased when the system developers are also evaluating its performance.11 This may be a particular concern with the study by Agostini et al. due to the lack of respondent anonymity and social desirability response bias. Social desirability response bias may have enhanced the reported positive perceptions of the reminder. All respondents were housestaff (the majority interns) and the principal investigator, an attending physician at the institution, was involved with reminder development and conducted all interviews. Utilizing other methods to complement the semistructured interviews, such as anonymous, phone-based interviews or having a nonattending physician interviewer, may have reduced this source of bias. Another area where the study could have been improved was in assessing respondent satisfaction with the reminder. The authors used a 10-point scale to assess satisfaction. Interpreting these results is difficult because these types of scales often skew upwards, producing biased results.20 This limitation points out a major deficit in research related to health information technology: the absence of validated instruments. Developing such instruments should be an area of future research emphasis.

The study by Graber and Mathew examines a different type of decision support modality: a computer-based diagnosis aide. The authors evaluated a web-based system in which clinical findings can be entered either as narrative text or as key findings and a list of 30 potential diagnoses is produced. This study builds on the existing literature by taking an innovative technological approach towards diagnostic decision support. Decision support systems focused on improving physician diagnosis were among the earliest applications of information technology to healthcare.10 These systems grew out of the field of artificial intelligence and often used if then logic to suggest potential differential diagnoses (e.g., if chest pain is present then myocardial infarction is a potential diagnosis). These types of expert systems proved to be limited in nature because keeping an updated database of appropriate rules that could encompass a substantial range of diagnoses was difficult to maintain, and inputting data into the systems proved cumbersome. Rather than using such if-then rules-based approaches, Graber and Mathew used search technologies to suggest potential diagnoses. Given rapid advances in machine learning methods and in the scope of the Internet, such approaches may prove to be particularly valuable in the future.

While using search technology to support diagnostic decision-making may be a promising approach, the results of the study by Graber and Mathew should be interpreted with caution due to methodological issues and the early phase nature of the evaluation. An important methodological issue in this study was the lack of blinding. The investigators who entered data from the sample cases were aware of the correct gold-standard diagnosis. This may have biased the reported sensitivities, particularly when entering key findings—the data input method recommended by the authors. Given that the investigator abstracting the 3-6 key findings for entry into the system from the sample cases was aware of the results, he may have entered more relevant findings than he would have if blinded to the correct diagnosis. Because diagnoses are not known beforehand in real clinical situations, the performance of the system would likely vary considerably in other settings. A second important consideration was that empirical clinical data were not used to assess the performance of the system. Early phase studies for new diagnostic technologies often produce results with limited internal and external validity due to the type of data sources utilized to assess the performance of the new diagnostic modality.21 In this study the authors use the “Case Records of Massachusetts General Hospital” from the New England Journal of Medicine as their source of key findings and whole text. Computer-based information systems manage structured and unstructured data in very different ways. Due to the preinterpreted, standardized, and tightly edited format of the “Case Records,” this is a highly structured source of data. Real clinical data from patient interviews or medical records are far less structured and would likely alter the performance of the system considerably. The effects of presenting the system with more tightly structured data are suggested by the differences in sensitivity found when whole text entry of the “Case Records” was compared with entry of key clinical findings abstracted from the “Case Records.” While much of the paper focuses on sensitivity and its importance in diagnosis, the specificity of the system was not examined. For rare conditions, maximizing sensitivity is often important. But many diagnostic dilemmas occur in the context of high-prevalence diseases, including chronic disease conditions in which secondary and tertiary diagnoses are critical to management. In these situations, specificity is perhaps even more important than sensitivity and is central to effective diagnosis making. The study would have also benefited from a stronger theoretical basis for how it defined a positive result. A positive result was defined as the appearance of the correct diagnosis anywhere among the 30 potential diagnoses returned by the system. A more rigorous explanation justifying the 30-item list versus another length would have strengthened the study.

Together, the studies by Agostini et al. and Graber and Mathew highlight the challenges faced in both developing computerized decision support systems appropriate for clinical use and in integrating such systems inside the structure and culture of a healthcare organization. The ever-accelerating proliferation of biomedical knowledge will continue to both drive forward scientific progress and to tax the cognitive capacities of clinicians as they struggle with the weight of new research findings, clinical guidelines, and recommended best practices. Paper-based healthcare information management is clearly insufficient to support the large-scale changes needed to redesign the healthcare system so that it rests on a more rational, evidence-based foundation. In this regard, the question is not whether computerized decision support systems should or will be integrated into care delivery. Rather, the question of fundamental importance is how can these systems be best used to improve care. If health information technology is going to transform healthcare, a deeper understanding of the complex dynamics underlying system adoption and use is needed. For these reasons, future research needs to focus on the effectiveness of adopted systems as much as it does on their efficacy.