Clinical decision support software for diabetic foot risk stratification: development and formative evaluation

Diabetes is now the leading cause of lower extremity amputation in Australia [1], with approximately 85 % of lower extremity amputations in people with diabetes preceded by a diabetic foot ulcer [1‐5]. The lifetime risk of foot ulceration in people with diabetes is estimated to be between 15 and 25 % [6‐8]. Identifying people at risk of foot complications is a crucial step in prevention. Diabetic foot risk stratification predicts foot ulceration and has accordingly become a cornerstone of management [9].

Diabetic foot risk stratification identifies clinical features of individuals with diabetes that are predictive of the relative risk of foot ulceration in the future. A large number and type of clinical indicators including both systemic and peripheral signs and symptoms have been tested for their predictive value. Systemic features have included age, sex, weight, height, body-mass index, duration of diabetes, type of diabetes, HbA1C, fasting glucose, insulin regimes, history of myocardial infarct, hypertension, erythrocyte sedimentation rate, serum creatinine, kidney disease, eye disease, smoking and alcohol intake [10]. Peripheral features have included peripheral arterial disease, peripheral neuropathy, foot deformity, prior foot ulceration or amputation, abnormal plantar foot pressures, absent tendon reflexes, ankle-brachial index, transcutaneous oxygen tension, lower extremity bypass, intermittent claudication, tinea pedis, onychomycosis, lower leg oedema, dry or fissured skin [10]. Social factors such as level of education, occupation, socioeconomic status, religion, ethnicity and marital status have also been assessed [10‐16].

Numerous diabetic foot risk classification systems are described in the literature [7, 9, 17‐25]. There is strong evidence to justify risk stratification systems from large cross-sectional and prospective studies [22, 26]. The risk stratification systems have ranged from two to six risk classification groups. Monterio-Soares validated five international risk systems [7, 16, 19, 21, 22] and reported no significant difference between them and all had a high accuracy to detect people who would develop foot ulceration [27]. Most recently the international collaboration, prediction of diabetic foot ulcerations study (PODUS), of more than 16,000 people with diabetes worldwide meta-analysis reported, “the use of a 10-g monofilament or one absent pedal pulse will identify those at moderate or intermediate risk of foot ulceration, and a history of foot ulcers or lower-extremity amputation is sufficient to identify those at high risk” [28]. Notably foot deformity, ethnicity and eye disease were not included in the analysis, as they were not consistently defined in the included data sets.

As a result of expert input and review, in 2011 Australia’s National Health and Medical Research Council (NHMRC) produced National Evidence-Based Guideline on Prevention, Identification and Management of Foot Complications in Diabetes (Guidelines) [24]. This delivered a new national foot risk stratification system, consisting of three levels; low, intermediate and high risk of developing foot complications and the Guideline recommended ‘any trained professional may perform the risk assessment’ and urged the ‘urgent integration of decision support tools into medical software’ [24]. Research has shown that although the procedure for the assessment can be done by any trained professional, the final assessment of level of risk still proved problematic in the absence of additional decision support [29].

Clinical decision support systems in medicine have progressed from early systems that were never used in a clinical setting to systems that are now integrated into electronic health records across diverse clinical settings [30, 31]. These have been known as artificial intelligence, expert systems or clinical decision support systems (CDSS). CDSS are defined as, “any electronic system designed to aid directly in decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” [32]. They prompt clinicians through a protocol of pertinent, evidence-based clinical actions and decisions to enhance health-related decisions and actions. CDSS support healthcare professionals to work at a higher level than their standard scope of practice. This is required when there is a workforce shortage of relevant expertise [33], as occurs in Western Australia where a shortage of podiatrists in rural and remote areas is recognised [34, 35]. Successive systematic reviews have shown the beneficial effect of CDSS on clinical decision making to improve practitioner performance in diagnostic systems, reminder systems, chronic disease management processes of care, drug dosing or prescribing systems, improve rates of screening, and improve adherence to recommended care standards [32, 36‐41]. CDSS are increasingly considered to be one of the most effective instruments to improve guideline implementation [38, 42‐44]. Trials of CDSS including diabetic foot processes of care have all shown improvements in practitioner performance, rates of screening and adherence to guidelines [45‐51]. Most recently, Moja’s systematic review of new generation CDSS in electronic health records reported no reduction in patient mortality but concluded that they might moderately decrease morbidity [31]. Obstacles to wider use of CDSS include poor usability or integration into practitioner workflow, failure to integrate with primary care information technology, the attitudes of end-users, practitioner non-acceptance of computer recommendations, lack of clinician input into the development and their failure to fulfil a perceived clinical need [38, 52, 53].

Diabetic foot risk stratification lends itself to CDSS because it has a strong evidence base, is unambiguous, has explicit input and output criteria for each risk stratification level, each foot risk choice is binary, every possible permutation of foot risks can be decided by an algorithm and produces a correct result. The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool is a CDSS and has been validated and shown to be predictive of ulceration [22].

In this study we sought to act on the NHMRC Guideline to integrate CDSS into Australian electronic health records to ensure it fitted within healthcare professionals’ usual workflow [24]. The underpinning question related to whether the use of a CDSS could assist non-podiatrists in risk stratification. The importance of a usable and accurate foot risk assessment CDSS is underpinned by the shortage of podiatrists in rural and remote areas of Western Australia [34, 35].

This paper reports on the process of usability testing that was conducted to develop and evaluate an electronic diabetic foot risk stratification tool for its use in the treatment and management of diabetes in a largely Aboriginal population in Western Australia. The aims of the usability testing were to not only produce a system that was clinically correct, but one that would fit within and enhance the workflow of clinical practice of the health professionals concerned.

Diabetic foot risk stratification with CDSS can be integrated into an electronic health record with minimal impact on a podiatrists’ usual workflow. A usability design process with an early focus on end users, integrated design and early and continued user testing was important in recognising major usability issues. Participants’ qualitative responses confirmed the language is suitable, generated extensive feedback for improvement and revealed essential design flaws. Live testing and participants’ quantitative results confirmed the accuracy of the tool. The resultant risk tool is fast, accurate, compatible with the workflow of a diabetic foot assessment and free from false negative errors of risk stratification based on the 2011 NHMRC Guidelines [24].

The mixed methods approach used in this study has been advocated in the development and evaluation of CDSS [57‐63]. This approach was appropriate given the obstacles recognized in the literature that limit the use of CDSS. User-centered design processes can help improve usability and result in more likable computer applications [54, 64]. The naturalistic design of the live testing with real patients for field tests is suggested by Kaplan [65] and was the approach used to validate the Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool [22]. Measurement of accuracy by peer review and comparison to a standard guideline avoids circularity, and the completeness of results can be supported by triangulation of data from complementary methods comparing data from semi-structured interviews, participant observation, online survey responses, NHMRC Guidelines and clinical testing [57, 60, 62].

Other studies have shown that major usability flaws in CDSS can be recognised by all methods regardless of the expertise of the evaluator as they rely on observation and are easy to perform [53, 66, 67]. Kilsdonk demonstrated a user-centred CDSS design can overcome usability problems when replacing paper-based clinical guidelines into an online format with CDSS as we have done in this study [64]. The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool, used similar evidence-based guidelines to implement a diabetic foot risk stratification tool in a central web-based database opposed to integrated into an electronic health record as we have done [22]. Our risk tool aligns with Curran’s findings of the clinical reasoning and diagnostic procedures of novice podiatrists of visual cues, touch cues, questions for the patient, and then a diagnostic statement by the podiatrist [68].

The implications of non-podiatrists using the risk tool have been considered, as CDSS allow healthcare practitioners to work at a higher level of expertise. Studies report that non-podiatrists overreport the presence of foot deformities [29, 69] and are unable to palpate reliably pedal pulses [22, 29, 69]. This would result in a higher risk stratification and is safe for patients with diabetes as the NHMRC recommendation for the higher risk is for referral to a podiatrist for second stage assessment [24]. The recent PODUS study has shown the consistent reliability of the 10-g monofilament regardless of the expertise of the tester, the number of sites and the anatomical sites tested based on five different studies and 11,522 people from three different countries [28]. Finally, the combination of clinical tests for integrated foot risk score as we have used in this study is more sensitive than individual clinical tests for predicting foot ulceration [24, 70, 71].

The strengths of this study were collaboration with an experienced development team, early input from multiple users; the mixed methods approach, and development within a well-established patient information record system [55]. Limitations of this study are the small sample size, no assessment of intraobserver reliability, and that formal usability techniques such as talk aloud protocols or walkthroughs were not used. Furthermore, this study reports the formative evaluation of the risk tool only. Further summative evaluation on the complete system is warranted [62] and prospective observational studies to determine if the risk tool accurately predicts foot ulceration in a primary healthcare setting.

This study illustrates that the views and experiences of potential end users can be used effectively to develop and evaluate a diabetic foot risk tool with CDSS based on evidence-based guidelines integrated into an electronic health record. The risk tool integrates simple assessment readily available in a clinical setting, and should have minimal impact on experienced healthcare professionals’ usual workflow and reflects current Australian guidelines [24]. It also structures foot examination for those who are less proficient in diabetic foot assessment and will ensure that identified predictors of risk for foot ulcers are referred in a more timely way for definitive assessment and management.