Evaluation of the Marburg Heart Score and INTERCHEST score compared to current telephone triage for chest pain in out-of-hours primary care

This study was part of the TRiage of Acute Chest pain Evaluation in primary care (TRACE) project. A more elaborate description of the study methods was published previously [12]. In short, this study involved a retrospective, observational cohort of consecutive patients (≥ 18 years) with chest pain who contacted a large, regional primary care facility in Alkmaar, the Netherlands in 2017. This facility provides OOH-PC to approximately a quarter of a million inhabitants. Details on the organisation of Dutch OOH-PC can be found in the Electronic Supplementary Material (Supplement 1).

We gathered registered triage information, including patient and symptom characteristics, urgency levels and course of action following triage, using electronic health records at the OOH-PC facility. Final diagnoses were obtained from the patient file at the affiliated daytime primary care practices, with supporting hospitalisation and/or discharge letters related to the index consultation. Data were collected and processed using a secure, web-based electronic data capture platform (Castor EDC, Amsterdam, The Netherlands) [13].

Trained triage assistants handle triage using NTS triage protocols [2]. These protocols are complaint-specific and start with an initial assessment of patients’ vital signs using the ABCDE method [14]. The protocol for chest pain continues with seven questions regarding symptom characteristics (Electronic Supplementary Material, Supplement 1). The NTS algorithm uses the available answers to calculate an urgency level, linked to a recommended time until care. Triage assistants ultimately decide on the most fitting course of action.

Supplement 2 (Electronic Supplementary Material) gives an overview of the components of each clinical decision rule. The MHS is a point-based score with five elements, developed for ruling out coronary artery disease (CAD) among primary care patients presenting with chest pain [8]. Patients are assigned either 1 (when present) or 0 points (when absent) for each item and the MHS equals the sum, thus ranging from 0 to 5.

The INTERCHEST score is a similar point-based tool developed for the same purpose as the MHS [7]. It consists of six elements, of which five are assigned either 1 (when present) or 0 points (when absent), and one element (‘pain reproducible by palpation’) is given minus 1 point when present and 0 points when absent. Thus, the INTERCHEST score ranges from −1 to +5 points. In order to evaluate the INTERCHEST score as a triage tool, we replaced the physician’s first impression with that of the triage assistant.

We defined ‘major events’ as a composite of all-cause mortality and urgent conditions, including ACS, linked to the initial complaint of chest pain which required hospital admission and/or urgent in-hospital treatment. Supplement 3 (Electronic Supplementary Material) specifies the diagnoses that constitute major and non-major events. The occurrence of an ACS was based on physician reporting.

We evaluated the performance of the NTS triage software, the MHS and the INTERCHEST score by assessing their discriminative ability (i.e. the ability to distinguish between patients with and without a major event) using C‑statistics and visualised using an area under the receiver operating characteristics curve. We also generated calibration plots to illustrate the agreement between the predicted and observed major event rates. Perfect calibration implies that predictions have an intercept of 0 and a coefficient of 1 (i.e. a diagonal 45-degree slope) [15].

We divided NTS urgency into high (U1–U2; recommended care < 1 h) and low (U3–U5; care > 1 h) and determined the most optimal threshold for the MHS and INTERCHEST score for predicting a major event. When data elements of the MHS or INTERCHEST score were not reported, researchers blinded for clinical outcomes made maximum effort to retrieve the elements from available data. If the data elements were unavailable, they were presumed absent. Patients with an unknown NTS urgency level or without follow-up data were excluded. Hence we did not impute the dataset if triage or outcome data were missing. Furthermore, we performed more restricted analyses specifically for the occurrence of ACS.

Performance analyses of the NTS triage software were based on the urgency level allocation and the occurrence of a major event. Fig. 1 illustrates the discriminative properties with a C-statistic of 0.66 (95% confidence interval [CI]: 0.62–0.69). Calibration was adequate when comparing observed versus expected major events (Fig. 2). Diagnostic properties were calculated after dividing urgencies into high (U1–U2) and low (U3–U5) (Tab. 2), resulting in 886 (61.8%) recommended referrals (i.e. high urgency), of which 691 (48.2%) were unnecessary (i.e. false positive). In this case, 40 of 235 (17.0%) major events would have been missed (i.e. false negative). The sensitivity and specificity of the NTS were 83.0 and 42.3%, respectively.

The MHS produced a C-statistic of 0.74 (95% CI: 0.70–0.77) (Fig. 1) and calibration appeared adequate (Fig. 2). Diagnostic properties were calculated at different thresholds. A threshold of 2 points showed optimal diagnostic accuracy with a sensitivity of 82.1% and a specificity of 58.8% (Tab. 2). Compared to the NTS software, MHS ≥ 2 resulted in 199 fewer referrals (−22.5%) and only two additional missed major events.

The INTERCHEST score resulted in a C-statistic of 0.76 (95% CI: 0.73–0.80) (Fig. 1), and calibration appeared to be reasonably good (Fig. 2). An INTERCHEST score with a threshold of ≥ 2 points showed optimal diagnostic accuracy with a sensitivity and specificity of 81.7 and 61.6%, respectively. Compared to the NTS, the INTERCHEST score improved efficiency by leading to fewer patient referrals (−234) with a similar number of missed cases (n = 43 vs n = 40).

Overall, 98 (6.8%) patients suffered from an ACS. For the occurrence of an ACS, C‑statistics were 0.68 (95% CI: 0.63–0.73), 0.70 (0.65–0.75) and 0.77 (0.73–0.81) for the NTS, MHS and INTERCHEST score, respectively (Fig. 1). The NTS software showed a sensitivity of 85.7% and specificity of 39.9%, with 14 missed cases and an unnecessary referral rate of 56% (n = 802) (Tab. 3). The MHS did not outperform the NTS, with more missed cases (n = 7) and a sensitivity and specificity of 78.6 and 54.3%, respectively. The INTERCHEST score showed more favourable diagnostic capability, with a sensitivity and specificity of 88.8 and 57.6%, 234 fewer referrals and fewer missed ACS diagnoses (n = −3).

To our knowledge, this is the first study to evaluate and compare the test accuracy of an existing telephone triage tool with validated diagnostic prediction rules in OOH-PC for a broader set of chest-pain-related conditions. We assessed a relatively large study population, comprising all chest pain patients that contacted OOH-PC without further selection. While the organisational structures for European OOH-PC show several differences between and within countries, general practitioners (GPs) are generally positioned as gate-keepers to secondary care and often use integrated telephone triage systems [3]. Our findings may therefore be generalisable to similar settings across Europe.

The main limitation of our study is its retrospective nature. Verification bias may have been in play, since only patients in whom there was a high suspicion of a major event underwent further diagnostic work-up to confirm or rule out the occurrence of an event. However, we tried to minimise this bias by using a 6-week follow-up window for the outcome of interest, allowing us to capture initially missed events. Second, since the MHS and INTERCHEST score are not integrated into current practice, score elements were not registered structurally. We presumed the absence of a symptom or score element when it was not recorded by the triage assistant. Third, a mentionable amount of lost-to-follow-up was caused by GPs refusing to provide data on their patients (6.0% of baseline patients). These GPs expressed concerns due to the recent implementation of the European General Data Protection Regulation.

The MHS was developed by Bösner et al. [8] for predicting CAD in daytime primary care among patients with chest pain. Overall, best results were obtained when applying a threshold of 3 points, and repeated validation showed a discriminative ability ranging from 0.84 to 0.90, with good sensitivity (87–91%) and specificity (61–81%) [8, 9, 16]. Additional external validation of the MHS among patients in Southeast Asia resulted in a less favourable discriminative ability of 0.66 (95% CI: 0.62–0.70) [17]. The MHS was also tested for its ability to rule out ACS among a selected patient population referred for immediate cardiac evaluation because an ACS was suspected [18]. Results of this study showed that the MHS alone had insufficient diagnostic accuracy (75.0% sensitivity and 44.0% specificity), and incorporation of the MHS in GP consultations produced a specificity of only 23.3% despite an increased sensitivity (94.4%). In the present study, the MHS was evaluated as a stand-alone triage tool for distinguishing between patients with and without major events. While the rule was not constructed for this purpose, it still outperformed the NTS with C‑statistics of 0.74 versus 0.66 and increased specificity (+16.5%).

Similar to the MHS, the INTERCHEST score was designed for ruling out CAD in daytime primary care [7]. It was based on five cohort studies, including the MHS derivation study. Overall, studies found a sensitivity of 82–88% and specificity of 74–82% for ruling out CAD (threshold ≥ 2) [16]. In urgent primary care, the INTERCHEST score demonstrated good discriminative ability (C-statistic > 0.80) for predicting major adverse cardiovascular events [19]. The usability of the INTERCHEST score seems to be corroborated by the present study, which showed it to have the most favourable diagnostic capability for predicting both major events and ACS.

Our results illustrate how existing clinical decision rules may improve triage risk-stratification in OOH-PC settings. However, in the light of the retrospective nature of our study, we should first conduct further prospective validation, preferably at multiple sites and settings, before recommending implementation in routine care. We believe that such studies should not focus on ACS alone, but rather focus on a broader set of chest-pain-related outcomes (or major events) that are relevant in unselected patient populations.