Perspective

Sensitivity and Specificity should be De-emphasized in Diagnostic Accuracy Studies☆

This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of “Radiomic ANalysis and DAta Modeling”), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies.

This study outlines the development of a new method (split component synthesis; SCS) for meta-analysis of diagnostic accuracy studies and assesses its performance against the commonly used bivariate random effects model.

The SCS method summarizes the study-specific diagnostic odds ratio (on the ln(DOR) scale), which mainly reflects test discrimination rather than threshold effects, and then splits the summary ln(DOR) into its component parts, logit sensitivity (Se) and logit specificity (Sp). Performance of SCS estimator was assessed through simulation and compared against the bivariate random effects model estimator in terms of bias, mean squared error (MSE), and coverage probability across varying degrees of between-studies heterogeneity.

The SCS estimator for the DOR, Se, and Sp was less biased and had smaller MSE than the bivariate model estimator. Despite the wider width of the 95% confidence intervals under the bivariate model, the latter had a poorer coverage probability than that under the SCS method.

The SCS estimator outperforms the bivariate model estimator and thus represents an improvement in the approach to diagnostic meta-analyses. The SCS method is available to researchers through the diagma module in Stata and the SCSmeta function in R.

Diagnostic imaging is essential for diagnosis and management of many common problems in veterinary medicine, but imaging is not 100% accurate and does not always benefit the animal in the way intended. When assessing the need for imaging, the probability that the animal has a morphological lesion, the accuracy of the imaging and the likelihood of a beneficial impact on the animal must all be considered. Few imaging tests are sufficiently accurate that they enable a diagnosis to be ruled in or out; instead, the results of imaging only modify the probability of a diagnosis. Potential problems with excessive use of imaging include false positive diagnoses, detection of incidental findings and over-diagnosis, all of which may contribute to a negative benefit to the animal. Veterinary clinicians must be selective in their use of imaging, use existing clinical information when interpreting images and sensibly apply the results of imaging in the context of the needs of individual animals. There is a need for more clinical research to assess the impact of diagnostic imaging for animals with common conditions to help clinicians make decisions conducive to optimal patient care.

La elaboración sistemática de la pregunta de investigación permite al investigador enfocar el problema, orientar los métodos y postular posibles soluciones. En la práctica, todavía se observan dificultades en la formulación del problema de investigación. La diversidad de los escenarios clínicos, de donde resulta buena parte de las preguntas de investigación, hace que su formulación no se ajuste siempre a la estrategia PICO. El objetivo de este artículo es aportar una guía que facilite la formulación de las preguntas de investigación que surgen en la práctica clínica a médicos, especialistas y personal en entrenamiento.

The systematic formulation of the research question allows the researcher to focus the study, guide the method decisions, and to put forward possible solutions. In practice, there are difficulties in the formulation of research problems. Diversity of clinical scenarios can lead to a mismatch between the structure of the research question and the classical PICO (population, intervention, control, and outcomes) format. The aim of this article is to provide guidelines that help in the proper formulation of clinical practice research questions for general practitioners, specialists, and healthcare personnel in training.

In the prehospital setting, differentiating patients who have sepsis from those who have infection but no organ dysfunction is important to initiate sepsis treatments appropriately. We aimed to identify which published screening strategies for paramedics to use in identifying patients with sepsis provide the most certainty for prehospital diagnosis.

We identified published strategies for screening by paramedics through a literature search. We then conducted a validation study in Alberta, Canada, from April 2015 to March 2016. For adult patients (≥ 18 yr) who were transferred by ambulance, we linked records to an administrative database and then restricted the search to patients with infection diagnosed in the emergency department. For each patient, the classification from each strategy was determined and compared with the diagnosis recorded in the emergency department. For all strategies that generated numeric scores, we constructed diagnostic prediction models to estimate the probability of sepsis being diagnosed in the emergency department.

We identified 21 unique prehospital screening strategies, 14 of which had numeric scores. We linked a total of 131 745 eligible patients to hospital databases. No single strategy had both high sensitivity (overall range 0.02–0.85) and high specificity (overall range 0.38–0.99) for classifying sepsis. However, the Critical Illness Prediction (CIP) score, the National Early Warning Score (NEWS) and the Quick Sepsis-Related Organ Failure Assessment (qSOFA) score predicted a low to high probability of a sepsis diagnosis at different scores. The qSOFA identified patients with a 7% (lowest score) to 87% (highest score) probability of sepsis diagnosis.

The CIP, NEWS and qSOFA scores are tools with good predictive ability for sepsis diagnosis in the prehospital setting. The qSOFA score is simple to calculate and may be useful to paramedics in screening patients with possible sepsis.