Original ArticleA Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study
Introduction
As residency programs continue to evolve toward a competency-based curriculum, there is an increasing need for assessment of resident technical skills. Adequate assessment and feedback remain a cornerstone of psychomotor skills acquisition, particularly within neurosurgery where the consequence of adverse operative events is great.1 Visual rating scales remain convenient tools for generating organized formative assessments. Different rating scales for surgery have been developed, including the Objective Structured Assessment of Technical Skills (OSATS), which has been used previously in a neurosurgical context.2, 3, 4 A theoretical limitation of visual rating scales is the risk of rater subjectivity in skills assessment. Furthermore, little information exists on the ability of rating scales to capture subtler aspects of performance, including instrument force applied during a procedure. This last point is particularly important because consistent evidence from the neurosurgical simulation literature suggests that applied force differentiates levels of expertise.5, 6, 7, 8, 9, 10 In addition, a recent study found that excess force applied during live neurosurgical operations is associated with increased intraoperative bleeding.11
The objective of the project was to conduct a generalizability study to better understand the use of a visual rating scale of operative performance in neurosurgery and to compare it with computerized metrics generated during a virtual reality neurosurgical operative procedure. We hypothesize that both methods will measure the same underlying construct, namely, surgical performance.
Section snippets
Subjects
Medical student applicants to a single Canadian neurosurgery program in 2015 were recruited to participate in a trial involving a simulated brain tumor resection task.8 Sixteen of the 17 applicants participated, comprising over 70% of the national neurosurgical applicant pool for that study year.11 Data were collected at a single time point within the Neurosurgical Simulation and Artificial Intelligence Learning Centre in a controlled laboratory environment void of distracting noise. No
Results
Four staff neurosurgeons evaluated 16 medical students for a total of 64 observations. Table 1 includes a descriptive analysis, demonstrating use of the full range of the Likert scale. Demographic information is available in a previous publication8 and can be summarized as follows: 7 out of 16 participants (43%) previously used a simulator, the mean number of neurosurgery elective weeks was 11.2 ± 4.6 (range, 4–22), and the mean number of surgical skin closures was 10.9 ± 6.3 (range, 1–25).
Five
Discussion
Based on studies of technical performance in neurosurgery, we have recently introduced a conceptual framework to understand surgical expertise in neurosurgery.14 Although it is clear that many nontechnical factors, such as clinical decision-making, contribute to expertise, having a framework allows one to better structure research questions relating to the interaction of cognitive and motor domains and how these contribute to operative outcomes, particularly at a challenging juncture in the
Conclusions
The visual rating scale can reliably be administered by as few as two raters and seems to reflect operative performance as measured on the simulator. However, force exerted during the neurosurgical operation and the quantity of brain volume removed and blood loss were less well captured by the visual rating scale. To our knowledge, this is the first study to be able to concurrently compare participant's ratings with their computationally measured performance and operative complications. We
Acknowledgments
The authors thank Dr. Valérie Dory and Dr. Meredith Young for their input. The authors also thank all the medical students and raters who participated in this study and Robert DiRaddo, Group Leader, Simulation, Life Sciences Division, National Research Council of Canada at Boucherville, and his team, including Denis Laroche, Valérie Pazos, Nusrat Choudhury, and Linda Pecora, for their support in the development of the scenarios used in these studies. The authors also thank all the members of
References (19)
- et al.
Neurosurgical skills assessment: measuring technical proficiency in neurosurgery residents through intraoperative video evaluations
World Neurosurg
(2016) - et al.
Bimanual psychomotor performance in neurosurgical resident applicants assessed using NeuroTouch, a virtual reality simulator
J Surg Educ
(2016) - et al.
Proficiency performance benchmarks for removal of simulated brain tumors using a virtual reality simulator NeuroTouch
J Surg Educ
(2015) - et al.
Improving performance by improving education
- et al.
Neurosurgeon as educator: a review of principles of adult education and assessment applied to neurosurgery
J Neurosurg
(2017) - et al.
Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery
J Neurosurg Pediatr
(2017) - et al.
Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery
J Neurosurg Pediatr
(2015) - et al.
Impact of acute stress on psychomotor bimanual performance during a simulated tumor resection task
J Neurosurg
(2016) - et al.
Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator
Oper Neurosurg
(2015)
Cited by (0)
Conflict of interest statement: This work was supported by the Di Giovanni Foundation, Montreal English School Board, Colannini Foundation, and Montreal Neurological Institute and Hospital. A. Winkler-Schwartz is supported by a doctoral training grant for applicants with a professional degree issued by the Fonds de recherche du Québec – Santé. R. Del Maestro is the William Feindel Emeritus Professor in Neuro-Oncology at McGill University.