2014 APDS SPRING MEETINGAssessment of Resident Operative Performance Using a Real-Time Mobile Web System: Preparing for the Milestone Age
Introduction
Surgical residents must demonstrate operative proficiency over the course of residency training to be considered competent to operate independently. Conventional end-of-rotation performance (CERP) assessments, in which faculty members recall general impressions of resident operative performance, do not account for the different types of operations residents perform, and they lack objective technical standards. Governing bodies in surgical education have identified a need for more thorough performance evaluations during residency training.
The Next Accreditation System (NAS) from the Accreditation Council for Graduate Medical Education features the Milestones Project, an evaluation template that systematically assesses the 6 core competencies, including technical and nontechnical operative performance.1 The American Board of Surgery (ABS) mandates 6 operative skills evaluations per resident in the 2015 to 2016 academic year for certification eligibility. The ABS suggests the Operative Performance Rating Scale (OPRS) developed by Southern Illinois University as a model evaluation system.2 The OPRS assesses multiple parameters of resident operative performance and has undergone thorough analyses of reliability and validity.3, 4, 5, 6, 7, 8
As residents and surgical educators routinely use mobile technology and cloud-based computing, we transformed the OPRS into a mobile, Web-based survey system. We hypothesized that this system would improve quality and timeliness of resident operative performance evaluation, facilitate feedback communication, and ensure compliance with the ABS and NAS Milestones requirements.
Section snippets
Study Design
This study was performed in an urban academic general surgery residency training program at the David Geffen School of Medicine of the University of California, Los Angeles between March 2013 and February 2014. Faculty participants included 43 surgeons, representing 5 clinical divisions and 4 affiliated institutions. Faculty whose practice did not include direct observation of resident operative performance were excluded from this study. Resident subjects included 66 categorical, designated
Results
Over the 11-month study period, 18 of the 43 eligible faculty members, representing 4 of the 5 clinical divisions and all 4 affiliated institutions, completed 176 OPRS forms evaluating 37 of the 66 eligible residents (56%). The study period included portions of 2 academic years. Faculty participants included 9 junior (assistant professors) and 9 senior (professors and associate professors) members. The numbers of junior faculty submissions (mean = 5.2) were greater than those of seniors (mean =
Discussion
Our study demonstrates that mobile OPRS evaluations offer timely performance feedback to surgical residents. These mobile OPRS reports fulfill the NAS and ABS minimum requirements of individual operative performance measures. The frequency and quality of mobile evaluations result in improved feedback dialog among the educators and trainees. Using this innovative approach, 56% of our trainees received procedure-specific feedback at an average of 169 minutes after procedure completion.
Compared
Conclusion
Mobile technology represents a new state-of-the-art tool with considerable potential in surgical education. Our mobile OPRS interface satisfies the ABS and NAS General Surgery Milestones Project requirements, stimulates real-time feedback dialog, and documents resident skills acquisition with high quality and efficiency.
References (18)
- et al.
Refining the evaluation of operating room performance
J Surg Educ
(2009) - et al.
Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents
Surgery
(2005) - et al.
Evaluating resident operative performance: a qualitative analysis of expert opinions
Surgery
(2011) - et al.
Prognostic value of resident clinical performance ratings
J Am Coll Surg
(2004) Direct observation of residents: a model for an assessment system
Am J Med
(2014)- et al.
Milestones: direct observation may be the key to accelerated training
Am J Med
(2013) - et al.
Competency champions in the clinical competency committee: A successful strategy to implement milestone evaluations and competency coaching
J Surg Educ
(2014) - et al.
A template for reliable assessment of resident operative performance: assessment intervals, numbers of cases and raters
Surgery
(2012) - et al.
Surgical education in the Internet era
J Surg Res
(2009)