Reliable and Valid Tools for Measuring Surgeons' Teaching Performance: Residents' vs. Self Evaluation

doi:10.1016/j.jsurg.2012.04.003

Journal of Surgical Education

Volume 69, Issue 4, July–August 2012, Pages 511-520

https://doi.org/10.1016/j.jsurg.2012.04.003 Get rights and content

Background

In surgical education, there is a need for educational performance evaluation tools that yield reliable and valid data. This paper describes the development and validation of robust evaluation tools that provide surgeons with insight into their clinical teaching performance. We investigated (1) the reliability and validity of 2 tools for evaluating the teaching performance of attending surgeons in residency training programs, and (2) whether surgeons' self evaluation correlated with the residents' evaluation of those surgeons.

Materials and Methods

We surveyed 343 surgeons and 320 residents as part of a multicenter prospective cohort study of faculty teaching performance in residency training programs. The reliability and validity of the SETQ (System for Evaluation Teaching Qualities) tools were studied using standard psychometric techniques. We then estimated the correlations between residents' and surgeons' evaluations.

Results

The response rate was 87% among surgeons and 84% among residents, yielding 2625 residents' evaluations and 302 self evaluations. The SETQ tools yielded reliable and valid data on 5 domains of surgical teaching performance, namely, learning climate, professional attitude towards residents, communication of goals, evaluation of residents, and feedback. The correlations between surgeons' self and residents' evaluations were low, with coefficients ranging from 0.03 for evaluation of residents to 0.18 for communication of goals.

Conclusions

The SETQ tools for the evaluation of surgeons' teaching performance appear to yield reliable and valid data. The lack of strong correlations between surgeons' self and residents' evaluations suggest the need for using external feedback sources in informed self evaluation of surgeons.

Introduction

To keep up with advances in technology and the ever-growing body of knowledge, the dynamic field of surgery must adapt their surgical education to maintain professional performance. While surgical research has a robust history, educational research in surgery is relatively new.¹ To improve education in surgery, high quality educational research is needed.1, 2 In this paper, we aim to describe the development and validation of robust tools that provide surgeons with insight into their teaching performance.

To maintain or enhance professional performance, surgeons need accurate performance feedback.3, 4 However, a review study showed that there is often a lack of accurate performance feedback and, consequently, the ability of physicians to self-evaluate and improve their clinical performance is generally poor.⁵ In addition, it has been stated that humans are by nature poor in performing accurate self evaluation and that personal unguided reflections on practice simply do not provide the information that is sufficient to guide performance improvements adequately.⁶ Therefore, it has been suggested that external feedback data sources could inform and subsequently improve surgeons' self evaluation.3, 7, 8 Tools that facilitate the flow of feedback to inform self evaluation of teaching performance could help surgeons to evaluate their teaching performance more accurately.3, 7, 9 There are several tools that could evaluate surgeons' teaching performance but, overall, they lack validity and are not specifically developed and validated for the use in surgical specialties.¹⁰ The development and validation of tools specifically for surgical education is essential to generate credible and applicable results.

For that purpose, we aim to test the reliability and validity of the System for Evaluation of Teaching Qualities (SETQ) constructed for surgical specialties (see Fig. 1). Beside the evaluation of surgeons' general teaching performance, this system aims to evaluate surgery-specific teaching performance, such as teaching technical skills and teaching in the operation theater. Similar systems were developed and validated for some nonsurgical specialties.11, 12, 13 This study aims to evaluate (1) if the SETQ tools for surgical specialties are valid and reliable, and (2) if surgeons' and residents' evaluations of surgeons' teaching performance match.

Section snippets

Development of the SETQ for Surgical Specialties

To fulfill the demand from several residency programs, the SETQ was developed as a dynamic system for continuous evaluation and development of surgeons involved in teaching residents. After it was piloted in 1 department,¹² it was successfully implemented institution-wide and later nation-wide. Currently, SETQ is used in over 160 residency training programs (of various specialties) in 34 teaching hospitals, involving around 2300 clinical teachers and 2200 residents in The Netherlands. For a

Results

In this study, 302 (87% response rate) surgeons and 269 (84% response rate) residents participated (Table 1). Surgeons' characteristics are described in Table 1. Responding residents were equally divided over the residency training years (first year: 17.8%, second: 15.3%, third: 15.3%, fourth: 16.5%, fifth: 21.1%, higher than fifth: 14%). About 44% of the residents were female.

For both SETQ tools, the principal components analysis revealed a 5-scale structure of teaching performance. In line

Main Findings

This study was set out to study the reliability and validity of the SETQ tools for surgical specialties and to study surgeons' and residents' evaluation of surgeons' teaching performance. The results of this study show that the tools underlying the SETQ for surgical specialties provide reliable and valid results and meet current standards.10, 20 Furthermore, the correlation between surgeons' self evaluation and residents' evaluation is low.

Explanation of Results

First, the extremely high response rate of both

Conclusions

The SETQ tools for surgical specialties appear to provide reliable and valid evaluation data for use in surgical departments of Dutch training hospitals. We recommend seeing development and validation as a continuous exercise across different settings. The lack of strong correlations between surgeons' self evaluations and residents' evaluations suggests the need for using external feedback sources in informed self evaluation of surgeons.

Acknowledgments

This study is part of the research project Quality of Clinical Teachers and Residency Training Programs, which is co-financed by the Dutch Ministry of Health, the Academic Medical Center, Amsterdam, and the Faculty of Health and Life Sciences of the University of Maastricht. BCMB, MJMHL, and ORCB are employed by the Academic Medical Center, Amsterdam. OAA is a recipient of a Veni grant (916.96.059) from The Netherlands Organization for Scientific Research (NWO). Funders had no role in study

References (26)

K. Ahmed et al.
Development of a surgical educational research program-fundamental principles and challenges
J Surg Res
(2011)
M.W. von Websky et al.
Trainee satisfaction in surgery residency programs: modern management tools ensure trainee motivation and success
Surgery
(2011 October 5)
J. Sargeant et al.
The processes and dimensions of informed self-assessment: A conceptual model
Acad Med
(2010)
E.A. Locke et al.
Building a practically useful theory of goal setting and task motivationA 35-year odyssey
Am Psychol
(2002)
D.A. Davis et al.
Accuracy of physician self-assessment compared with observed measures of competence: A systematic review
JAMA
(2006)
K.W. Eva et al.
“I'll never play professional football” and other fallacies of self-assessment
J Contin Educ Health Prof
(2008)
J. Lockyer et al.
Feedback data sources that inform physician self-assessment
Med Teach
(2011)
J. Sargeant et al.
Features of assessment learners use to make informed self-assessments of clinical performance
Med Educ
(2011)
J. Sargeant et al.
“Directed” self-assessment: practice and feedback within a social context
J Contin Educ Health Prof
(2008)
T.J. Beckman et al.
How reliable are assessments of clinical teaching?A review of the published instruments
J Gen Intern Med
(2004)

R. van der Leeuw et al.

Systematic evaluation of the teaching qualities of obstetrics and gynecology faculty: Reliability and validity of the SETQ tools

PLoS ONE

(2011)

K.M. Lombarts et al.

Development of a system for the evaluation of the teaching qualities of anesthesiology faculty

Anesthesiology

(2009)

O.A. Arah et al.

New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey

PLoS ONE

(2011)

Cited by (42)

Standardization and Implementation of a Surgical Coaching Model for Pediatric Urology
2024, Journal of Surgical Education
To bridge gaps in proficiency and encourage life-long learning following training, coaching models have been utilized in multiple surgical fields; however, not within pediatric urology. In this review of our methodology, we describe the development of a coaching model at a single institution. In our initial experience, the perceived most beneficial aspect of the program was the goal setting process with logistics around debriefs being the most challenging. With our proposed coaching study, we aim to develop a model based upon prior coaching frameworks,¹^,² that is feasible and universally adaptable to allow for further advancement of surgical coaching, particularly within the field of pediatric urology.
Development of a Competency Framework Defining Effective Surgical Educators
2024, Journal of Surgical Education
The objective of this study is to develop a practical framework of competencies and behaviors which define an effective surgical educator.
A modified Delphi approach was used. A literature review and series of discussions with surgical education experts led to creation of a survey instrument which was sent to surgical faculty and trainees from a single academic institution. The results from this initial survey informed the creation of the subsequent survey instrument which was also sent to surgical faculty and trainees. Focus groups with surgical faculty and residents were conducted separately, transcribed, deidentified, and then evaluated for recurring themes. A competency framework was developed.
The surveys were administered and focus groups were conducted at the University Hospitals Cleveland Medical Center, a tertiary care academic institution.
Residents, fellows, and faculty surgeons from the fields of general surgery, plastic surgery, vascular surgery, orthopedic surgery, otolaryngology, neurosurgery, and urology.
There were 115 responses (31.3%) from 367 faculty surgeons, residents, and fellows invited to complete the initial survey examining 50 competencies. Eighteen competencies received a mean Likert score of at least 4 by both faculty and residents and were included in the subsequent survey instrument which was completed by 72 participants (19.6%). Focus groups were held separately with 6 faculty surgeons and 6 residents. Analysis of the survey results and focus group discussions identified several themes which informed the development of a competency framework consisting of 5 overarching competencies as well as 16 specific behaviors.
A practical framework was developed consisting of 5 competencies and 16 behaviors which define an effective surgical educator. The 5 competencies are: 1) fosters psychological safety, 2) displays exemplary medical knowledge and patient care, 3) diagnoses the learner and adjusts teaching, 4) communicates thought process to trainee, and 5) displays learner-centeredness. Based on the competency framework, residency leadership may specifically tailor faculty development initiatives to improve surgical education programming.
Emotional Intelligence and Burnout Related to Resident-Assessed Faculty Teaching Scores
2021, Journal of Surgical Education
Citation Excerpt :
This instrument is based on the Stanford criteria and has been largely adopted in the Netherlands. Boerbach et al. previously evaluated the SETQ as a faculty self-assessment along with resident assessment of faculty teaching ability in surgeons.24 Surgery faculty self-evaluations correlated strongly with the residents’ SETQ scores.
Emotional intelligence (EI) is associated with job success in multiple fields, in part, because EI may mitigate stress and burnout. Research suggests these relationships may include teaching. Our purpose is to further explore the relationships between EI, burnout, and teaching for faculty surgeons.
With IRB approval, surgical faculty were offered the opportunity to complete personal demographics, the Maslach Burnout Inventory, the SETQ-SMART assessment of teaching ability, and the SEF:MED self-assessment of emotional intelligence. Surgical residents rated faculty teaching ability using the SETQ-SMART
A medium-sized academic medical center in the Southeast approved to graduate 6 residents per year.
ACGME surgical faculty and general surgical residents PGY1 to PGY5 including preliminary residents, were given the opportunity to participate.
Faculty self-assessed teaching scores were significantly different from resident scores for nine (60%) faculty; three (33%) overrated their and 6 (67%) under rated their overall teaching ability, relative to resident ratings. The 3 SEF:MED scales correlated low-moderate to strongly with the SETQ-OTS: IS (r = 0.41, p = 0.13), EM (r = 0.67, p < 0.01), and EA (r = 0.43, p = 0.11). Overall, 8(53%) faculty scored moderate to high on at least 1 of the 3 MBI subscales.
Overall self-rated faculty teaching scores correlated negatively with higher EE and DP and positively with PA (r = -0.08, -0.21, and 0.52, p = 0.047; respectively). EI negatively correlated with MBI-EE and DP and positively with PA (r = -0.31, -0.18, 0.45, respectively), though due to the small sample none reach statistical significance with alpha set to 0.05.
In this pilot study, EI is positively correlated to surgical faculty members’ teaching ability. Burnout was less strongly correlated with resident-assessed faculty teaching scores, but with similar trends. Finally, EI was correlated with MBI EE, DP, and PA as expected given the literature in other fields. Expanded study is warranted.
The Role of Preoperative Briefing and Postoperative Debriefing in Surgical Education
2021, Journal of Surgical Education
Citation Excerpt :
The System for Evaluation of Teaching Qualities (SETQ) instrument, a reliable and validated survey tool, was distributed in-person at separate resident and faculty meetings. Previously used for evaluating surgical education, the survey assesses 5 domains of teaching: learning climate, professional attitude toward residents, communication of goals, evaluation of residents, and feedback.8 The SETQ tool has been used by learners to evaluate the quality of medical teaching they have received, and we used this tool here to have learners assess, as well as attendings self-assess, the teaching environment within the Department.
To study the impact of a new preoperative briefing and postoperative debriefing tool on the perceived quality of surgical education and to assess attitudes of residents and attendings regarding this tool.
Surrounding introduction and use of the tool (JHFIRE: Joint Huddles for Improving Resident Education), perceived quality of surgical education was assessed with pre- and postintervention System for Evaluation of Teaching Qualities (SETQ) surveys. Additionally, a postintervention Likert survey regarding the JHFIRE tool itself was completed by residents and faculty.
Johns Hopkins University Department of Otolaryngology-Head and Neck Surgery, a tertiary care academic institution.
All residents and attendings who used the tool were invited to participate. 40 participants (13 residents, 27 attendings) completed the preintervention SETQ. 11 participants (3 residents, 7 attendings, 1 unspecified) completed the postintervention SETQ. For postintervention qualitative assessment of the tool itself, 12 participants (3 residents, 7 attendings, 2 unspecified) provided feedback.
The tool was well-received with large subjective benefit in improving resident surgical education. A total of 88% thought that the time spent on the debriefings was “just right” and 91% planned to make the debriefings a regular part of operative performance assessments. Despite this overwhelmingly positive feedback, there was no overall difference in pre- and postintervention SETQ scores for climate of surgical education in the Department (4.25 ± 0.55 vs. 4.10 ± 0.88, p = 0.63).
Introduction of 4 item preoperative briefing and 4 item postoperative debriefing checklists was welcomed by both residents and faculty for its ability to shape surgical education in the operating room into a guided discovery model of hands-on education. Overall SETQ scores did not change, but most participants found value in the tool and plan to continue its use.
How learning preferences and teaching styles influence effectiveness of surgical educators
2021, American Journal of Surgery
Citation Excerpt :
The educator’s score is calculated based on the responses to questions for each of the six styles (1- the flexible and all round adaptable teacher, 2- the students centred, sensitive teacher, 3- the official curriculum teacher, 4- the straight facts no nonsense teacher, 5- the big conference teacher, and 6-the one off teacher, definitions in Table 1). The SETQ is a validated tool initially developed for assessment of teaching of anesthesia faculty, that has been adapted for use by surgeons.8,9 The tool requires self- and resident-assessment of each attending and comprises 26 questions in five pre-determined domains (learning climate, professional attitude towards residents, communication of goals, evaluation of residents and feedback).
Effective surgical educators have specific attributes and learner-relationships. Our aim was to determine how intrinsic learning preferences and teaching styles affect surgical educator effectiveness.
We determined i) learning preferences ii) teaching styles and iii) self-assessment of teaching skills for all general surgery attendings. All general surgical residents in our program completed teaching evaluations of attendings.
Multimodal was the most common learning preference (20/28). Although the multimodal learning preference appears to be associated with more effective educators than kinesthetic learning preferences, the difference was not statistically significant (80.0% versus 66.7%, p = 0.43). Attendings with Teaching Style 5 were more likely to have a lower “professional attitude towards residents” score on SETQ assessment by residents (OR 0.33 (0.11, 0.96), p = 0.04). Attendings rated their own “communication of goals” (p < 0.001), “evaluation of residents” (p = 0.04) and “overall teaching performance” (p = 0.01) per STEQ domains as significantly lower than the resident’s assessment of these cofactors.
Identification of factors intrinsic to surgical educators with high effectiveness is important for faculty development. Completion of a teaching style self-assessment by attendings could improve effectiveness.
Surgery Resident Perceptions of the Clicker Evaluation System: A Novel Approach to Collecting and Utilizing Clinical Faculty Performance Data
2021, Journal of Surgical Education
Medical trainees often have a process in place to receive feedback from clinical faculty regarding overall performance. While there is guidance on effective methodologies for faculty to provide feedback for learners, there is a dearth of literature analyzing trainees’ evaluation of faculty performance. We sought to identify an effective and anonymous method for surgery residents to evaluate clinical faculty.
The Department of Orthopedic Surgery at Virginia Commonwealth University (VCU) Health implemented a novel process to gather annual clinical faculty performance data from residents for the purpose of program improvement starting in 2012. Specifically, residents used a web-based audience response system, also known as a “clicker” system, to evaluate faculty performance over the academic year. During the June 2018 evaluation session, residents also completed an anonymous, 9 question survey to assess the residents’ perceptions regarding this clicker evaluation process.
VCU Health System, a tertiary care hospital in Richmond, Virginia.
All 24 orthopaedic surgery residents at VCU Health participated in the evaluation process and completed the perception survey in 2018.
Ninety-six percent (n = 23) of the residents agreed that they are able to accurately rate their attendings’ performance, felt confident that their responses remained anonymous, and that their departmental chair values their opinion when evaluating their attendings’ performance through the clicker process. Qualitative responses identified anonymity as a strength of the clicker process, while opportunities for improvement included refinement of questions.
The clicker evaluation system is an effective and anonymous method for resident evaluation of clinical faculty performance in academic settings. Future steps include refinement of questions based on departmental goals for education, adoption of the clicker evaluation system by other specialties, as well as research into ways to optimize the clicker evaluation process. Additional research should be done to see if and how the clicker evaluation feedback translates into change in clinical faculty behavior.

View all citing articles on Scopus

View full text

Original reportReliable and Valid Tools for Measuring Surgeons' Teaching Performance: Residents' vs. Self Evaluation

Background

Materials and Methods

Results

Conclusions

Introduction

Section snippets

Development of the SETQ for Surgical Specialties

Results

Main Findings

Explanation of Results

Conclusions

Acknowledgments

J Surg Res

Trainee satisfaction in surgery residency programs: modern management tools ensure trainee motivation and success

Surgery

The processes and dimensions of informed self-assessment: A conceptual model

Acad Med

Building a practically useful theory of goal setting and task motivationA 35-year odyssey

Am Psychol

Accuracy of physician self-assessment compared with observed measures of competence: A systematic review

JAMA

“I'll never play professional football” and other fallacies of self-assessment

J Contin Educ Health Prof

Feedback data sources that inform physician self-assessment

Med Teach

Features of assessment learners use to make informed self-assessments of clinical performance

Med Educ

“Directed” self-assessment: practice and feedback within a social context

J Contin Educ Health Prof

How reliable are assessments of clinical teaching?A review of the published instruments

J Gen Intern Med

Systematic evaluation of the teaching qualities of obstetrics and gynecology faculty: Reliability and validity of the SETQ tools

PLoS ONE

Development of a system for the evaluation of the teaching qualities of anesthesiology faculty

Anesthesiology

New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey

PLoS ONE

Original report
Reliable and Valid Tools for Measuring Surgeons' Teaching Performance: Residents' vs. Self Evaluation