Evaluating clinicians’ teaching performance

Clinician teachers play an essential role in residency training as they are responsible for educating residents to become excellent clinicians and at the same time to safeguard the quality of patient care. Valid and reliable evaluations of the strengths and weaknesses of clinicians’ teaching performance are usually a preliminary, although essential, activity in quality management and improvement activities [1]. Such evaluations can facilitate clinicians in generating a critical appraisal of their own performance in order to explore necessary follow-up actions to maintain or improve performance [2]. Although there is a strong history of evaluating the role of clinicians in teaching, a complete system with feedback and follow-up to improve clinicians’ teaching performance was often not available [2]. With the purpose to fulfil that need, the System for Evaluation of Teaching Qualities (SETQ) was developed by Lombarts et al. [3, 4].

This thesis focused on testing the validity, reliability and impact of the SETQ across specialities and settings. Most importantly, we studied (1) whether the SETQ instruments (questionnaires) can be used to evaluate clinicians’ teaching performance across specialities and centres with satisfactory validity and reliability and (2) whether clinicians’ teaching performance improved after receiving performance feedback. Finally, to get an indication of the predictive (criterion) validity of the SETQ evaluation scores, we studied (3) how clinicians’ teaching performance scores were related to their role modelling.

1: The results of study 1 and 2 indicated that the SETQ tools appear to be valid for gathering clinicians’ teaching performance data from residents and clinicians’ self-evaluation data. Based on the generalizability analysis of study 2, we can conclude that teaching performance evaluations based on three or more completed SETQ resident evaluations can generate statistically reliable teaching domain and overall scores.

2: The results of study 3 and 4 suggest that clinicians can improve their teaching performance by participating in the teaching performance evaluation system. In study 3 we showed that clinician teachers enhanced their performance scores, after receiving performance feedback for the first and the second time. However, this was not true for clinicians who overestimated their teaching performance during a previous evaluation (compared with the resident evaluations). That subgroup of clinician teachers received lower performance scores during subsequent evaluations. The results of study 4 indicate that residents perceived about 40 % of the clinician teachers to have slightly improved their teaching performance. This was indicated by residents’ ratings in between ‘not improved’ and ‘improved’ (when the ratings given by all residents were aggregated). About 7 % had greatly improved their performance, indicated by aggregated ratings between ‘improved’ and ‘greatly improved’. The strongest predictor of improvement was the number of narrative suggestions for improvement received by clinician teachers during a previous evaluation. The number of positive narrative comments did not predict performance improvement.

3: The results of study 5 showed that relatively strong associations exist between the teaching performance domain scores and clinicians’ role modelling. Overall, displaying a good professional attitude towards residents impacted role modelling most. In study 6, the associations remained—although they differed in strength—under a variety of plausible causal assumptions. The results of these two studies together confirmed our hypothesis that good clinician teachers would be evaluated as better role models by residents and contributed to the predictive validity of the SETQ performance scores.

The SETQ evaluation instruments appear to be valid and reliable in gathering performance feedback from residents. Besides, the SETQ instruments are suitable for both small and larger training programmes where other instruments are more suitable for larger training programmes only [12].

There is a robust body of literature showing that feedback can improve performance [13]. Nevertheless, there is a need for research focussing on which elements of feedback are most helpful for improving performance [13]. Our two studies focusing on performance improvement add to this body of knowledge by identifying that narrative suggestions for improvement predicted performance improvement, while narrative positive comments were not associated with performance improvement. Future evaluations should therefore stimulate feedback providers to include suggestions for improvement in their feedback.

Also, we found that overestimating performance resulted in a decline in performance scores. This finding is consistent with findings from the psychological literature which indicate that both overestimating and underestimating performance are frequently identified as constricting factors for performance improvement [14]. Additional research can study whether interventions such as structured guidance in interpreting and appreciating feedback can help overestimating clinicians in benefiting from performance feedback [15]. A recent systematic review found that only a few of the studies on performance evaluation tools assessed the predictive (criterion) validity [16]. In this thesis, we aimed to add to this knowledge by relating teaching performance scores to clinicians’ role modelling. The findings of our studies, which indicate a strong relationship between teaching performance scores and role modelling, strengthen the value of the performance scores. By enhancing their teaching performance, clinicians are likely to also enhance their role modelling, which will make the effect of their teaching more powerful and more effective.

First, we found that the SETQ can yield valid and reliable evaluations of clinicians’ teaching performance. Second, the performance evaluations can help clinicians in improving their teaching performance. Third, clinicians with higher teaching performance scores were also seen as better role models by residents, and can therefore probably enhance the power of their role modelling by improving their teaching performance.

Choose your collaborators wisely and carefully. Make sure that enough variety of expertise is grounded in your team. When certain expertise is underrepresented, seek out collaborators outside your current team. You will meet new people, have a lot of interesting interactions, and learn many new useful skills. And above all, it will benefit the quality of your research.