Abstract
BACKGROUND
Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.
OBJECTIVE
We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.
DESIGN
This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.
MAIN MEASURES
Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.
KEY RESULTS
Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785–0.866 vs. 0.749, 0.686–0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.
CONCLUSIONS
This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
ACGME. Program Director Guide to the Common Requirements. 2011 [cited 2011 January 13]; Available from: http://www.acgme.org/acWebsite/home/home.asp.
Chaudhry SI, Holmboe E, Beasley BW. The state of evaluation in internal medicine residency. J Gen Intern Med. 2008;23(7):1010–5.
Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387–96.
Gray JD. Global rating scales in residency education. Acad Med. 1996;71(1 Suppl):S55–63.
Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings' post-rotation performance ratings detect residents' clinical performance deficiencies? Acad Med. 2004;79(5):453–7.
Levine HG, Yunker R, Bee D. Pediatric resident performance. The reliability and validity of rating forms. Eval Health Prof. 1986;9(1):62–74.
Marienfeld RD, Reid JC. Six-year documentation of the easy grader in the medical clerkship setting. J Med Educ. 1984;59(7):589–91.
Ryan JG, Mandel FS, Sama A, Ward MF. Reliability of faculty clinical evaluations of non-emergency medicine residents during emergency department rotations. Acad Emerg Med. 1996;3(12):1124–30.
Thorndike EL. A constant error in psychological ratings. J Appl Psychol. 1920;4:25–9.
McKinstry BH, Cameron HS, Elton RA, Riley SC. Leniency and halo effects in marking undergraduate short research projects. BMC Med Educ. 2004;4(1):28.
Hemmer PA, Pangaro L. The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med. 1997;72(7):641–3.
Hemmer PA, Grau T, Pangaro LN. Assessing the effectiveness of combining evaluation methods for the early identification of students with inadequate knowledge during a clerkship. Med Teach. 2001;23(6):580–4.
Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students' professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75(2):167–73.
Albritton TA, Fincher RM, Work JA. Group evaluation of student performance in a clerkship. Acad Med. 1996;71(5):551–2.
Beckman TJ, Mandrekar JN. The interpersonal, cognitive and efficiency domains of clinical teaching: construct validity of a multi-dimensional scale. Med Educ. 2005;39(12):1221–9.
Beckman TJ, Mandrekar JN, Engstler GJ, Ficalora RD. Determining reliability of clinical assessment scores in real time. Teach Learn Med. 2009;21(3):188–94.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Risucci DA, Lutsky L, Rosati RJ, Tortolani AJ. Reliability and accuracy of resident evaluations of surgical faculty. Eval Health Prof. 1992;15(3):313–24.
Jacobs R, Kozlowski SW. A closer look at halo error in performance ratings. Academy of Management Journal. 1985;28(1):pp.
Murphy KR, Jako RA, Anhalt RL. Nature and consequences of halo error: a critical analysis. J Appl Psychol. 1993;78(2):218–25.
Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med. 2004;79(4):343–6.
Iglehart JK. Revisiting duty-hour limits–IOM recommendations for patient safety and resident education. N Engl J Med. 2008;359(25):2633–5.
Holmboe ES, Rodak W, Mills G, McFarlane MJ, Schultz HJ. Outcomes-based evaluation in resident education: creating systems and structured portfolios. Am J Med. 2006;119(8):708–14.
Silber CG, Nasca TJ, Paskin DL, Eiger G, Robeson M, Veloski JJ. Do global rating forms enable program directors to assess the ACGME competencies? Acad Med. 2004;79(6):549–56.
Holmboe E, Hawkins RE. Practical Guide to the Evaluation of Clinical Competence. Philadelphia: Mosby; 2008.
Speer AJ, Solomon DJ, Ainsworth MA. An innovative evaluation method in an internal medicine clerkship. Acad Med. 1996;71(1 Suppl):S76–8.
Hemmer PA, Pangaro L. Using formal evaluation sessions for case-based faculty development during clinical clerkships. Acad Med. 2000;75(12):1216–21.
Battistone MJ, Milne C, Sande MA, Pangaro LN, Hemmer PA, Shomaker TS. The feasibility and acceptability of implementing formal evaluation sessions and using descriptive vocabulary to assess student performance on a clinical clerkship. Teach Learn Med. 2002;14(1):5–10.
Williams RG, Schwind CJ, Dunnington GL, Fortune J, Rogers D, Boehler M. The effects of group dynamics on resident progress committee deliberations. Teach Learn Med. 2005;17(2):96–100.
Kisiel JB, Bundrick JB, Beckman TJ. Resident physicians' perspectives on effective outpatient teaching: a qualitative study. Adv Health Sci Educ Theory Pract. 2010;15(3):357–68.
Acknowledgement
This project was supported in part through an Educational Innovations grant, Mayo Clinic Rochester.
Conflicts of Interest
None disclosed.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Thomas, M.R., Beckman, T.J., Mauck, K.F. et al. Group Assessments of Resident Physicians Improve Reliability and Decrease Halo Error. J GEN INTERN MED 26, 759–764 (2011). https://doi.org/10.1007/s11606-011-1670-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-011-1670-4