Correspondence
Interrater agreement of emergency medicine milestone levels: resident self-evaluation vs clinical competency committee consensus,☆☆,

https://doi.org/10.1016/j.ajem.2016.04.055Get rights and content

Introduction

With the implementation of the Next Accreditation System (NAS), the Accreditation Council for Graduate Medical Education (ACGME) introduced 2 major paradigm shifts toward outcomes assessments: specialty-specific, competency-based milestones, and Clinical Competency Committees (CCC) [1] . As 1 of the 7 specialty “early adopters,” representatives from emergency medicine (EM) stakeholder organizations convened the EM Milestone Working Group, which developed and subsequently validated 23 milestones spanning the 6 core competencies [2], [3]. In December 2013, 162 EM residency training programs reported their initial milestone data to the ACGME for 5806 residents, which represented the consensus evaluations made by their respective CCCs [4] .

One of the resident-centric goals of the milestones is the provision of transparent expectations of performance to support better self-directed assessment and to more accurately assess plans for improvement and remediation [5] . Although the EM milestones have undergone validation by program directors and academic faculty, the extent to which residents are able to accurately assess their own performance using the milestones framework has yet to be determined [3] . Therefore, the objective of this study is to understand the extent to which milestone-based resident self-assessments agree with assessments made by the CCC. We hypothesize that interrater agreement will be higher for milestones with quantifiable measures (ie, those pertaining to the patient care and medical knowledge core competencies) compared with milestones that relate more to resident behavior (ie, those pertaining to the professionalism and interpersonal/communication skills core competencies) and that overall agreement will improve between the mid- and end-of-year reporting periods.

Section snippets

Study design and setting

We performed a retrospective, cross-sectional study at a single, 4-year EM residency training program in a high-volume, urban, academic medical center. The study was approved by the institutional review board at Columbia University Medical Center with waiver of written informed consent.

Study participants and protocol

The study participants included the EM house staff and CCC members at New York–Presbyterian Hospital, the University Hospital of Columbia and Cornell. At the beginning of the academic year, every member of the

Results

A total of 35 (73%) unique residents were compliant with the submission of self-assessment milestone data over the course of the reporting period. Twenty-nine (60%) submitted midyear self-evaluations, and 20 (42%) submitted end-of-year self-evaluations. Self-assessment submissions by resident year-of-training is delineated further in Table 1 . All residents (100%) were evaluated by the members of the CCC.

Overall, there was a moderate level of agreement (k= 0.458; 95% confidence interval,

Discussion

Agreement between resident self-assessment and CCC evaluation was moderate at best, with both junior and senior residents overestimating their level of proficiency. Although we hypothesized that with more exposure to the milestones framework, overall agreement would improve between mid- and end-of-year assessments, such acclimatization is not supported by our data. Our secondary hypothesis that agreement would be higher among patient care milestones for which there have traditionally been

Conclusions

Given the importance of self-assessment in determining ongoing learning needs, both in residency and careers beyond, these findings suggest the presence of an important gap for program directors to target to graduate residents with a skill set required for successful professional development. Although this outcomes based assessment program was specifically designed around graduate medical education, one of its covert hopes was to have residents become comfortable with this process, understand

References (8)

  • W.A. Carter

    Milestone myths and misperceptions

    J Grad Med Educ

    (2014)
  • M.S. Beeson et al.

    The development of the emergency medicine milestones

    Acad Emerg Med

    (2013)
  • R.C. Korte et al.

    Emergency medicine milestones working G, Reisdorff EJ. The emergency medicine milestones: a validation study

    Acad Emerg Med

    (2013)
  • Accreditation Council for Graduate Medical Education. Milestone News. ACGME e-Communication....
There are more references available in the full text version of this article.

Source of support: This publication was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant number UL1 TR000040. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

☆☆

Previous presentations: Council of Emergency Medicine Residency Directors, Annual Academic Assembly, March 2016.

Society for Academic Emergency Medicine, Annual Meeting, May, 2016.

View full text