Skip to main content
Top
Gepubliceerd in: Perspectives on Medical Education 3/2020

Open Access 30-03-2020 | Show and Tell

Longitudinal resident coaching in the outpatient setting: A novel intervention to improve ambulatory consultation skills

Auteurs: Ryan Graddy, Stasia S. Reynolds, Scott M. Wright

Gepubliceerd in: Perspectives on Medical Education | Uitgave 3/2020

share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail
insite
ZOEKEN

Abstract

Background

Direct observation with feedback to learners should be a mainstay in resident education, yet it is infrequently done and its impact on consultation skills has rarely been assessed.

Approach

This project presents the framework and implementation of a longitudinal low-frequency, high-intensity direct observation and coaching intervention, and elaborates on insights learned. Internal medicine interns at one residency training program were randomized to an ambulatory coaching intervention or usual precepting. Over one year, coached interns had three complete primary care visits directly observed by a faculty clinician-coach who provided feedback informed by a behavior checklist. Immediately after each of the coached patient encounters, interns completed a structured self-assessment and coaches led a 30-minute feedback session informed by intern self-reflection and checklist items. Interns with usual precepting had two mini-CEX observations over the course of the year without other formal direct observation in the ambulatory setting.

Evaluation

As part of the post-intervention assessment, senior faculty members blinded to intervention and control group assignments evaluated videotaped encounters. Coached interns completed an average of 21/23 behaviors from the checklist, while interns from the control group completed 18 (p < 0.05). The median overall grade for coached interns was B+, compared to B−/C+ for controls (p < 0.05).

Reflection

Coaching interns longitudinally using a behavior checklist is feasible and associated with improved consultation performance. Direct observation of complete clinical encounters followed by systematic coaching is educationally valuable, but time and resource intensive.

Background and need for innovation

There is widespread agreement that direct observation and feedback on clinical practice is critical for professional growth during residency education; this formative assessment is believed to be invaluable [1, 2]. Residency training has the potential for development of rich consultation skills—with repeated, intense practice alongside peers and under the supervision of committed educators. Despite these opportunities, the vast majority of resident clinical practice is unobserved by anyone other than the patient. Usual precepting of resident physicians is of short duration, often haphazard, and can involve fragmented supervision by faculty preceptors. Because faculty cannot offer constructive feedback on behaviors they have not seen, residents often rely on self-assessments which are commonly inaccurate [3, 4]. Time constraints, the absence of structure to accommodate direct observation, and lack of financial compensation have all been cited as barriers to incorporating direct observation regularly into graduate medical education [5].
Coaching paradigms place emphasis on self-assessment, reflection-on-practice, and thoughtfully stimulated feedback immediately following direct observation [68]. While focused, episodic direct observation has been shown to improve some procedural techniques in surgical training [9], its impact on observed consultation performance—particularly longitudinally—has not previously been studied in outpatient settings.

Goal of innovation

We set out to develop and evaluate a longitudinal direct observation and coaching intervention designed to improve outpatient consultation skills among internal medicine interns. Here, consultation skills refers to the ingredients of an outpatient primary care encounter including professionalism, verbal and non-verbal communication, medical knowledge, diagnostic acumen, use of electronic medical records, and negotiation of the healthcare system. In this article, we discuss the conceptual framework and implementation of our intervention, present outcomes data from our pilot study, and share details about the successes and pitfalls that can inform other educators and training programs who may be considering similar educational methods.

Development and implementation of innovation

This direct observation intervention was grounded in the coaching paradigm, a model that places an emphasis on reflection-on-practice, self-assessment, and direct observation by a skilled ‘coach’ who provides detailed, specific, and timely feedback [68]. Using this model in graduate medical education, residents engage in self-assessment of their performance prior to receiving feedback from trusted faculty educators; this practice allows trainees to develop refined self-assessment techniques and to enhance their consultation skills [10]. Clinical coaching requires repeated direct observation and a longitudinal relationship between the learner and the experienced guide [11].
The intervention was implemented with interns in the Johns Hopkins Bayview Internal Medicine Residency in Baltimore, MD, over the course of the 2017–18 academic year. At the beginning of the year, half of the interns (8 of the 16 trainees) were randomly selected (using random.org) to the intervention group where they would be longitudinally coached by one of two faculty preceptors with extensive coaching and precepting experience. The four male interns (25%) were equally distributed between the two study groups. Over the course of 12 months, each coach performed three complete outpatient visit observations with each of their learners. During these patient encounters, faculty remained in the room for the entirety of a patient visit; a 23-item consultation behavioral checklist, adapted from previous work [12], was used to focus the observations. Immediately following each encounter, there was a 30-minute coaching and debriefing session. First, the intern would fill out a self-assessment that included the behavioral checklist. Next, the intern would be asked to reflect on his/her performance, guided by insightful questioning by the coach. Finally, the coach would deliver specific, detailed feedback about the intern’s performance. These formative feedback sessions were used to capitalize on longitudinal coaching relationships between faculty and interns through the creation of specific goal-setting for future outpatient encounters, as well as reviewing progress toward previously articulated goals.
The eight interns who did not receive any targeted intervention in coaching or direct observation received the standard outpatient precepting that was offered to all residents. This includes twice annual partially observed encounters by faculty using a mini-CEX format [13], which was also continued for interns in the intervention group.

Evaluation of the innovation

At the end of the academic year, 15/16 interns had a complete patient visit in the practice videotaped. Encounters that were videotaped were selected as a convenience sample on days in June 2018 when a study team member (SR) was available to do so. One intern in the control group declined to be videotaped for the project, all others consented. Interns were asked to deliver care as per usual and to disregard the taping; no guidance was given to the interns about who would view the tapes or how they would be scored.
Seven experienced general internal medicine and geriatrics faculty members (average 26 years post-residency clinical practice) with no direct involvement in the coaching portion of the project were recruited to independently evaluate videotaped intern consultation performances. These clinician-educators received an email introduction to the project and were asked if they would be willing to volunteer as video raters; they were not compensated for their time. They were blinded to intervention and control group interns and were given brief email training on use of the same 23-item checklist used by coaching faculty in the intervention group. Additionally, raters were asked to assign a letter grade (A–F) to each video based on their evaluation of the intern’s consultation performance, independent of the checklist and instead based on their overall impression. Grades were transitioned from letters to numbers for statistical analysis where F = 0 and A+ = 12. Each video was viewed by two raters. Discrepant yes/no responses were reconciled to ‘yes,’ where one rater’s identification of a completed behavior was deemed accurate over another’s reported omission of the behavior (this occurred in 19% of items and equally in both groups). The a priori primary outcome of the analysis for this study was the number of items observed on the video as per the assessment of the raters. We hypothesized that coached interns would perform a higher proportion of checklist behaviors and receive higher grades than interns in the control group. The proportions of each behavior performed were compared using chi-squared tests, and differences in the two groups’ overall performance of behaviors on the checklist and grades were assessed with two-tailed Mann-Whitney U tests with α = 0.05. This study was reviewed and deemed exempt by the Johns Hopkins University School of Medicine Institutional Review Board.
On the videotaped encounters, eleven of the 23 behaviors were executed with greater frequency among interns who had been coached, and 2/23 were performed more often among the controls. No individual checklist items were performed statistically significantly more often by either group (all p > 0.05, see Tab. 1). Overall, coached interns were judged to complete a median of 19/23 behaviors, while uncoached interns completed 16 (Mann-Whitney U—p < 0.05 two-tailed). The median overall grade for coached interns was B+, compared with B−/C+ for the controls (Mann-Whitney U—p < 0.05 two-tailed). The distribution of grades across groups is shown in the Fig. 1.
Table 1
Intern completion of checklist behaviors as assessed by experienced senior clinicians at end of academic year
Checklist item
Coached
Control
p value
N = 8 (% complete)
N = 7 (% complete)
Is fully present during encounter
8 (100)
7 (100)
1.00
Listens attentively
8 (100)
7 (100)
1.00
Avoids medical jargon
8 (100)
7 (100)
1.00
Positions self to facilitate communication
8 (100)
7 (100)
1.00
Gives undivided attention to patient before/after charting
8 (100)
7 (100)
1.00
Employs adequate eye contact when using EMR
8 (100)
7 (100)
1.00
Obtains adequate information to arrive at diagnosis/plan
8 (100)
7 (100)
1.00
Interprets data and formulates plan appropriate for problem
8 (100)
7 (100)
1.00
Creates comprehensive management plan for patient
8 (100)
7 (100)
1.00
Enables patient to understand comprehensive management plan
8 (100)
7 (100)
1.00
Describes evidence base to treat patient
5 (63)
5 (71)
0.71
Closes visit with open-ended question
6 (75)
4 (57)
0.46
Completes appropriate physical exam
7 (88)
7 (100)
0.33
Asks open-ended questions
8 (100)
6 (86)
0.27
Collaborates with patient using EMR
8 (100)
6 (86)
0.27
Manages visit time appropriately
8 (100)
6 (86)
0.27
Counsels on behavioral change
6 (75)
3 (43)
0.21
Uses appropriate physical exam technique
6 (75)
3 (43)
0.21
Outlines reasons to re-contact
6 (75)
3 (43)
0.21
Demonstrates ability to prioritize multiple issues during visit
7 (88)
4 (57)
0.19
Addresses social/financial concerns
7 (88)
4 (57)
0.19
Asks thoughtful questions
8 (100)
5 (71)
0.10
Assesses understanding (teach back)
6 (75)
2 (29)
0.07

Critical reflection

This longitudinal, structured coaching intervention in the ambulatory setting demonstrates promise as a model for increasing the frequency of high quality, focused direct observation of resident encounters with patients. Interns experienced the value of reflection on practice, self-assessment, and collaborating with a coach. This intervention was associated with better overall consultation performance and completion of more checklist items compared with usual ambulatory precepting.
Previous direct observation and coaching interventions in the outpatient setting have demonstrated acceptability among residents and faculty, improved resident confidence in faculty feedback, and better identification of blind spots in resident self-assessment [11, 13, 14]. This paper adds to this sparse literature by describing how this is grounded in a conceptual framework and providing some evidence for its educational value in changing behaviors.
In the course of formulating and executing the intervention, we learned a great deal. First, we found that directly observing the entirety of outpatient consultation encounters using a detailed checklist provided substantially more information than traditional precepting or sporadic mini-CEXes. Witnessing a whole visit allowed coaches to evaluate interns’ ability to prioritize multiple issues during a visit and manage time, skills that are incompletely captured in partial evaluations yet are important to understand. Further, the checklist provided more specific details than the broad categories included in the mini-CEX checklist; it breaks down outpatient consultation skills into discrete behaviors. Previous studies suggest that feedback from mini-CEXes is often felt to be non-specific and of limited value [15, 16]. Formative assessment of discrete, actionable behaviors is an important means of creating consistency in evaluations and for providing specific feedback. The longitudinal nature of our intervention led to the development of trusting coach-coachee relationships in which feedback became more focused and targeted over time. Coaching relationships in other disciplines, including music and sports, depend on longitudinal exposure in which feedback is more easily tailored to the individual over repeated sessions and evolves over time [17]. In medical student training, learners perceive preceptor assessment and feedback as more authentic when observed consistently by a single faculty expert [18]. This longer and more consistent coaching contact with residents may be difficult to schedule; in addition, this is both time and resource intensive. The optimal number of coached patient encounters and duration of coaching relationship are unknown. Finally, our project benefited significantly from institutional support to pursue coaching; both faculty members serving as coaches had built-in flexibility to their academic schedules to pursue projects to improve the quality of ambulatory resident education. Previous work has shown that the quality of clinical teaching is linked to the amount of protected time in academic settings [19, 20].
Several limitations of this study should be considered. First, this project was conducted at a single clinical site in a single internal medicine residency program. Second, the number of interns in our training program is fairly small, so there were few trainees in the intervention and control groups. With a small n, it is possible that some results may be due to chance. Given the frequent interactions of interns in the control and coached groups in and out of clinic, it is also possible that coached interns shared some pearls of wisdom learned through coaching with the other interns. Third, the faculty raters who reviewed videotaped encounters may not have interpreted items exactly as intended. We sought to minimize the problem by providing them with brief training in how to complete the checklist; as such, we gave learners credit for performance of a behavior if either independent observer endorsed completion of the behavior, because it may have been difficult for observers to track all checklist items. Fourth, the checklist has not been validated as a summative scoring rubric. Finally, the coached group’s familiarity with the checklist may have influenced their ability to perform better on the videotaped encounter.
Further evaluations of the utility and viability of this direct observation and coaching strategy in other settings will help to determine its broader applicability. Future steps may include validation of the checklist tool and application to other resident practices to assess both feasibility and impact on trainees’ consultation skills. Additional work to evaluate the impact of different doses of coaching on learner consultation performance may also assist in the development of the best and most efficient models for outpatient coaching.
In conclusion, this pilot found that an innovative longitudinal outpatient direct observation and coaching intervention for interns was associated with improved consultation performance compared with usual precepting in the ambulatory setting.

Funding

This study was supported by the Johns Hopkins Osler Center for Clinical Excellence.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creativecommons.​org/​licenses/​by/​4.​0/​.
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail
Literatuur
1.
go back to reference Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90:411–3.CrossRef Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90:411–3.CrossRef
2.
go back to reference ACGME, ABIM. The internal medicine milestone project. 2015. ACGME, ABIM. The internal medicine milestone project. 2015.
3.
go back to reference Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094–102.CrossRef Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094–102.CrossRef
4.
go back to reference Mazor KM, Holtman MC, Shchukin Y, Mee J, Katsufrakis PJ. The relationship between direct observation, knowledge, and feedback: results of a national survey. Acad Med. 2011;86:S63–S8.CrossRef Mazor KM, Holtman MC, Shchukin Y, Mee J, Katsufrakis PJ. The relationship between direct observation, knowledge, and feedback: results of a national survey. Acad Med. 2011;86:S63–S8.CrossRef
5.
go back to reference Madan R, Conn D, Dubo E, Voore P, Wiesenfeld L. The enablers and barriers to the use of direct observation of trainee clinical skills by supervising faculty in a psychiatry residency program. Can J Psychiatry. 2012;57:269–72.CrossRef Madan R, Conn D, Dubo E, Voore P, Wiesenfeld L. The enablers and barriers to the use of direct observation of trainee clinical skills by supervising faculty in a psychiatry residency program. Can J Psychiatry. 2012;57:269–72.CrossRef
6.
go back to reference Min H, Morales DR, Orgill D, Smink DS, Yule S. Systematic review of coaching to enhance surgeons’ operative performance. Surgery. 2015;158(5):1168–91.CrossRef Min H, Morales DR, Orgill D, Smink DS, Yule S. Systematic review of coaching to enhance surgeons’ operative performance. Surgery. 2015;158(5):1168–91.CrossRef
7.
go back to reference Rangachari D, Brown LE, Kern DE, Melia MT. Clinical coaching: evolving the apprenticeship model for modern housestaff. Med Teach. 2017;39:780–2.CrossRef Rangachari D, Brown LE, Kern DE, Melia MT. Clinical coaching: evolving the apprenticeship model for modern housestaff. Med Teach. 2017;39:780–2.CrossRef
8.
go back to reference LeBlanc C, Sherbino J. Coaching in emergency medicine. Can J Emerg Med. 2010;12:520–4. LeBlanc C, Sherbino J. Coaching in emergency medicine. Can J Emerg Med. 2010;12:520–4.
9.
go back to reference Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008;15:988–94.CrossRef Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008;15:988–94.CrossRef
10.
go back to reference Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50:943–54.CrossRef Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50:943–54.CrossRef
11.
go back to reference Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90:1471–86.CrossRef Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90:1471–86.CrossRef
12.
go back to reference Graddy R, Reynolds SS, Wright SM. Coaching residents in the ambulatory setting: faculty direct observation and resident reflection. J Grad Med Educ. 2018;10:449–54.CrossRef Graddy R, Reynolds SS, Wright SM. Coaching residents in the ambulatory setting: faculty direct observation and resident reflection. J Grad Med Educ. 2018;10:449–54.CrossRef
13.
go back to reference Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX : a method for assessing clinical skills. Ann Intern Med. 2003;138:476–81.CrossRef Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX : a method for assessing clinical skills. Ann Intern Med. 2003;138:476–81.CrossRef
14.
go back to reference Smith J, Jacobs E, Li Z, Vogelman B, Zhao Y, Feldstein D. Successful implementation of a direct observation program in an ambulatory block rotation. J Grad Med Educ. 2017;9:113–7.CrossRef Smith J, Jacobs E, Li Z, Vogelman B, Zhao Y, Feldstein D. Successful implementation of a direct observation program in an ambulatory block rotation. J Grad Med Educ. 2017;9:113–7.CrossRef
17.
go back to reference Watling C, Driessen E, van der Vleuten CPM, Lingard L. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 2014;48:713–23.CrossRef Watling C, Driessen E, van der Vleuten CPM, Lingard L. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 2014;48:713–23.CrossRef
18.
go back to reference Bates J, Konkin J, Suddards C, Dobson S, Pratt D. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47:362–74.CrossRef Bates J, Konkin J, Suddards C, Dobson S, Pratt D. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47:362–74.CrossRef
19.
go back to reference Schiekirke-Schwake S, Anders S, von Steinbuchel N, Becker J, Raupach T. Facilitators of high-quality teaching in medical school: findings from a nation-wide survey among clinical teachers. BMC Med Educ. 2017;17(1):178.CrossRef Schiekirke-Schwake S, Anders S, von Steinbuchel N, Becker J, Raupach T. Facilitators of high-quality teaching in medical school: findings from a nation-wide survey among clinical teachers. BMC Med Educ. 2017;17(1):178.CrossRef
20.
go back to reference Denton G, Griffin R, Cazabon P, Monks S, Deichmann R. Recruiting primary care physicians to teach medical students in the ambulatory setting: A model of protected time, allocated money, and faculty development. Acad Med. 2015;90:1532–5.CrossRef Denton G, Griffin R, Cazabon P, Monks S, Deichmann R. Recruiting primary care physicians to teach medical students in the ambulatory setting: A model of protected time, allocated money, and faculty development. Acad Med. 2015;90:1532–5.CrossRef
Metagegevens
Titel
Longitudinal resident coaching in the outpatient setting: A novel intervention to improve ambulatory consultation skills
Auteurs
Ryan Graddy
Stasia S. Reynolds
Scott M. Wright
Publicatiedatum
30-03-2020
Uitgeverij
Bohn Stafleu van Loghum
Gepubliceerd in
Perspectives on Medical Education / Uitgave 3/2020
Print ISSN: 2212-2761
Elektronisch ISSN: 2212-277X
DOI
https://doi.org/10.1007/s40037-020-00573-5

Andere artikelen Uitgave 3/2020

Perspectives on Medical Education 3/2020 Naar de uitgave