Observation of Resident Clinical Skills: Outcomes of a Program of Direct Observation in the Continuity Clinic Setting

doi:10.1016/j.acap.2011.02.008

Academic Pediatrics

Volume 11, Issue 5, September–October 2011, Pages 394-402

https://doi.org/10.1016/j.acap.2011.02.008 Get rights and content

Abstract

Objective

To assess the feasibility of a new multi-institutional program of direct observation and report what faculty observed and the feedback they provided.

Methods

A program of direct observation of real patient encounters was implemented in 3 pediatric residency programs using a structured clinical observation (SCO) form to document what was observed and the feedback given. Outcome variables included the number of observations made, the nature of the feedback provided, resident attitudes about direct observation before and after implementation, and the response of the faculty.

Results

Seventy-nine preceptors and 145 residents participated; 320 SCO forms were completed. Faculty provided feedback in 4 areas: content, process of the encounter, patient-centered attitudes and behaviors, and interpersonal skills. Feedback was 85% specific and 41% corrective. Corrective feedback was most frequent for physical examination skills. After program implementation, residents reported an increase in feedback and a decrease in discomfort with direct observation; in addition, they agreed that direct observation was a valuable component of their education. Participation rates among faculty were high.

Conclusions

Direct observation using SCOs results in timely and specific feedback to residents about behaviors rarely observed in traditional precepting models. Resident competency in these clinical skill domains is critical for assessing, diagnosing, and managing patients. The SCO methodology is a feasible way to provide formative feedback to residents about their clinical skills.

Section snippets

Setting

We implemented the program in a pediatric residency continuity clinic setting where residents work with 1 or 2 faculty members in a longitudinal clinical experience, and would be expected to have developed a trusting relationship in which to be observed and receive feedback. Implementation took place at 3 institutions: the Jefferson-duPont, Children’s National Medical Center (CNMC), and National Capital Consortium Military (NCC) Pediatrics Residency Programs during 2005 and 2006. CNMC has 3

Feasibility

During the implementation year, 67% of faculty at the 3 residency programs used the SCO method to observe residents. Among hospital-based preceptors, 80% of faculty used the SCO form at least once, with 100% of Jefferson-duPont faculty, 86% of NCC faculty, and 63% of CNMC faculty using the form. The CNMC program had a lower rate of participation as a result of the addition of a hospital-based continuity clinic based in another facility shortly before the direct observation program was begun on

Discussion

In this study, we analyzed the feasibility of implementing a program of direct observation of residents interacting with their patients in continuity clinic, the feedback given to residents after direct observation using the SCO tool, and the resident attitudes about being observed. Implementation of this method is feasible in training programs of different sizes and teaching settings. The feedback provided was on a broad range of skills and was mainly very specific. Although many tools for

Conclusions

Use of SCOs results in timely and specific feedback to residents about domains rarely observed in traditional precepting models that do not include direct observation. Resident competency in these domains is critical for assessment, diagnosis, and management of patients. The SCO methodology is an efficient way to provide formative feedback to residents about their clinical skills and implementation of a program of direct observation is feasible at both hospital-based and community-based

Acknowledgment

This work was supported by grants from Region IV Academic Pediatrics Association, Children’s National Medical Center Research Advisory Committee, and from the Association of Pediatric Program Directors Special Project Award. We are grateful for the support of the continuity clinic preceptors and residents at National Capital Consortium Military Pediatrics Residency program (NCC), Jefferson Medical College/duPont Hospital for Children Pediatric Residency Program, and Children’s National Medical

References (32)

P.G. Ramsey et al.
History-taking and preventive medicine skills among primary care physicians: an assessment using standardized patients
Am J Med
(1998)
R.C. Baker et al.
Exam room presentations and teaching in outpatient pediatrics: effects on visit duration and parent, attending physician, and resident perceptions
Ambul Pediatr
(2007)
P. Lye et al.
A pleasure to work with—an analysis of written comments on student evaluations
Ambul Pediatr
(2001)
Accreditation Council for Graduate Medical Education. Outcomes project. Available at: http://www.acgme.org. Accessed...
A. Beaumier et al.
Nature of the clinical difficulties of first-year family medicine residents under direct observation
CMAJ
(1992)
C.H. Braddock et al.
Informed decision making in outpatient practice: time to get back to basics
JAMA
(1999)
N.P. Wray et al.
Detection and correction of house staff error in physical diagnosis
JAMA
(1983)
S. Mangione et al.
Cardiac auscultatory skills of internal medicine and family practice trainees. A comparison of diagnostic proficiency
JAMA
(1997)
S. Mangione et al.
Physical diagnosis skills of physicians in training: a focused assessment
Acad Emerg Med
(1995)
J.T. Li
Assessment of basic physical examination skills of internal medicine residents
Acad Med
(1994)

R.A. Fox et al.

A study of pre-registration house officers’ clinical skills

Med Educ

(2000)

J.R. Meuleman et al.

Evaluating the interview performance of internal medicine interns

Acad Med

(1989)

G.L. Noel et al.

How well do internal medicine faculty members evaluate the clinical skills of residents?

Ann Intern Med

(1992)

A. Kalet et al.

How well do faculty evaluate the interviewing skills of medical students?

J Gen Intern Med

(1992)

A.R. Pulito et al.

What do faculty observe of medical students’ clinical performance?

Teach Learn Med

(2006)

A.L. Wendling

Assessing resident competency in an outpatient setting

Fam Med

(2004)

Cited by (18)

Teaching and assessing operative skills: From theory to practice
2017, Current Problems in Surgery
Citation Excerpt :
Entrustability of surgical trainees is dependent on direct observation of operative and patient care skills. Without witnessing the technical performance, anatomical knowledge, and leadership of a trainee, to name a few qualities contributing to trust and responsibility, there would be a chasm in the evidence needed to support a faculty surgeon’s entrustment decision.85 Pair this with an operative performance assessment that a faculty member completes for a trainee which captures that the trainee demonstrates hesitation, has a tremor, or causes tissue damage, and the feedback from the tool simply states “more practice.”
Targeting Simulation-Based Assessment for the Pediatric Milestones: A Survey of Simulation Experts and Program Directors
2016, Academic Pediatrics
Citation Excerpt :
Tobler et al24 showed that a simulation-based workshop, including standardized parents, improved pediatric residents' skills in having difficult conversations as evaluated by the trainees, physician expert reviewers, and actual parents. Debriefing after simulated educational experiences is used to explore learners' cognitive frameworks and shared mental models25–27 and is an ideal venue to assess self-insight (PBL1) and other internal domains of competency (including recognition and acceptance of ambiguity, PROF6) identified as difficult to assess by our survey.20,21,23,25,28–30 Responses by simulation experts appear to be grounded in practicality as well as in existing frameworks for simulation education described in the literature.
To determine which of the 21 general pediatrics milestone subcompetencies are most difficult to assess using traditional methodologies and which are best suited to simulation-based assessment.
We surveyed 2 samples: pediatric simulation experts and pediatric program directors. Respondents were asked about current use of simulation for assessment and to select 5 of the 21 pediatric subcompetencies most difficult to assess using traditional methods and the 5 best suited to simulation-based assessment. Spearman rank correlation was used to determine a correlation between how the 2 samples ranked the subcompetencies.
Forty-eight percent (29 of 60) simulation experts and 20% (115 of 571) program directors completed the survey. Few respondents reported using simulation for summative assessment. There are clear differences across the pediatric subcompetencies in perceived difficulty of assessment and suitability to simulation-based assessment. The 3 most difficult to assess subcompetencies were “recognize ambiguity,” “demonstrate emotional insight,” and “identify one's own strengths and deficiencies.” The subcompetencies most suitable to assessment using simulation were “interprofessional teamwork,” “clinical decision making,” and “effective communication.” Program directors and simulation experts had high agreement for both questions: difficult to assess (rho = 0.76, P < .001) and suitable to simulation-based assessment (rho = 0.94, P < .001).
Several general pediatrics milestone subcompetencies were identified by pediatric simulation experts and pediatric program directors as difficult to assess using current methodologies and as amenable to simulation-based assessment. The pediatric simulation community should target development of simulation-based assessment tools to these areas.
Setting standards to determine core clerkship grades in pediatrics
2014, Academic Pediatrics
One of the greatest challenges for clerkship directors is assigning a final grade and determining the precise point at which a student either passes or fails a clinical clerkship. The process of incorporating both subjective and objective assessment data to provide a final summative grade can be challenging. We describe our experience conducting a standard-setting exercise to set defensible cut points in a 4-tiered grading system in our pediatric clerkship.
Using the Hofstee standard-setting approach, 8 faculty members participated in an exercise to establish grade cut points. These faculty members were subsequently surveyed to assess their attitudes toward the standard-setting process as well as their reactions to these newly proposed standards. We applied the new cut points to a historic cohort of 116 Johns Hopkins University School of Medicine students from the academic year 2012–2013 to assess the potential impact on grade distributions.
The resultant grading schema would lead to a significant increase in the number of students receiving a failing grade and a decrease in the number of students receiving a grade of honors in a historical cohort. Faculty reported that the Hofstee method was easy to understand and fair. All faculty members thought that grade inflation presently exists within the pediatric clerkship.
This study demonstrates that practical standards using the Hofstee method can be set for medical students in a pediatric clerkship in which multiple performance measures are used.
Do you know it when you see it? Tips for recognizing good studies in medical education
2011, Academic Pediatrics
Implementation, Reach, and Effectiveness of the Direct Observation of Residents in Clinic Program
2023, Education for Health: Change in Learning and Practice
Application of the ABCDE method by residents in clinical practice: a prospective observational study
2023, Research Square

View all citing articles on Scopus

: The authors have no conflicts of interest to disclose.

View full text

EducationObservation of Resident Clinical Skills: Outcomes of a Program of Direct Observation in the Continuity Clinic Setting

Abstract

Objective

Methods

Results

Conclusions

Section snippets

Setting

Feasibility

Discussion

Conclusions

Acknowledgment

Am J Med

Ambul Pediatr

Ambul Pediatr

Nature of the clinical difficulties of first-year family medicine residents under direct observation

CMAJ

Informed decision making in outpatient practice: time to get back to basics

JAMA

Detection and correction of house staff error in physical diagnosis

JAMA

Cardiac auscultatory skills of internal medicine and family practice trainees. A comparison of diagnostic proficiency

JAMA

Physical diagnosis skills of physicians in training: a focused assessment

Acad Emerg Med

Assessment of basic physical examination skills of internal medicine residents

Acad Med

A study of pre-registration house officers’ clinical skills

Med Educ

Evaluating the interview performance of internal medicine interns

Acad Med

How well do internal medicine faculty members evaluate the clinical skills of residents?

Ann Intern Med

How well do faculty evaluate the interviewing skills of medical students?

J Gen Intern Med

What do faculty observe of medical students’ clinical performance?

Teach Learn Med

Assessing resident competency in an outpatient setting

Fam Med

Education
Observation of Resident Clinical Skills: Outcomes of a Program of Direct Observation in the Continuity Clinic Setting