Second year medical students (
n = 152) completed 53 projects that spanned a broad range of clinical microsystems across three affiliated health systems, five types of clinical settings, and 16 medical sub-specialties. Tab.
1 describes the key characteristics of the projects, Box 1 shows a sample project; a description of two further sample projects can be found in the ESM Box and ESM resource 5 lists all the projects. Twenty-six coaches (96%) and 13 QI leads (68%) completed follow-up surveys for 49 of the projects (92%) (ESM resource 3). We organized our findings into three categories of system-level outcomes with multiple indicators for each category. The first category represents immediate outcomes (accomplishment of project goals), the second captures effects after the project ended, and the third describes project alignment with health system priorities and processes.
Table 1
Key features of medical students’ clinical microsystem clerkship health systems improvement projects, 2017–18
Health system |
Academic | 22 (42) |
Safety net | 19 (36) |
Veterans affairs | 12 (23) |
Clinical microsystem type |
Adult medicine | 48 (91) |
Pediatric medicine | 5 (9) |
Ambulatory care | 27 (51) |
Acute care | 18 (34) |
Surgical/perioperative care | 7 (13) |
Other | 1 (2) |
Project aims to improve … (IOM priorities34 projects may address multiple) (Indicator 3.1) |
Effectiveness | 41 (77) |
Safety | 18 (34) |
Patient-centeredness | 10 (19) |
Efficiency | 9 (17) |
Equity | 8 (15) |
Timeliness | 5 (9) |
Type of measure (Indicator 2.3) |
Structural change | 1 (2) |
Process change | 36 (68) |
Patient outcome | 16 (30) |
Project achieved at least 1 aim (based on data provided in project summary report or poster) (Indicators 1.1 and 2.3) |
Yes | 28 (53) |
– Structural changes | 1 |
– Process changes | 19 |
– Patient outcomes | 8 |
No | 24 (45) |
– Process changes | 16 |
– Patient outcomes | 8 |
Could not determine | 1 (2) |
– Process changes | 1 |
Category 1: Project goals accomplished by end of the curriculum
Indicator 1.1 Achievement of project aims: More than half (53%) of the project summaries and posters indicated achievement of at least one aim (Tab.
1).
Indicator 1.2 Achievement of educational goals: Survey data indicated that most physician coaches and QI leads (86%) agreed that students achieved the primary educational goal of applying HSI principles. Respondents commented on students’ appreciation of the complexities of quality improvement, valuing patients and interprofessional colleagues as key stakeholders in the improvement process, applying the steps of Lean methodology through a hands-on experience, conducting outcome measurement from Plan-Do-Study-Act (PDSA) cycles, and presentation skills. These findings suggest that members of the health system felt that students learned to apply HSI knowledge and skills, a potentially important health system goal since many students will go on to train and work in these settings.
Category 2: Effects of the HSI project on the system
Indicator 2.1 Perceived effects at the end of the curriculum (15 months): Based on coach and QI lead survey responses, many projects (67%) had a moderate or substantial impact on the microsystem at the time students completed their involvement with the project (Tab.
2)—even if project aims were not achieved. For projects perceived as having minimal or no effect on the microsystem, reasons included structural or bureaucratic barriers (e.g. difficulty getting intervention approved), insufficient time for students to complete the project (e.g. poor project management, unrealistic scope, or schedule conflicts that limited participation in key activities), and project completed or adopted by another group separate from students’ efforts.
Table 2
Perceived impact of medical students’ clinical microsystem clerkship health systems improvement projects from 2017–18 based on follow-up survey responses from coaches and QI leads (collected in May 2019)
Indicator 2.1 Perceived impact at project end (15 months) |
Impact of project on microsystem at time students left (approx. Nov 2018) |
None (e.g., did not meet proposed goal) | 3 (6) |
Minimal (e.g., project part of larger effort, hard to connect outcomes to students’ efforts, persistent structural barriers) | 13 (27) |
Moderate (e.g., students sparked interest that increased outreach efforts, pilot phase) | 23 (47) |
Substantial (e.g., improvement in valued metrics such as documentation, screening, illuminated a problem that was then addressed, provided a framework or foundation for future efforts) | 10 (20) |
Indicator 2.2 Perceived lasting impact of project (7-month post-curriculum) |
Did the project have impact after the students left the microsystem? (approx. May 2019) |
Yes because … | 37/49 (76) |
– Project (including all, some piece of it, or next phase) was taken on by health system staff | 26/37 (70) |
– Project has been ‘hardwired’ into the microsystem, requires minimal effort to sustain | 16/37 (43) |
– Project (including all, some piece of it, or next phase) was taken on by a new cohort of students | 3/37 (8) |
– Other (e.g., educational efforts help sustain change; culture shift among key health professionals in ways that support the intervention; identified a need for outside consultants who then had a huge impact) | 5/37 (14) |
No because … | 12/49 (25) |
– When the students left there was no one to do the work | 6/12 (50) |
– Project was no longer relevant (circumstances/priorities changed) | 3/12 (25) |
– Project identified other areas that needed to be prioritized | 2/12 (17) |
– Other (e.g., lack of buy in from key health professionals) | 5/12 (42) |
Indicator 2.2 Perceived effects post-curriculum (7 months post-curriculum): Based on coach and QI lead survey responses, many projects (76%) had a sustained impact on the microsystem seven months later (Tab.
2). One respondent described how student engagement could inspire subsequent efforts,
“Student work sparked interest in focusing specifically on hypertension control for African Americans in the broader primary care organization. Outreach efforts were expanded.” Coaches and QI leads attributed sustainability to health system staff continuing the improvement work and/or incorporation of interventions into usual processes. The most commonly cited reasons projects were not sustained included a shift in health system priorities, and lack of resources to continue the work without students, as one respondent wrote, “
There was no one to push to work forward after the students left and the project fell by the wayside.”
Indicator 2.3 Structures, processes, and patient outcomes: Projects focused on changing structures (1 project), processes (36 projects), or patient outcomes (16 projects). One project made a structural change by producing a handout about delirium for family members of patients in the intensive care unit. Of the 36 projects focused on process changes, 19 (53%) achieved their primary aim. For example, the depression screening rate in a primary care clinic increased from 58% to 63%, the median time from door to thrombolytic infusion in patients with stroke decreased from 38 to 30 min, and the proportion of psychiatric inpatients attending a first outpatient mental appointment increased from 55% to 64%. Of the 16 projects focused on changing a patient outcome measure, 8 (50%) achieved their primary aim. For example, the rate of in-hospital injury from falls decreased from 0.58 to 0.52 per 1000 patients and influenza vaccination rates increased from 37 to 58% in a primary care clinic.
Indicator 2.4 Balance of costs and benefits: While our data did not allow a true cost-benefit analysis, information about barriers and facilitators to project success gave us insight into some required resources or investments from the system to support the HSI projects. These included staff time to meet with students, participate in the design and pilot implementation of interventions, as well as helping students navigate the microsystem (e.g., accessing data and connecting them with key stakeholders and local champions). Additional time for coaches and QI leads to assist with project management was also important.
The benefits described by coaches and QI leads largely focused on the educational value to students such as learning and experiencing “the steps of QI work,” “how to work with a multidisciplinary team,” and “the crucial role of getting buy-in from all stakeholders.” While some coaches and QI leads highlighted small but valuable improvements from students efforts, such as raising awareness of gaps in care, introducing processes that helped streamline and standardize care, figuring out why a proposed intervention might not work and “improving the quality of the (patient satisfaction) tool and testing workflow for its administration”, others felt it was difficult or even unrealistic “for students to truly improve a system with the amount of time they have.” Nonetheless, many saw the value of the system long-term, noting “if they get an exposure to Lean and design thinking and QI methods, that is itself a win.”
Category 3: Project alignment with health system priorities and processes
Indicator 3.1 Alignment with health system or microsystem goals: From our analysis of all three data sources, we found many signs of project alignment with health system priorities. We used Institute of Medicine (IOM) priorities [
34] as all three health systems embraced these. All projects addressed one or more IOM priority area; improving the effectiveness of care was most common (Tab.
1). Most projects also addressed a national and/or local priority (77%). Nearly half referenced a published benchmark (45%). Addressing a healthcare disparity was an explicit goal in 15% of projects.
Indicator 3.2 Involvement of key stakeholders: Analysis of project reports and posters showed that all student teams engaged clinicians from multiple health professions in one or more core improvement step. Professions included nursing, pharmacy, physical therapy, psychology, social work, and data science as well as administrative and research personnel. Twenty-three projects (43%) involved direct interactions between students and patients or families to understand the gap or to design, implement, or measure the intervention. Projects that did not involve direct interaction with patients focused on interventions for clinicians (e.g., note templates or clinician training). These findings suggest success in engaging key stakeholders from clinical microsystems in students’ HSI projects.
Indicator 3.3 Use of health system improvement tools and interventions: We found that projects used tools familiar to the health systems in which the projects occurred. Most projects (87%) used fishbone diagrams to represent their gap analysis. Others used process maps, Possible-Implement-Challenge-Kill (PICK) charts, and five why’s [
35,
36]
. This finding suggests alignment between the language, concepts and tools students learned in the formal medical school curriculum and the language, concepts and tools used in the clinical microsystems. Projects often included more than one intervention (74%) reflecting sensitivity to the need for multi-pronged approaches. Almost all projects (89%) attempted to provide in-process support (e.g. team huddles, or electronic health record templates, alerts, or checklists) that could enhance existing processes in the microsystem. Other common interventions included training and education for healthcare professionals, staff, patients, and/or family members (72%) and audit and feedback (26%).
Indicator 3.4 Reasonable scope for implementation: Most projects (79%) fully implemented at least one intended intervention within the timeframe of the curriculum and 19% partially implemented at least one intervention. All but one project completed at least one PDSA cycle. The average duration of the intervention period was five months. These findings suggest that projects could be designed and managed to support completion of specific goals and activities within the expected 15 month period.