Assessment of final-year medical students’ entrustable professional activities after education on an interprofessional training ward: A case-control study

With increasing complexity in health care, health professionals need to acquire appropriate competencies to work efficiently and collaboratively in interprofessional teams [1]. These developments require competency-based interprofessional education (IPE) [2], which has been integrated in undergraduate and postgraduate educational frameworks [3, 4]. In a systematic review, positive outcomes of IPE with regard to learners’ satisfaction, their perceptions of other health care professions, and an increase in their collaborative knowledge and skills could be documented [5]. Many countries have established interprofessional training wards (ITWs) for IPE, where learners from more than one profession are responsible for comprehensive patient care [6‐8]. Positive outcomes of ITWs with respect to learners’ professional role perception within a health care team and a positive impact on interprofessional collaboration have been described [9]. Additionally, patients’ perceived quality of care on ITWs was high [10] and patient outcomes did not differ between ITWs and usual wards regarding patients’ readmission rates and one-year mortality [11]. ITWs provide excellent opportunities for medical students to be immersed in professional medical activities and to acquire competencies like ‘Responsibility’, ‘Teamwork and collegiality’, ‘Verbal communication with supervisors’, and ‘Knowing and maintaining personal bounds and possibilities’, which are needed for interprofessional care, but also resemble important basic competencies for beginning residents [12, 13] that are relevant for entrustment decisions [14, 15]. However, little is known about medical students’ general performance in competency-based assessments after ITW training.

One possibility to assess medical competencies more holistically is the concept of entrustable professional activities (EPAs) [16]. EPAs integrate different competency domains and provide a construct for training outcomes. Recommendations for precise descriptions of professional activities for EPA-based curricula have been suggested [17] and development processes for EPAs with good content validity and assessment strategies have been described [18, 19]. When used as an assessment framework, EPAs are proposed to best occur in workplace-based situations [20, 21]. Assessment of EPAs was also performed in simulated, experiential situations [22, 23] and can be facilitated by using mobile devices in daily routine [24]. Another approach to assess medical students’ competencies by EPAs was used in competency-based simulations of a first day of residency [14, 25]. Supervisors, who had observed the participating students during a simulated handover of simulated patients before, made prospective entrustment decisions for EPAs, which were unrelated to the content of the simulation [15, 26]. These EPAs were based on basic medical competences relevant for beginning residents [12, 13]. When senior physicians were asked to make prospective entrustment decisions for EPAs based on either watching a 1-minute video sequence of a supervisor greeting a participant at the beginning of the simulation or watching a 1-minute video sequence from the handover of the same participant, no significant differences in EPA assessment were found [27].

In 2020, an ITW for interprofessional education of medical students and nursing trainees was opened at our Medical Center. On this ward, final-year medical students and nursing trainees provide comprehensive patient care for internal medicine patients and are supervised by facilitators of both professions. Basic medical competencies, e.g., ‘Responsibility’, ‘Teamwork and collegiality’, and ‘Verbal communication with supervisors’ [12, 13] were integral part of the ITW’s learning objectives. Communication and interpersonal skills with respect to taking patients’ histories were not part of the ITW curriculum’s specific learning objectives, because communication with patients during history taking is a required skill for all students when they enter the final year of medical school. The aim of this case-control study was to evaluate final-year medical ITW students’ versus control students’ communicative and interpersonal skills by simulated patients as well as prospective EPAs by senior physicians in a telemedicine simulation of a first day of residency. Our hypotheses were: 1) all final-year students communicate equally well with simulated patients, because communication with patients during history taking is not a specific learning objective for final-year students and 2) ITW students will receive higher entrustment scores in prospective EPAs which are based on basic competencies, because basic competencies are a specific part of the ITW curriculum’s learning objectives.

A total of 32 students participated in the competency-based assessment in March 2021. The mean age was 27.16 ± 3.09 years and 18 students were female (56.3%). Sixteen students attended the ITW and 16 students completed their internal medicine rotation without training on an ITW. There was no significant difference between the groups in terms of the students’ age (ITW: 26.94 ± 3.32; control: 27.38 ± 2.94, t(30) = 0.40, p = 0.696) or gender (ITW: 7 females; control: 11 females; χ²(1) = 2.03, p = 0.154). The assessment of communicative and interpersonal skills by the simulated patients resulted in an overall ComCare mean score of 4.10 ± 0.47 for the entire study group. No significant differences were found between the ITW and the control group for any of the ComCare index items (Table in supplementary material).

The overall entrustment decisions for all participants and the ICC results of the three raters for each EPA are shown in Tab. 1. The highest mean EPA score was reached for EPA 6, ‘Solving a management problem’ (3.21 ± 0.70), the lowest for EPA 12, ‘Acting according to patient’s will’ (1.76 ± 0.62). A good to excellent intraclass correlation between the raters was demonstrated for all 12 EPAs, ranging from 0.77, 95% CI [0.59, 0.88] (EPA 2, ‘Handling a patient’s complaint’) to 0.96, 95% CI [0.92, 0.98] (EPA 8, ‘Handling a critically ill patient’).

The mean entrustment decision over 12 EPAs, based on video-recorded medical students’ case presentation performance, showed a significantly higher performance of ITW students versus control students (ITW: median = 3.15 vs. control: median = 2.22; U = 33.0, Z = 3.583, p < 0.001, r = 0.63) (Tab. 2). Raters entrusted ITW students at a significantly higher level in 10 out of 12 EPAs compared to control students. For EPA 5, ‘Clinical reasoning under time pressure’, and EPA 8, ‘Handling a critically ill patient’, no significant differences were seen between the groups. The greatest differences of students’ entrustment levels (∆ means ≈ 1 point on the entrustment scale) were found for EPA 6, ‘Solving a management problem’, EPA 9, ‘Interaction with a consultant’, and EPA 10, ‘Presentation of an oncology patient in a tumor board meeting’, with large effect sizes.

While we found no differences between the two groups of final-year students with regard to their communicative and interpersonal skills during history taking, students from the ITW group received significantly higher entrustment scores for 10 of the 12 EPAs than those of the control group. The largest differences and effects were found for EPA 9, ‘Interaction with a consultant’, and EPA 10, ‘Presentation of an oncology patient in a tumor board meeting’, which have been shown to correlate well with the competency facet ‘Verbal communication with colleagues and supervisors’ [15]. This competency was ranked highly when physician educators were asked to assess the relevance of different competencies for beginning residents [12]. When final-year medical students in Germany were asked to self-assess their competency with respect to ten competencies relevant for beginning residents, they rated their competence level for this facet on average the second lowest [33]. We assume that the students in our study, who were placed on the ITW, gained a lot of experience in verbal communication with colleagues and supervisors because presentation of their patients to the team was an explicit part of their daily work routine on the ITW and could have led to higher entrustment levels in the above-mentioned EPAs.

The competency ‘Responsibility’ was represented in ten, the competency ‘Structure, work planning and priorities’ in eleven of the twelve EPAs we used in our prospective assessment [15]. Responsibility has been found to be a key ingredient of reliability that clinical supervisors find important when making entrustment decisions [34]. Additionally, responsibility was found to be the competency facet with the highest priority for residency training as defined by clinical supervisors from different medical schools [13] and by medical students themselves [35]. Providing learners with opportunities to practice taking responsibility was suggested for the implementation of EPAs for assessment [36]. Such opportunities were presented on a daily basis to the students from our ITW group while caring for their patients on the ITW. The curriculum of the students from the control group did not provide such possibilities and therefore might have led to lower EPA scores. Better EPA scores in the ITW group could be related to the competency ‘Structure, work planning and priorities’. Its performance level is rated as one of the lowest in final-year students’ self-assessment [33], while it is regarded to be highly important for residency by physicians [13] and medical students [35]. Students on our ITW are constantly in the position to prioritize and structure their work according to the demands of the patients’ situation. This could have led to a good structure in the patient handovers, which represented the basis for the assessors’ prospective entrustment decisions.

The communicative and interpersonal skills, measured by the simulated patients in our study with the ComCare index after history taking [31], did not significantly differ between the ITW and the control group. Communication competency with patients is gained longitudinally at our medical school [37] according to recently published guidelines [38]. It is not a particular learning focus on our ITW, which could be a reason for not finding any differences. Furthermore, the results for the prospective EPAs 5, ‘Clinical reasoning under time pressure’, and 8, ‘Handling a critically ill patient’, showed very low overall entrustment scores and also did not significantly differ between ITW and control group. Both EPAs require entrustment in time-critical performance that could compromise patient safety if not delivered. This is neither taught in general in our curriculum nor was it taught on the ITW, where facilitators took over in critical situations, which also underscores our hypothesis that the differences in entrustment scores we discovered could be related to specific competencies taught on the ITW.

However, this finding could also be due to the raters using the prospective assessment scale incorrectly [39] with the observed handover performance in mind and the knowledge that timely reactions to critically ill patients are not a learning objective in the final year of medical students’ training. This would be a limitation to this study. It has been shown that prospective rating scales bear challenges to the raters because they have to deduce entrustment, for instance, for situations they did not observe [40]. Even though our raters received a frame of reference training for prospective assessment [14], this bias could have occurred. Another limitation with respect to the EPA rating is that one of the three raters was not fully blinded to the ITW participation status of the participating students. However, good interrater reliability between the three independent raters was reached. This study was performed at only one medical school and the study sample was small, which limits the generalizability of our results. Furthermore, the control group participants were chosen on a first come, first served basis. This could have led to a self-selection bias of very skilled or motivated students. Despite this limitation, differences in entrustment decisions were large between the two groups. A strength of our study is that we used a control group design to assess outcomes of final-year medical students after working on an ITW, whereas other studies used ITW groups only [41, 42]. Another strength is that we used rater-based assessment for competency measurements performed by senior physicians and simulated patients rather than self-reports on competencies by medical students who participated in ITW training [43‐45].

In conclusion, final-year medical students in our study who were trained on an ITW showed similar communicative and interpersonal skills in simulated patient interviews compared to a control group without ITW training. The ITW students received significantly higher entrustment scores in assessment of prospective EPAs based on raters’ observations of a simulated patient handover. To test our hypothesis that this difference in entrustment is related to basic medical competencies that are taught on the ITW as part of the interprofessional training, assessment of the respective competencies or core EPAs which include these competencies could be performed by direct workplace or simulation-based observation.