The role of training in student examiner rating performance in a student-led mock OSCE

Peer assessment in medical education has become more prevalent and popular in recent years. Traditionally, this has been used to gather more quantitative information on trainee’s performance [1]. The benefits of peer assessment are both to the student being assessed, who is given more opportunities to practice, and to the assessor, who is able to learn through teaching in a process known as assessment for learning (AfL) [2]. In addition, having a larger pool of examiners for formative assessments may alleviate faculty teaching burden in an environment where there is an increasing medical student population and limited teaching resources. As peer assessment becomes more commonplace, an emerging area of interest involves the role of student examiners in Objective Structured Clinical Examinations (OSCEs).

A 2014 systematic review revealed that one of the more well-studied modalities of peer assessment is in student-led OSCEs [3]. Mock OSCEs are simulations of clinical examinations involving physical examination or history taking with standardised patients and senior students as examiners. Mock OSCEs have been found to be reasonable alternatives in providing additional practice for medical students while also allowing them more opportunities for early intervention [4‐7]. Various topics such as history taking, physical examination and patient communication are commonly tested. During these mock OSCEs, examiners typically assign checklist ratings (CR) and global ratings (GR) to participants [8‐10]. The CR assess one aspect of a participant’s behaviour in the OSCE while GR describe the overall impression of a participant’s performance.

Previous studies comparing year‑4 (Y4) or year‑5 (Y5) medical students and faculty examiners showed that senior medical students are comparable to doctors as OSCE examiners [11, 12]. Particularly for GR, peer examiners typically have moderate to high agreement and positive correlations with faculty examiners [13]. In terms of CR, the reliability of peer examiners is less clear. Peer examiners may award lower, higher or similar CR depending on the station [13]. For instance, some studies suggest that junior examiners with less experience as examiners (e.g. medical students or junior doctors) are more lenient [14, 15]. Others suggest that they are stricter [16], and some found no significant difference [17]. Despite efforts to standardise features of the OSCE to make it more objective, examiner marking behaviour remains a significant source of variation in scores [18, 19].

It has been suggested that better OSCE assessment is likely if medical students undergo training before taking on the role of examiners [14], and that examiner training should be directed at both teaching skills and OSCE-specific assessment skills [20]. In addition, evaluation should be carried out to determine the type and amount of training necessary for each specific OSCE [21]. Van der Vleuten recommended that in order for peer assessment to be effective for learning, it should be approached as a complex professional skill that requires intensive training [1]. In a study on faculty examiners, those who had undergone training performed more consistently in their rating of students in an OSCE when compared to untrained faculty examiners [22]. Despite this evidence, there exists a critical gap in existing literature, with few peer examiners in previous studies having undergone training prior to assessment.

With this gap in knowledge, it would be of interest to compare untrained and trained peer examiners in their performance as peer assessors in a mock OSCE. In this study, we examine whether junior year‑3 (Y3) student examiners are as effective as senior year‑5 (Y5) student examiners in performing as peer examiners and whether undergoing training will improve the marking behaviour of the Y3 student examiners compared to their seniors. We hypothesise that:

A total of 104 Y2 candidates took part in the mock OSCE, with 108 and 66 attempts for Hx and PE stations, respectively.

Previous studies have established the role of medical students as OSCE examiners [12]. This study was designed to compare the performance of junior peer examiners with more senior peer examiners and to see if training would further improve their performance. Our findings showed that overall correlations between Y3 and Y5 student examiners were moderate to excellent in both the PE and Hx stations. When broken down into domains, correlation between Y3 and Y5 student examiners was generally higher in CR domains compared to GR domains across both PE and Hx stations. The effect of training on correlation between Y3 and Y5 student examiners was mixed, with improvement in correlation only in PE stations but not in Hx stations. Comparison of mean scores showed generally small effect sizes between examiners, and all were smaller than the minimum detectible effect sizes that this study was powered to detect.

The high correlations seen between the Y3 and Y5 student examiners support our hypothesis that Y3 student examiners are as effective as their seniors when grading a mock OSCE for more junior Y2 students. This builds upon previous studies which have shown that senior medical student examiners are equivalent to faculty examiners in OSCEs that test basic medical skills [11]. Of note, correlations seen in this study were higher than those of previous studies [13]. One explanation is that most previous studies looked at student-faculty correlations, and it is possible that the use of junior and senior peer examiners in this study resulted in higher student-student correlations.

When comparing the subtypes of OSCE stations, PE stations had higher correlations compared to Hx stations and this has been observed in other studies as well. Physical examination stations may be less susceptible to measurement error [15] because the steps are prescriptive and techniques are well documented [25]. There is therefore little room for examiners’ subjective interpretation when assessing a candidate’s PE performance. Essentially candidates who perform all the necessary steps in the PE station would receive good scores even if examiners were less experienced. This is especially so in a mock OSCE where candidates examine standardised patients who do not have abnormal findings and are not assessed on their detection and interpretation of findings.

The overall correlations for the Hx stations are lower than PE stations. This is in keeping with the findings from a previous study whereby history taking and clinical communications are more susceptible to examiners’ interpretation [15]. Effective communication for information gathering is closely related to personal style and may be affected by examiners’ preferred style. Nonetheless, the Y3s appeared to be effective examiners as evidenced by moderate–excellent correlations across all domains.

For both Hx and PE stations, the correlation between Y3 and Y5 student examiners was the poorest for GR regardless of examiner training. Previous studies also found relatively poor agreement between novice examiners and trained examiners on global pass/fail decisions [26]. It is suggested that novice examiners may have more difficulty in assigning accurate global scores due to lack of experience and therefore have poorer judgment of a candidate’s overall performance [27, 28]. In addition, the mean global scores given by Y3s were higher than corresponding Y5s in all groups except for untrained Y3s grading the PE stations. This is a finding seen in previous studies as well, where peer examiners typically grade higher in GR domains [13, 29].

The effect of training on the effectiveness of Y3 student examiners was less clear.

For PE stations, trained Y3s had higher correlations with Y5s compared with the untrained Y3s. However, correlations between trained Y3s and Y5s in the Hx stations were lower across all domains. In addition, the effect sizes of differences in mean scores in both groups A and B were smaller than the minimum detectible effect. This presented a limitation in the analysis of comparison between Y3 and Y5 student examiners. Due to the limited sample sizes and low effect sizes, the authors were unable to definitively conclude that training resulted in Y3 student examiners grading more similarly to Y5 student examiners.

Nonetheless, current evidence seems to suggest that training positively impacts examiner behaviour in various domains. Improved correlations between trained Y3 and Y5 student examiners in the PE stations seem to suggest that training junior students makes them more effective examiners. This could possibly be due to the more prescriptive nature of examining PE stations, which requires looking out for certain steps that are performed and relies less on clinical judgement.

Effects of training on scores in the Hx stations were more mixed. Lower correlation in the content domains could be due to lack of focus of the training programme on content of the individual Hx stations, and more on general examiner skills. Future studies could consider implementing training which is more specific to the content of individual exam stations.

Although our study did not evaluate the effect of examiner training on assessing non-verbal communication, previous authors found that non-verbal expression may be the domain most impacted by training [30]. This might be particularly relevant to OSCE stations such as counselling on breaking bad news, where both verbal and non-verbal communication play key roles.

This study shows that in an OSCE testing basic medical skills, junior Y3 student examiners are effective substitutes for more senior Y5 student examiners. Scores in CR domains were comparable between Y3s and Y5s, while Y3s typically gave higher GR scores than Y5s. The effect of training remains unclear from this study, but likely results in juniors marking more similarly to senior student examiners especially in physical examination stations. Future work should include dedicated training sessions for peer examiners in addition to same-day examiner calibration in order to better evaluate the impact of training peer examiners.