As we hope is clear in the introduction to this paper, inequalities and minoritization occur both inside higher education practices and within the wider societies in which medical schools are located. The complexity of institutional (e.g., assessment and teaching approaches) and societal practices (e.g., poverty and longitudinal educational opportunity), norms, privileges and biases that contribute to differential attainment make it challenging to understand the fundamental causes of differential attainment and what actions will effectively redress it. Nevertheless, based on analysis and collation of the literature, we bring our interpretation and understanding of pedagogical practices which seem to contribute and may need to be addressed in relation to differential attainment in minoritized medical students to the guidelines.
Ensure assessments are fair and defensible
Fairness is fundamental to good assessment yet remains elusive to define [58
]. We define fair assessments as those which offer learners similar or equitable opportunity to demonstrate their knowledge, understanding and competence with minimal impact from external factors, such as gender, race or ethnicity. Such external factors might be mediated by the test question context or assumptions in the case of written exams, or by examiner bias (implicit or otherwise) in the case of performance exams. Importantly, fairness does not mean or imply that assessments need to pose a similar challenge to every student; differential knowledge and understanding is a legitimate discriminator [58
]. In fair assessments, however, it will be the only significant discriminator.
Proactively ensuring valid and defensible assessments which minimize the impact of extraneous (or “construct-irrelevant”) factors on student performance by paying attention to the basic elements of assessment will be helpful in respect of fairness. Such factors traditionally include design, question writing, peer review, test construction, appropriate scoring and post hoc analysis [59
], as well as blueprinting, adequate sampling, careful planning and a programme-level focus [60
]. But each of these aspects of assessment carries certain epistemological assumptions and constructed notions which should also be interrogated in determining how fair assessments are, in the same way that discourses of ‘merit’ involve far-reaching assumptions and potential biases [61
]. We propose that examination content and underlying assumptions must be routinely analyzed and reviewed to ensure they do not present inappropriate progression barriers for any group.
Use robust and recognized analytic approaches to scrutinize assessments and assessment data (Strong)
Assessment data should be routinely analyzed to identify differential attainment. Item response theory (IRT) can provide information about items (e.g., exam questions) which function differently across subgroups [62
]. One IRT approach, Differential Item Functioning (DIF), can be used to see if certain questions in a written or objective structured clinical examination (OSCE)-style examination explain group-level differences in performance [65
]. Where test questions discriminate between groups, these items can be revised or removed, and replaced with less biased questions. Recent studies have indicated that another IRT approach, Many Faceted Rasch Modelling (MFRM), can also be used to identify sources of error (e.g., examiner, domain and station) which may influence the student outcome [67
]. These statistical approaches are used post hoc
, after an assessment, and may offer opportunities to moderate marking. Whether this is possible or not will depend on the wider context. In some countries, information on personal characteristics (e.g., ethnicity, gender) is collected and protected but can be accessed for legitimate purposes which adhere to ethical and governance processes. In other countries, the legal stance is that information on ethnicity and race cannot be collected, thus meaning this type of research is not possible.
Sample broadly across assessment types and assessors to minimize bias
There is ample evidence that assessor judgements involve a degree of subjectivity particularly in clinical assessments [68
], and examiner variability is a significant source of variability in OSCE scoring [71
]. Many (more positivist) studies have raised the possibility that examiner bias may be a contributing factor to minoritized students performing poorly on assessments [75
]. However, the outcomes of studies looking at examiner bias in medical education are inconclusive. A study in one UK medical school found that student ethnicity did not influence examiners’ scores or feedback [78
]. Other studies have suggested that conscious or unconscious assessor biases can impact on the assessment process [79
], with one study concluding that “we cannot confidently exclude bias from the examiners in the way that they assessed non-white candidates” [81
Alternative literature has explored the opposing idea that subjectivity should not and cannot be removed from assessment [82
]. Such literature acknowledges and accepts examiner variability as not only unavoidable, but also as a potential source of greater defensibility and validity of assessment judgements [84
This perspective promotes the notion of broad sampling in assessment as a guiding principle in assessment systems. Broad sampling involves the conscious use of multiple assessments utilizing different assessors and on multiple occasions. This approach to assessment is a key cornerstone of programmatic assessment [83
], and has been shown in one study to reduce ‘ethnicity-related differences in grades’ [81
]. While broad sampling cannot be expected to fully reduce any latent examiner bias, the use of multiple assessment moments may also expose discrepancies stemming from those judgements about students’ attainment [80
]. Interestingly, we could not find any literature which directly investigated whether “broad sampling” of assessors should deliberately include “diverse assessors.” However, drawing on the literature more broadly indicates the need to do so [85
Include the concept of ‘cultural validity’ in the design, development and review of assessments
This recommendation derives from wider literature which explores the impact of social background on academic attainment, with particular attention to the representation (or not) of minoritized students in assessment activities, and the way minoritized students view and interpret assessment tasks [86
]. This guideline is related to Guideline 5 sharing the position that the nature of the students’ experience of the curriculum is central to their learning and attainment but focusing specifically on the nature of the assessment tasks themselves.
The concepts of culturally valid and responsive assessment [87
] challenge assumptions that a test designed for and implemented under standardized conditions is inherently fair or appropriate for all learners [89
]. Instead, (written and clinical) assessments should reflect the diversity of the test-taking population, minimizing the risk that students might feel excluded or alienated. This can be done by: considering the ethnic characteristics of people represented in assessments and whether there is any stereotyping of minoritized groups (unintended or otherwise); simulated patient and scenario ethnicity; the positioning of minoritized people in relation to majority members; and the inclusivity of the language used [59
Minoritized students may have different experiences and different ways of viewing the world, or epistemologies, which impact on how they approach learning and respond to assessments [91
]. It is not acceptable to assume that the responsibility lies with minoritized students to exclusively “accommodate” their ways of knowing to assumed “normative” assessment practices [89
]. Accordingly, assessment review practices need to consider how items may be viewed from the minoritized perspective, both affectively and cognitively. Involving minoritized students and faculty in assessment planning and review may be one approach to address this issue [89
] (but please see Guideline 9 for a caveat).
Recognize unequal privilege and power in the learning environment as causes of differential attainment and take steps to mitigate their negative impacts
There is a strong and consistent body of literature describing discrimination faced by racially-minoritized medical students. This racism occurs at personal and structural levels.
At a personal level racism is experienced in the form of microaggressions [95
], stereotype threat [97
] and harassment from peers, patients and faculty [98
]. At a structural level, racism operates through biased curricula [99
], and knowledge production [3
] in which Eurocentric knowledge and practices are valued above others.
Given the discrimination that minoritized students face, we must consider personal and institutional racism within the learning environment as causes of differential attainment. Indeed, a review of 28 studies concluded that minoritized students reported less positive learning environments and were more likely to experience racial harassment compared with non-minoritized students. Across these studies academic achievement was worse and academic progress slower for minoritized students [50
]. More recent studies indicate that things have not changed notably in the 10 years since Orom and colleagues completed their review [95
What can we do differently in the future to lead to improvements? There are some studies exploring what might make a difference. Being part of an established social network and building relationships with faculty are correlated with (positive) medical student experiences and outcomes [101
]. Having access to both formal and informal social networks are important building blocks for creating social capital [49
] and a sense of belonging [103
], a lack of which is also associated with poor achievement and discontinuation of studies [105
Medical schools must take responsibility for their role in creating and perpetuating differential attainment
Inequality in the broader societal context can impact on attainment within medical education. Minoritized students often carry a burden of historical disadvantage with them when they enter medical school [41
]. Universities may further exacerbate this disadvantage through discriminatory systems and practices [48
One example of this is “deficit thinking”, where there is an implicit or explicit assumption that minoritized students are in some way deficient and institutions hold minoritized students responsible for the inequalities and challenges they experience [2
]. As a starting point to addressing differential attainment, medical schools must seek to recognize and revise systemic deficit thinking, as this will allow them to become aware of and take responsibility for factors in their curricula, policies, learning environments and physical environments that contribute to discrimination and differential attainment [4
]. There is, however, little practical guidance on how to do so—we hope these guidelines will go some way to helping medical schools reflect on their practices.
Recruit and promote under-represented faculty
A diverse faculty can contribute to the creation of equitable inclusive structures and practices that will support equitable student outcomes [42
]; bring diverse perspectives to curricula and assessments [48
]; and provide representative role models.
The importance of learning from role models has long been recognized within medical education [107
]. Role models are influential in terms of career and specialty choice [110
], developing clinical knowledge and skills, as well as understanding the culture of medicine [49
]. Students identify more strongly with role models who are similar to them in some way [112
] and there is value in having role models from the same ethnic background as students [114
]. However, diversity of ethnic backgrounds is often under-represented in faculty, especially in senior positions, giving rise to a paucity of role models for medical students [98
]. We suspect, but do not know, that the lack of representation and role models within the broader learning environment may impact on the learning and performance of minoritized students [78
Importantly, research shows that under-represented faculty also experience bias and discrimination [123
], and greater difficulty in achieving promotion and advancement [125
]. They are also more likely to feel isolated and less satisfied with their career, professional development and networking opportunities [123
]. Medical schools must consider how best to support under-represented faculty via, for example, support from senior leaders [127
], peer networking [127
], professional skills development [132
] and mentoring programmes [112
]. Campbell et al. [135
] point out that faculty development initiatives for under-represented faculty must be complemented by initiatives that address inequalities in opportunities and recognition and foster an inclusive culture and environment [126
]. Price et al. [131
] advocate that institutions undertake a formal assessment of their ‘diversity climate’ in order to better understand and then inform organizational changes.
Don’t assume that minoritized students will have the same exposure to and familiarity with particular assessment methods as their non-minoritized peers
This guideline builds on the concept of sampling broadly across assessment types (Guideline 3) and specifically recognizes that educators cannot assume that minoritized students will have had the same exposure to and experience of particular assessment methods as their non-minoritized peers [41
]. Differences in prior experience may impact on actual performance, and therefore may contribute to relative underperformance compared with students who are familiar with those methods [80
]. Greater use of formative assessments for minoritized students, to both improve familiarity with specific assessment formats and, arguably even more importantly, to alter minoritized students’ sense of agency with particular assessment systems and their own role in the learning process, may be helpful [87
Don’t implement ‘diversity’ practices which place unacknowledged and unrewarded burdens on minoritized faculty and students (Moderate)
Under-represented faculty (i.e., faculty from minoritized groups) are often called upon to work on diversity initiatives, mentor minoritized students and applicants, and serve on equity and diversity committees [132
]. Such activities are often not valued in terms of promotion, and time spent on such activities takes faculty away from pursuits which are more beneficial in terms of individual career progression [138
]. This often-unacknowledged burden has been described as a ‘minority tax’ [128
]. Minoritized medical students can also experience minority tax, feeling pressure to take part in activities such as mentoring or outreach activities for minoritized students [117
]. It is challenging to balance engagement with minority taxation but, as per Guideline 6, change at a systems level is required so these activities are appropriately recognized and rewarded.
Other faculty can also address some of this unbalanced workload by challenging discrimination, supporting under-represented faculty and actively participating in initiatives to address lack of diversity amongst faculty and students [140
]. To contribute effectively in this way, faculty development could usefully include: understanding different forms of discrimination such as implicit bias and microaggressions [135
]; learning how to challenge racism and examining everyday practices that reinforce existing power structures [103
]; and mentoring minoritized students [137
], engaging in critical reflection [142
] and recognizing when to refer students to others [112
Don’t attribute to the individual what is systematic
As per Guideline 7, increasingly, differential attainment is recognized as a systemic issue and not an individual one [2
]. That prolonged disadvantage is an issue is unarguable. Yet historically, efforts at redress have too often focused almost exclusively on interventions that target the individual learner. For example, many schools have responded to this academic achievement divide between minoritized and non-minoritized students by establishing premedical school “enrichment”, gateway or pipeline programmes, to bridge the gap for minoritized students [143
]. (Consideration of the assumptions of such programmes and how they may perpetuate deficit models is beyond the remit of this paper). However, as we hope is obvious so far, the past few decades have moved this conversation on to recognize that minoritization within medicine is itself an independent predictor of exam success. Therefore, systematic and structural changes need to take place, rather than merely focusing on individual student success. The terms differential awarding instead of differential attainment, and awarding gap rather than attainment gap have been proposed recently as a way to shift the focus more toward institutional responsibility [21
Are supportive interventions for specific groups effective in respect of assessment outcomes?
Some educators and researchers argue for targeted interventions for specific groups to address differential attainment [79
]. This is an approach which is relatively underutilized in medical education and training (see [144
] for an exception) but is reported more extensively in the wider education literature. For example, studies have examined the efficacy of self-affirmation interventions in relation to assessment performance in ethnic minority groups [145
]. Another approach reported as effective in terms of addressing differential attainment is that of enhancing goal-directed conceptualization and action [147
]. This approach aims to support minoritized students in challenging learning environments to enhance their motivation and self-regulation [148
]. Again, however, while promising and encouragingly well-grounded in theoretical principles, neither self-affirmation nor goal-directed approaches/interventions have been tested empirically in minoritized medical students.
More generally, the onus is on medical schools to scrutinize routine assessment and other data to develop targeted strategies in areas of particular concern [149
]. Extrapolating from Cleland et al. [150
], it is then crucial to evaluate the impact of additional support for specific groups of students to identify what works, from whom and why in terms of assessment support.
To what extent do professionalism norms and assessments of professionalism discriminate against minoritized students?
As described earlier, there is growing evidence that cultural norms and values are embedded in assessment design and practice. If these reflect only the norms of the privileged group, this can result in biased assessments that lack cultural validity and may contribute to differential attainment. In particular, the assessment of professionalism in examinations is likely to reflect culturally embedded values about the practice of medicine [121
]. For example, medical educators may have a white Eurocentric view of what clinical communication looks like or how empathy is expressed [116
Although ethnicity is acknowledged to be a part of one’s personal identity [151
], the understanding of the influence of race and ethnicity on the process of professional identity formation (PIF) and its impact on the assessment of professionalism is largely absent from published literature [141
]. The wider literature suggests areas for further research. For example, socialization is critical in PIF [155
]. Given minoritized students have less extensive peer and faculty networks (see Guidelines 6 and 8), it would be worth exploring if and how these may impact on their PIF and progression from student to physician. In trying to make sense of their race or ethnicity and becoming a member of an apparently “white” profession, minoritized students may struggle to see how they can conform to this stereotyped image of a doctor [156
] and may conceal or repress aspects of their racial or ethnic identity [138
] and/or experience tensions in trying to reconcile these and a new professional identity [159
]. Conversely, if dominant professional norms are rejected in this process of PIF, this could have an impact on attainment and progression through medical education and training. Direct evidence is lacking, but in many countries doctors from minority groups tend to receive disproportionately more sanctions or warnings than those identifying as from the dominant group [160
]. Retrospective and prospective studies which examine patterns of performance and explore underlying reasons for differential attainment are needed.
Research is also needed to understand how aspects of student identity relate to experiences of minoritization within medical education [162
], and how this in turn relates to differential attainment.
How can formal curricula influence differential attainment?
Research from higher education posits that curricula are important to redressing differential attainment [48
]. Inclusive pedagogies and curricula are theorized to combat the centrality of whiteness and thus experiences of minoritization within higher education [3
]. Two particular approaches are being implemented within medical education: inclusive curricula and removing racism from curricula.
Inclusive curriculum strategies aim to make curricula accessible and acceptable for all students [4
]. These include “decolonizing” curricula by bringing different voices and knowledge sources into courses (epistemic pluralism) [4
]. Some medical schools are actively undertaking work to ensure previously unheard voices (voices that legitimize issues of gender, race and class) within their curricula and to address issues such as ethnic biases in cases and teaching materials [34
]. Future research could examine the processes of strategy implementation [168
] and strategy impact on the learning environment, student experience and differential attainment.
Throughout their curricula, medical schools teach directly and indirectly about race, ethnicity and health. Race-based medicine positions race as a biological variable that influences physiological functioning and thus becomes a basis for differential clinical care [169
]. Biological conceptualizations of race, which are deeply embedded in medical education [99
], contribute to institutional racism impacting on student learning, and ultimately racial health inequalities [99
]. “Anti-racist” curricula shift teaching about race as a biological variable to that of a social construct, thus prompting explanations about health inequalities that focus on equity and discrimination rather than spurious genetic explanations [99
]. There is the potential for anti-racist curricula to re-frame conceptions and misconceptions of race and racism, that impact on values, cultures and norms [163
However, empirical evidence as to the effectiveness of curricular strategies to redress differential attainment is currently lacking.