Best practices for interviewing applicants for medical school admissions: a systematic review

Interventions and study populations varied between studies (ESM 2). Nine (69%) studies examined the MMI [24, 25, 27‐29, 31‐33], and nine (69%) used a comparative observational study design [9, 23, 25, 27‐30, 32, 33]. Eleven (85%) studies were conducted in high-income countries [9, 23‐31], and ten (77%) were based at public medical schools [9, 23‐25, 27, 28, 30, 32, 33].

Five main outcomes were identified: interview bias, reliability, feasibility, acceptability, and validity (ESM 3). All outcomes, including feasibility (costs), were determined independently by authors of included studies and reported in their manuscripts. Included studies had high risks of bias and high quality of research design (see Tab. 1); MERSQI scores ranged between 12.5 and 17 (maximum = 18) and Oxford scale scores ranged from 0–2 (maximum = 5). MERSQI scores above 12.5 and Oxford scale scores above three have been considered the cut-off for high-quality studies in the literature [21, 34, 35].

The MMI demonstrated little bias, good reliability, moderate costs, strong acceptability, and predictive validity for clinical and academic performance, although most findings were based on outcomes from a single study and thus should be interpreted carefully. The MMI produced similar results regardless of whether some students received summaries of interview questions two weeks prior to their interview [24] or the order in which students rotate through the MMI stations [31]. Station order also did not affect perceived difficulty or anxiety among students either [31]. Making an MMI station more challenging did not show bias based on gender or disadvantaged status and improved reliability (from G = 0.58–0.71) [25]. Additionally, four studies found that MMI ratings had good interview reliability [24, 25, 27‐29]. Cost savings could be achieved without sacrificing reliability or creating interviewer bias by reducing the number of MMI stations from twelve to nine (from $915 to $495 per student) and switching to a virtual MMI format (from $61,887 to $10,145 total) [28, 29]. The MMI displayed predictive validity for a first year of medical school year written examination and Objective Structured Clinical Examination (OSCE) scores, especially with more traditional interview stations rather than task-based ones [27]. Although the MMI displayed higher convergence with social self-perception than the semi-structured panel interview [32], the MMI did not show predictive validity for stress, anxiety, depression, or burnout [33].

The unstructured interviews studied in the four (31%) remaining studies had several issues, including potential bias, bad reliability, and low acceptability, although all but one of the outcomes were based on the results of a single study. Unstructured interview ratings were significantly influenced by gender, age, GPA, and MCAT scores [9, 30]. Blinding interviewers to GPA and MCAT scores reduced this bias without affecting reliability (from Cronbach’s α = 0.496–0.473) [9]. However, reliability was unacceptable for unstructured interviews [9]. Unstructured interview ratings were similar between panels of faculty members and mixed panels of students and faculty across six years [26]. Adding structure to an interview by developing standard interview questions, escorting students to meetings across campus, and planning interactions with current students improved student perceptions of the interview process, including its perceived usefulness and thoroughness [23].

This systematic review of controlled studies investigating methods of interviewing found that the MMI approach provided impartial, reliable ratings. Conducting the MMI virtually and asking more difficult questions at MMI stations increased its reliability in single studies. In addition, reducing the number of stations or using virtual interviews saved costs without impacting MMI reliability, although these two findings were based on data from two single-institution studies. Using unstructured interviews, blinding interviewers to academic scores reduced bias; in another study, student interviewers had similar levels of bias to faculty members. In anonymous surveys, applicants preferred structured over unstructured interviews.

All the included studies had high risks of bias based on the Oxford scale but were mostly rated as high quality by the MERSQI. A prior systematic review of simulation training in obstetrics and gynecology similarly found that MERSQI scores were overall higher than Oxford scale scores [36], possibly because the Oxford scale puts great weight on whether the method of randomization was appropriate and whether participants were masked (80% of total score) [22], whereas the MERSQI only asks whether randomization occurred (17% of total score) [21]. Randomization is challenging in interviewing studies as applicants may perceive certain types of interviews as more difficult [37], raising concerns about the fairness of admitting students based on separate interview tracks. Masking is also challenging as applicants may discuss interview experiences and schools often conduct interviews on the same day to conserve resources. Additionally, most RCTs in this review did not report on their method of randomization, leading to further deductions on the Oxford scale. Hence, the differences in Oxford and MERSQI ratings may reflect inherent challenges in medical education research [36] and insufficient transparency regarding methodology.

There were several shortcomings in the current evidence base for interviewing in medical education. First, there were few rigorously designed studies investigating interview techniques other than the MMI and unstructured interviews in medical education. All included studies had high risks of bias. Of our thirteen studies, nine focused exclusively on the MMI. Other common interview methods that were not studied included structured interviewing with hypothetical behavioral or experience-based questions. Second, although quality of evidence was above average for medical education, there were only four RCTs. Third, most studies failed to describe randomization or blinding in sufficient detail, increasing the difficulty of quality assessments [38, 39]. Fourth, many studies did not describe the development of interview questions, making their studies difficult to reproduce [40]. Fifth, few studies assessed key outcomes such as the interviewee’s perception of the interview process, the predictive validity of interview scores for future clinical performance in medical school, or applicant yields based on different interview types. Lastly, study heterogeneity impacted comparison of interventions and outcomes. Almost all studies created their own rating scales for interview performance and interviewee perceptions, which made comparison of interventions from different trials unreliable. Some studies trained interviewers, which may have improved their interrater reliability. Trained students interviewed applicants in several studies; faculty members conducted the interviews in other studies.

Our systematic review was restricted to comparative studies and RCTs published in English; nine of the thirteen included studies were conducted in English-speaking countries. Also, the review focused only on interview types, omitting related topics such as the financial cost of attending interviews and interview weighting in the admission process [2, 41]. In addition, as our studies were focused on medical school admissions, our findings may be less generalizable to the graduate medical education (GME) setting. Studies were conducted mostly in high-income countries and public medical schools; variations in medical school applicant populations may reduce generalizability.

Admissions interviews are used in almost all medical schools despite the weak evidence base [1]. Due to the coronavirus disease 2019 (COVID-19) pandemic, interview procedures are being modified to include virtual interviews [42]. As medical schools revise their interview processes, they should ensure that applicants are aware of the intended purpose of the selection interview [3‐8]. The classical test theory, which holds that an applicant’s observed interview performance may not accurately reflect their true personal characteristics due to interview day deviations (e.g., bad weather, difficult circumstances, etc.) [43, 44], can be applied to decide on the weighting of interview scores, examination results, and GPA [3]. Therefore, medical schools should consider reducing the weight of the admissions interview given the lack of strong, multi-study evidence indicating its effectiveness.

The interview of choice is the MMI, which limited bias against applicants, was resistant to question leaks, and had higher reliability when transitioned to a virtual format. One reason may be that most MMI studies trained their interviewers. Another may be that MMI interventions integrated several types of interviewing, such as behavioral, situational, and unstructured interviewing [45]. The MMI’s strong prediction of medical school clinical and academic assessments relative to other admission tools (UK Clinical Aptitude Test, Universities and Colleges Admissions Service) suggests that it holds predictive validity as an interviewing method in medical education.

Previous observational studies on MMI have raised questions about potential bias against applicants who do not speak English as their first language [8], lower reliability with fewer MMI stations [46], and preference for extroverted applicants [47]. However, many of these issues have been identified in traditional interview systems as well [14]. Given the scarcity of RCTs and high-quality studies examining medical school admission interviews, more research is necessary.

We used our findings to construct a logic model (Fig. 2) to develop an admissions interview system and to rigorously evaluate its validity. This conceptual framework highlighted the resources and activities needed to develop and conduct interviews as well as the direct outputs (e.g., admission decisions) and long-term outcomes (e.g., changes in student body).

The key inputs in the model were interviewer time, faculty and staff salaries, interview development, and facilities. The model recommended that personnel and professional psychologists be recruited to help develop structured, behavioral interview questions. It also recommended that the interviewing mode be standardized to mitigate potential differences in interview scoring between virtual, telephone, and face-to-face interviewing [48] and structured to improve organizational attractiveness to applicants [49]. The activities in the logic model included interview meetings, interviewer training, and assessments of interviewee satisfaction with the interview process. Interviewer training helped ensure all interviewees are treated similarly [50]. It was also important to conduct sensitivity and anti-harassment training to mitigate any potential interviewer biases. Ideally, interviewers should be blinded to academic scores so that their ratings purely reflect non-academic traits; blinding also resulted in higher interview ratings for female and minority interviewees [9].

Admissions decisions, matriculation choices, improvements to interview process, and quantitative data were targeted outputs in the model. In the short term, interviewees will receive similar interview experiences and more equitable selection decisions. In the intermediate term, there will be improved understanding of interviewees, a more valid and reliable interviewing process, and a student body that better aligns with the goals of the admissions office. In the long run, schools and programs will improve interview reliability and interviewee perceptions and altered composition of the student body will improve the academic performance of students. The relationship between each approach to interviewing and its outcomes should be assessed to evaluate the effectiveness of an interviewing intervention.

In summary, despite the widespread use and attendant costs of medical school interviews, there is a paucity of studies that have rigorously examined the role of interviewing in selecting applicants. We recommend further research to address this gap, including examining the equity and predictive value of different selection interview formats and implementing interviewer and outcome assessor training to reduce study heterogeneity. Pending an evidence base with more depth and breadth, we suggest scaling down the relative weight of the interview in the admissions process [7] and using a virtual MMI, which offers an alternative that is safer (i.e., during the COVID-19 pandemic) and reliable while also reducing the financial impact of interviews for applicants, medical schools, and GME programs. We hope our logic model will help educators conduct rigorous admissions interviews for medical school.