Multiple independent sampling within medical school admission interviewing: an “intermediate approach”

Admission to medical school is predicated upon applicants possessing academic ability alongside a range of key personal competencies [1]. Admission personal interviews are commonly used to rate personal competencies. However, evaluation of the admission personal interview demonstrated low inter-rater reliability and prediction of performance. Albanese et al. [2] reported that panels with multiple interviewers jointly interviewing applicants appeared to have minimal impact upon inter-rater reliability. An Association of American Medical Colleges (AAMC) 2011 Analysis in Brief reported that the admission personal interview has been slow to change with it still being the default interview format [3]. Koenig et al. reported that schools used interview formats ranging from unstructured to structured with the most commonly used format being semi-structured [1]. This publication [1] noted interviewing formats warranting attention, such as the Multiple Mini-Interview (MMI) [4]. Despite low measurement properties, admission personal interviews provided institutional opportunities to present a ‘human touch’ reflective of the importance ascribed to person-to-person interactions and personal attributes in the selection of future physicians within an otherwise stressful application process [2]. Moreover, a recent AAMC Last Page publication identified admission interviews as important to applicants’ decision-making regarding medical school selection [5]. Substantial recruitment value was ascribed to interviews enabling interaction with senior medical students, faculty and administrative staff.

One solution to optimize interview reliability in admission interviews is adoption of multiple independent sampling (MIS). Instead of one evaluation of an applicant, several, independent evaluations of the same applicant are averaged to generate an applicant’s evaluation [6]. However, uptake by American medical schools of the MMI, a MIS-based interview format, has been limited due to 1) its substantial resource requirements and 2) associated modifications of campus recruitment activities and diminution of recruitment advantages associated with admission personal interviews [6]. Of note, successful applicant recruitment contributes to student selectivity within a medical school’s admission processes and its associated prestige ranking [7], with prestige being a vaunted element of today’s medical school environment [7‐10]. Furthermore, recruitment advantage for admission personal interviews above the MMI has also been reported for Emergency Residency selection decision-making [11]. To address MIS and the combined issues of reliability, recruitment and resourcing within admission decision-making, Axelson and Kreiter investigated adapting MIS to admission personal interviewing. For their study, they conducted secondary data analysis of multiple admission interview cohorts; predicting reliability of 0.48 for three brief (one independent interviewer) personal interviews to 0.73 for nine brief (one independent interviewer) personal interviews. They recommended adapting admission personal interviewing via MIS as an ‘intermediate approach’ for schools wishing to address this combination of issues yet unable or unwilling to adopt MIS as utilized in the MMI, given its resourcing and/or recruitment impacts [6].

Inspired by the ‘intermediate approach’ model of Axelson and Kreiter [6] we developed the Modified Personal Interview (MPI) employing MIS for small-scale specialized medical student selection tasks (leadership [10] and MD PhD [12]). As a medical school intent upon bringing MIS to large-scale admission interviewing but unwilling to implement the MMI due to recruitment and resourcing issues we adapted our MPI small-scale model to our large-scale admission context. (Please see Appendix A and Fig. 1 for a description of this large-scale MPI adaptation). Cognizant of the aforementioned combination of admission issues, we evaluated outcomes relevant to each issue through our MPI implementation large-scale process: MPI reliability, MPI predictive validity plus MPI admission recruitment and resourcing outcomes. We hypothesized 1) MPI reliability to be intermediate between 0.48 to 0.73 as predicted by Axelson and Kreiter [6], 2) positive MPI predictive validity for in-programme skills assessments such as a first year objective structured clinical examination (OSCE) and 3) maintenance of recruitment outcomes and resource efficiency.

The study’s methods were selected to evaluate the MPI against multiple admission outcomes; 1) psychometric evaluation of MPI reliability and predictive validity and 2) institutional recruitment and resourcing outcomes associated with MPI implementation.

The purpose of this study was to evaluate the combined admission issues of reliability, recruitment and resourcing for a modified personal interview that incorporated multiple-independent sampling while balancing resourcing and recruitment requirements. Additionally, we explored other psychometric properties of this admission interview format such as predictive validity.

Our MPI implementation measured reliability of 0.56 fell within the reliability range per number of interviews as predicted by Axelson and Kreiter [6] but is less than the traditionally accepted standards of 0.7 to 0.9 [13]. As noted with our D study, an MPI with eight interviews would attain a reliability of 0.7 or higher. An eight station MMI is not always feasible for medical schools and in some cases may not be desirable as schools may wish to spend the interview day hosting other recruitment activities for applicants. Furthermore, while high reliability is always a desirable goal, educational assessment has progressed beyond simply relying on measurement as the marker of assessment success [15]. The capability of educational assessments to incorporate institutional goals – in our case recruitment and resourcing – is important. For our context, the MPI implementation was associated with maintenance of our school’s acceptance yield and associated markers of high academic ability in our cohort such as MCAT and GPA scores. Just as important, it was a resource efficient method of interviewing students.

Furthermore, our study advances Axelson and Kreiter’s work [6] with our preliminary validity evidence of MPI score correlation with the ASCM year 1 OSCE. Prediction of performance is a significant marker of validity evidence since selection processes aim to choose the candidates most likely to succeed in medical school. The MPI shows early evidence of predicting aligned outcomes – communication skills performance – via an OSCE. Communication skills are best evaluated in face-to-face encounters and thus it is no surprise that MPI interview performance should predict performance in a similar in-programme simulated communication scenario. While the magnitude of the correlations is small, the score should be contextualized considering that the ASCM OSCE is a low stakes, short examination used for generating feedback and identifying skills deficits. Furthermore, the MPI was the significant positive predictor in the admission process. Like the MMI [16] it could be expected that the magnitude of MPI prediction could increase over time as more robust and reliable OSCE examinations are presented to our medical students. Validation is a continuing process that relies on the sufficiency of evidence to substantiate claims. Further work must clearly be done to build a chain evidence for the MPI’s validation.

Moreover, our MPI used caring and conscientiousness as the global competency creating an opportunity to help construct the importance of caring, compassion and conscientiousness as an institutional value for both faculty and learners. Thus, in addition to creating a more ‘human touch’ [2] for applicants, putting caring and conscientiousness as a focus of the MPI necessitated thoughtful consideration of the constructs by raters in every interview. We were able to leverage MPI rater training as a faculty, current medical student and resident educational development opportunity for sharing this instructional value through online and face-to-face sessions. These activities enriched the institutional value of the explicit attention afforded the institutional goals pertaining to recruitment and resourcing.

Our admission file assessment scores were not highly correlated with our MPI scores. This is likely a restriction of range issue but further evaluation of whether file assessment is a good screen for interview or if it in fact provides other information is worth investigating in the future. This finding presents our school with new admission resourcing and recruitment questions. Admission file review scores as a stand-alone method do not predict performance on the ASCM OSCE or indeed other assessments within our programme. However, ‘open-file interviewing’ via review of admission file materials within our self-reflection MPI station had similar measurement characteristics with other MPI stations. The resourcing and recruitment questions to now be addressed involve the heavy resource demand of the admission file review phase in the face of poor prediction capability. However, use of the applicant file in the interview has recruitment value as it is highly valued by our applicants. A next step is consideration of further use of ‘open-file interviewing’. For example, expansion of the MPI circuit to five interview stations to include a second MPI using ‘open-file interviewing’ warrants consideration.

Our focus upon institutional resourcing and recruitment factors ignored applicant resourcing and recruitment factors. Recruitment strategies integrated within interviews and interview day activities obviously requires applicants’ attendance and travel costs for many applicants. Attention to this matter is essential as medical education costs and medical student debt continues to rise. A next step will be in-depth qualitative investigation of the differential recruitment impact of the MPI and other recruitment strategies, including both institutional and applicant components. Investigation of recruitment strategies is essential as the field explores internet-based MMIs [17]. Internet-based interviewing may eliminate much of the need for applicant travel to attend campus-based admission interviews. To keep pace with technological innovation, comparative investigation of campus-based admission interviewing versus internet-based admission interviewing and their associated resourcing needs and recruitment strategies is necessary.

Our assessment of MPI implementation has limitations. We were interested in assessing the combination of admission issues of psychometric rigour (reliability and predictive validity), recruitment and resourcing with MPI implementation by comparing these issues and associated outcomes across pre-MPI and MPI admission cycles. We were able to do so with respect to recruitment and resourcing. However, we solely assessed reliability of administration of our implementation MPI and were unable to assess reliability of our pre-MPI use of a traditional admission interview format due to each applicant being interviewed by only a single interview station.

Our MPI seven-point Likert rating scale was selected to remain consistent with our large cadre of interviewers’ experiences in other local medical school in-programme assessment contexts. Selection of a broader Likert rating scale may have attained greater MPI reliability than the currently measured 0.56.

In addition, our resourcing measure related to the size of admission interviewer cadre alone, we did not include other resource components such as space and administrative staff requirements, and question development time as outlined by Rosenfeld et al. [18].

We believe this report provides evidence to advance uptake of Axelson and Kreiter’s ‘intermediate approach’ model for schools considering adoption of MIS-based interviewing in the face of questions regarding resourcing and/or recruitment [6]. Future study will explore psychometric factors such as longer-term MPI predictive validity. We will also investigate the differential value of various resourcing and recruitment strategies in preparation for a future without onsite medical school admission interviewing.