Characterizing the literature on validity and assessment in medical education: a bibliometric study

Assessments must be of high quality, and validity is a key marker of quality. Recent reviews report suboptimal application of modern validity frameworks. Conducting a bibliometric study, we found that work is created by many authors, published across education and clinical journals, reports a variety of assessment approaches, and prolific authors work in highly interconnected networks. This suggests literature reflects a variety of perspectives intended for a range of audiences. The variability in validation practices identified in previously published systematic reviews may reflect disciplinary differences, or perhaps even differences in understanding of validity and validation, rather than poor uptake of modern frameworks.

Assessment is omnipresent throughout the continuum of medical education whether for admissions, to monitor knowledge acquisition, to follow trajectories of learning, to support maintenance of competence, or for formal gatekeeping for professional practice [1‐7].  As such, it spans training levels, contexts, areas of medicine, and assessment modalities including multiple choice questions and high fidelity simulation contexts [8‐12]. Regardless of the multiple contexts and purposes, it is imperative that the assessments put in place in medical education are of the highest quality.

One of the markers of quality assessment is the validity evidence available to support the use and the interpretation of the assessment scores. There have been recent literature reviews reporting that published validation studies are falling short of recommended practices [13‐16]. Cook et al. [14, 15] have suggested that we need to engage in better knowledge translation of validation frameworks to ensure increased quality in the assessment literature. Knowledge translation efforts to document the gaps between recommendations and actual validation practice implicitly assume that there is one ‘best’ framework that should be translated (e. g. [16‐19]). However, medical education has been labelled as an emerging field, informed by various disciplines [20, 21], and intended for several different audiences, whether clinical, specialty-specific, educational, or focused on communicating to researchers in the field [20]. It is therefore possible that the medical education literature may facilitate the coexistence of different frameworks, or even conceptualizations of validity. St-Onge et al. have recently documented at least three different conceptualizations of validity present in the health professions education (HPE) literature [22], and the observed variation in validation practices [14, 15] could also be reflecting the differences across the disciplines that inform HPE. More specifically, some documented differences in validation practices reflect various validity frameworks or theories, from notions of the ‘trinity’ (construct, content, and criterion) [23], the unified theory of validity [24], and an argument-based approach to validation [25]. If a lag in knowledge translation [14, 15] and differences in conceptualizations of validity can result in differences in observed differences in validation, could there be other potential factors that could explain the observed variation in validation practices?

It may be possible that an analysis of the breadth, scope, and characteristics of the assessment literature in medical education may provide a window into understanding the previously reported gap between actual and recommended practices [15]. We employed bibliometric approaches [26‐28] to document the breadth, scope, nature, and heterogeneity of the literature reporting on validation of assessments within medical education literature. Bibliometric analyses can provide means to describe a body of published work, which is based, in part, on the assumption that published literature reflects the state of knowledge in a given field [26‐28]. These analyses can provide data concerning publication characteristics, patterns, and estimates of absolute and relative productivity [21, 26, 29, 30]. Several bibliometric studies have been published in medical and health professions education [26, 29‐32], but many of these have limited scope by limiting to one journal, or one type of journal, in order to characterize medical education as a field [29, 30]. Rather than focusing on medical education as a field, or publication patterns of a given journal, here we focus on describing the literature on validation of assessment in medical education across multiple publication contexts. In this study, we identify articles reporting on validity of assessment within the broad field of medical education in order to explore the scope and variability of work on validation of assessments.

For the purpose of this bibliometric study, individual publications reporting on validity in the context of learner assessment in medical education are the units of analysis, as the data regarding the publication (such as where it is published, by whom, and when) are of interest, rather than the content of the article itself.

This study reports a comprehensive bibliometric profile of the literature reporting on validation of assessment within medical education between 1999 and 2014. The amount of literature reporting on validation of assessment appears to be growing rapidly, with the majority of publications generated in North American contexts. The largest number of publications are produced in the United States, the United Kingdom, and Canada; however, the countries with the largest proportional productivity (number of papers per medical school) are Canada, the Netherlands, and Denmark—similar to findings reported by Doja et al. [29]. The largest number of articles were published in the journals Medical Education, Academic Medicine, Medical Teacher, and Journal of General Internal Medicine.

When attending to the categories of journals, relying on the NLM Broad Subject Terms for Indexed Journals [36], Education represented the category with the largest number of publications, meaning that the largest number of papers were published in journals classified in Education. However, General Surgery and Medicine represented the largest proportion of journals—meaning literature reporting on validation of assessments is spread broadly across a large number of journals in Medicine and Surgery, perhaps rendering some of the assessment literature difficult to synthesize.

The primary observation from this bibliometric profile is the heterogeneity present in the literature on assessment and validity in medical education—heterogeneity in authors, training backgrounds, location, journal of publication, publication domain, and assessment topic or tool. This heterogeneity has been noted in more restricted bibliometric studies (e. g. [26, 29, 30]) and has been expanded on here by including Medicine and Education journals in our search and analysis approach. Our findings align well with these earlier bibliometric studies, suggesting that the integrated nature of medical education as a field contributes to its breadth in terms of published literature. Furthermore, our study demonstrates the rapidity of growth of this literature base [13, 22, 29]. Here, we note a similar growth in the assessment literature with notable expansion since 1999. We can offer no current specific evidence-informed explanation for the growth pattern observed in this study; however, speculative considerations include expanded roles of assessment within medical education (e. g., assessment for learning, programs of assessment, etc.), and possible increases in available avenues for publication.

The literature included in this study represented a broad range of assessment tools and approaches, generated by a large group of authors (a total of 4,015 authors listed on a total of 2,863 papers). A group of 89 authors were identified as ‘authors of influence’, having published more than five papers included in our database, and appeared as authors on 1,069 of the articles in our study. These authors represented MDs, PhDs, and MD/PhDs from a variety of clinical specialties and disciplinary fields, and the majority represent a highly-integrated group of authors. Prolonged and highly-integrated patterns of co-authorship may suggest that the multiple perspectives and training backgrounds represented in these collaborations are contributing to the literature in a valuable and meaningful way.

Many prolific authors were broadly connected; however, other authors were less-well centrally embedded in the authors network. An examination of assessment approach, assessment tool, construct of interest, context of assessment, and discipline was not able to fully explain the presence or absence of collaborative links. Similar institutional affiliations appear to facilitate collaborative links (e. g. McGaghie and Wayne); however, several long-term collaborations appear to happen across institutions. While entirely speculative, we believe that some clusters of authors may conceptualize validity, or approach validation, very differently. It may not be unreasonable to assume that an approach to validation would be very different if focused on longitudinal programmatic assessment than if focused on ensuring the appropriateness of assessing within a simulated, or virtual reality, environment [25, 37]. We hypothesize that different conceptualizations of validity, or different approaches to validation, may be one factor that underlies the likelihood of collaborative links between individuals within our network. We have some indications that individuals who share similar understandings of a given construct are more likely to collaborate, or likely to engage with each other’s work; however, documenting this phenomenon remains in its infancy [38]. We also speculate that individuals with different conceptualizations of validity, or different approaches to validation, would be unlikely to collaborate extensively. This remains an area for future research, but approaches to validity and validation may be an important underlying factor in predicting successful collaborative relationship within assessment in medical education.

This study has limitations. The work reported here focuses solely on the meta-data available for literature on validity of assessment within medical education, and as such can speak to the breadth, scope, and nature of the published literature, but not to its specific content and quality. Examination of the context, content, and quality were beyond the scope of the current study, however represent important avenues for future research. Given the search strategies and databases used, this may have led to an overrepresentation of papers published in English language venues, or an overrepresentation of papers from North America. Another limitation of this study could be how we structured the author search in order to identify authors of influence. We decided to focus our analysis on first, last, and sole authors, and adopted the assumption that authors of influence would be more likely represented within those author positions. We recognize that this is an assumption, and the meaning of author order within health professions education research could remain an interesting avenue for future research.

This study could not examine the specific approaches, frameworks, or practices associated with validation of assessment tools and approaches, as the in-depth analysis of validity evidence was not possible with a bibliometric approach. However, the heterogeneity of journals, contexts, assessment tools, authors, and respective training backgrounds reported here could imply heterogeneous approaches to validity and validation. While speculative, it may be reasonable to assume that such a heterogeneous literature, generated by a heterogeneous group of authors, is unlikely to approach validity and validation of assessment in a homogeneous manner [21, 37]. It has been suggested that there is a disconnect between current recommended validation approaches and actual practice [16, 19], and it is possible that this disconnect may be due, at least in part, to different conceptualizations of validity [21]. This study may suggest the presence of multiple conceptualizations of validity and validation; however, this suggestion is based only on the observed heterogeneity of the bibliometric qualities included in this study.

In conclusion, the purpose of this study was to examine the bibliometric characteristics of the literature reporting on validation of assessments within medical education. This literature is growing rapidly, and is quite diverse, which likely suggests that approaches to validation of assessment within medical education are heterogeneous in nature. A gap has been identified between recommended and reported validation practices in medical education assessment literature, and we would agree that better translational work of several relevant assessment frameworks [24, 25, 39] for application in medical and health professions education is one important means for improving the quality and communication of validation studies. There is little evidence to support the unilateral endorsement of one validation framework in the context of assessment in medical education; rather, we would suggest that authors explicitly report which framework was chosen, how it aligns with the purpose, context, and intended use of assessment scores. It is possible that the variation in assessment practices previously documented [15] is reflective of the diversity in assessment tools, approaches, contexts, audiences, constructs, and publication venues represented in this bibliometric analysis of the literature reporting on validation of assessment in medical education.