Joining the meta-research movement: A bibliometric case study of the journal Perspectives on Medical Education

PME is the official journal of the Netherlands Association of Medical Education (NVMO). The journal’s mission is to support and enrich collaborative scholarship between education researchers and clinical educators, and to advance new knowledge regarding clinical education practices by publishing a variety of publication types including original research. The NVMO’s sponsorship enables PME to be a diamond Open Access journal, which levies no author charges and makes all articles freely available immediately under a Creative Commons attribution license. This is the least restrictive Creative Commons license enabling reuse while still providing credit to the authors. PME is published by Bohn Stafleu van Loghum, part of Springer Nature, and indexed in over 20 databases, including MEDLINE, Web of Science (WoS), and Embase. In 2020, PME was recognized with the Directory of Open Access Journals Seal of Approval for Open Access Journals and was also awarded a journal impact factor by WoS. PME is a single-blind, peer-reviewed journal. Manuscripts are handled by an associate editor and reviewed by at least two reviewers. The editor-in-chief renders all final decisions.

Data for this case study were collected for manuscripts submitted and published between 2012–2019. We focused on this time period because after 2012 PME transitioned from a Dutch to an English language journal and established an international editorial board. We primarily extracted data from PME’s version of Editorial Manager with the exception of usage data, which were provided by Springer, and basic citation data, which were obtained via WoS.

Author names, institutions, and countries were cleaned to combine various spellings and variants of the same entity into one (e.g., UCSF was combined with University of California, San Francisco). This was done using the VOSviewer thesaurus function. For author names we changed 58 spellings of author names for 53 authors to merge them with alternative spelling variants. The majority of changes affected first name initials only (e.g., “driessen, e” to “driessen, ew”). Author name disambiguation reduced the total number of distinct authors from 1285 to 1227. We also cleaned institution names, reducing the total number of institutions from 423 to 333. We found 955 keywords and, for ease of analysis, selected only those that appeared at least three times in our dataset, which reduced the keywords most frequently found to 101.

To predict the gender of authors and peer reviewers, we used the tool Genderize.io [16], which calculates a probability score for whether each first name is more likely to refer to a woman or man. If the tool predicted with 70% confidence that a given name was a given gender, we accepted the designation. For those names with less certainty or unknown gender (n = 62), we searched the individuals’ online presence. We recognize that gender is a complex social construct that is best described by an individual and that the binary nature of these results is not ideal. However, because the field of bibliometrics currently lacks better alternatives, we decided to use this approach, which has been employed to predict gender in several recent studies with similar aims [17‐19].

Descriptive statistics were calculated using GoogleSheets and Excel [20, 21]. To visualize and analyze collaboration patterns, citation relationships, and term co-occurrence, we conducted social network analyses. We used VOSviewer [22] to extract network data from the bibliographic metadata downloaded from WoS and to visualize the networks. Layout and cluster resolution settings vary depending on the size and structure of the particular network in order to optimize the network graphs [23].

Our author team includes researchers with expertise in HPE and information science. Three of the authors have official connections to PME. ED is the Editor-in-Chief, LM is the Deputy Editor-in-Chief, and AA is an Associate Editor of PME. SH is a guest editor for PME’s special issue on meta-research. We recognize that our relationship with the journal gives us information access that is traditionally not publicly available (e.g. usage data); this is a strength of this case study. However, we also realize that this could be construed as a conflict of interest. Therefore, for transparency, we deposited the study data obtained from Editorial Manager in an open-access database to allow for independent review and replication of our findings [23]. Moreover, in the results and discussion sections, we propose some interpretation of the data based on our own experience as editors and researchers in the field. However, we recognize that these interpretations are subject to our own biases and that other researchers may draw different conclusions. Thus, our aim is not to provide a definitive interpretation of these network maps, but rather to invite the HPE community to draw their own conclusions and to engage in a broader discussion. To that end, we have embedded questions to the community for each of the visualizations to spark thought and encourage dialogue. These questions are labeled: For Discussion.

Reviewers were based in 49 countries with those in the United States (US) invited most frequently (n = 325; 27.1%) followed by the Netherlands (n = 217; 18.1%) and Canada (n = 200; 16.7%). Women were invited to review more often than men (women n = 655; 54.5%) and completed more reviews (n = 369, 56.7%).

On average, reviewers took 16.1 days from accepting a review assignment to submitting their final review (range = 0–82). Women took on average 16.7 days to submit a review (range = 0–82; median = 17; SD = 10.4) and men 15.4 days (range 0–65; median = 16; SD = 11). Values of 0 indicate that the reviewers returned their reviews on the same day that they received their assignments.

A total of 1227 unique authors published in PME between 2012–2019. Overall, 49% of authors were predicted to be women and 46% men. We were unable to determine the gender of 5% of names. Men and women were similarly productive (1.4 vs. 1.3 documents per author, respectively) and collaborative (5.0 vs. 4.7 co-authors per author, respectively), which aligns with related findings suggesting relative gender parity in HPE publications in recent years [26, 27]. The most productive authors were Lingard with 20 and Leppink with 16 publications (see Tab. 3 for the top 10 most productive authors). Notably, their high productivity can be explained by a large number of invited publications: 17 of Lingard’s articles were from the Writer’s Craft series and six of Leppink’s articles were from the Statistical Points and Pitfalls series. All of these articles were invited by the Editor-in-Chief, led by Lingard or Leppink, and peer reviewed.

PME received manuscripts from authors in 81 countries. US-based authors submitted the most (n = 572; 24.7%), followed by the UK (n = 312; 15.0%) and the Netherlands (n = 276; 13.3%). Notably, in 2019, the journal received its first submissions from authors based in Cuba, Cyprus, Lebanon, Oman, and Zimbabwe. PME articles listed affiliations in 35 countries with the largest number of authors based in the Netherlands (n = 151; 29.3%), US (n = 111; 21.5%) and Canada (n = 110; 21.3%).

In the next sections, to visualize author collaborations, we present network maps illustrating relationships based on different levels of co-authorship.

PME authors were quite collaborative. The authors with the largest number of co-authors were Ten Cate, Durning, and Maggio, who collaborated with 40, 33 and 33 different authors, respectively. Based on the co-authorship network displayed in the center of Fig. 2, 447 of the 1227 authors were part of the largest component in the co-authorship network. This means that these 447 authors were connected either directly (through co-authoring at least one article together) or indirectly (by having a co-author in common). On the other hand, there were 57 clusters, positioned at the periphery of the network, composed of individual authors who published all their work in PME as single authors.

Authors were grouped into 220 clusters (Fig. 2). The largest cluster (red, right of center of the network) contains 57 authors with Varpio (12 publications), Driessen (8), Maggio (8), Young (8), Artino (7), St Onge (7), and Paradis (5) being the most productive. This cluster contains 56% women and 38% men (6% of authors’ gender could not be determined). The second largest cluster (dark green, top left of center) consists of 52 authors who, on average, published less frequently in PME: Dekker published 6 articles, followed by O’Flynn, Cantillon-Murphy, De Jong, Ommering, and Dankbaar with 3 PME publications each. This cluster is slightly more male dominated with 53% men and 45% women authors (2% of authors’ gender could not be determined). The third cluster (dark blue, bottom left of center) connects 47 authors, the most prominent being Van der Vleuten, Durning, and Schuwirth with 11, 10 and 8 articles, respectively. This cluster contained 49% authors with male first names and 45% with female first names (6% of authors’ gender could not be determined).

When considering these three dominant co-author clusters, it is interesting to note that many of these authors are leaders of, and faculty members in, units that administer graduate programs in HPE. For example, at the time of this data extraction, Durning, Maggio, Varpio, and Artino were faculty members in the Uniformed Services University’s HPE graduate program. Such graduate programs in HPE, many of which include publication requirements in order to graduate and would likely include faculty member co-authors, have the potential to increase an author’s opportunities for collaborative authorship. Additionally, we see in many of these clusters Dutch members who were key in PME’s founding and who have played leadership roles in the NVMO.

Authors were affiliated with 333 different institutions, 233 of which were connected in the largest component of the institutional collaboration network. Authors affiliated with Maastricht University published the most articles (n = 65), followed by Western University (n = 34) and University of Toronto (n = 32) (see Tab. 3 for top 10 institutions). Institutional clusters often reflect close domestic collaboration. For example, in Fig. 3, the largest cluster (red; to the right of center) includes 36 institutions, with three Dutch universities—University of Utrecht (n = 25), Radboud University Nijmegen (n = 19) and Vrije University (n = 19)—being the most prolific. The second largest cluster (green; bottom center) includes 30 institutions, the most prolific of which are in the US, including University of Michigan (n = 9), Harvard University (n = 8), and Cleveland Clinic (n = 7). The third largest cluster contained 28 institutions, led by University of Aberdeen (n = 9), University of Liverpool (n = 6), and New York University (n = 5).

PME articles included authors with institutional affiliations in 40 countries. The great majority of PME articles were written by authors based in the Netherlands (171 publications), reflecting the former national focus of the journal. The US (138), Canada (131), and UK (76) are the second, third and fourth most represented countries, respectively. In terms of collaboration patterns, these countries showed the strongest ties with each other. The largest number of co-publications occurred between the Netherlands and the US (24), the US and Canada (16), and Canada and the UK (10). We identified that authors from nine countries represented only collaborated with authors from their own country. See Zenodo file for the top 10 institutions by number of articles published [23].

The presence of these predominantly Western institutional and geographical affiliations aligns with patterns observed in HPE more broadly [26‐28], although authors from the Netherlands are more heavily represented. The prevalence of Dutch authors and editorial board members is likely an artifact of the journal’s history; that is, the journal originated in the Netherlands, where it remains today, and is sponsored by the NVMO.

Author keywords were available for 401 of the 518 articles. After cleaning terms (see Methods), 955 unique terms were used to describe these documents. The most frequent terms were medical education (assigned to 73 documents), medical students (26), assessments (24), feedback (21), undergraduate medical education (19) and curriculum (15). As one might expect, the distribution of documents per author keyword was very skewed, and the majority of terms (746, 78%) were assigned to one document only.

Fig. 4, which presents a network of keywords that co-occur, is limited to author keywords assigned to a minimum of 3 documents and thus reduced from 955 to the 101 most frequent keywords. As the most frequent keyword that co-occurred with the largest number of other keywords, medical education, is at the center of the network as the largest node. It is part of the second largest cluster 2 (green) together with 21 other author keywords, including undergraduate medical education (19 documents), curriculum (15), qualitative research (13) and clinical reasoning (10). Overall, the network contained 7 clusters. The largest cluster 1 (red) contains 31 different keywords with assessments (24), feedback (21), faculty development (13) and professionalism (12) being the most frequent. The third cluster (blue) contained 14 keywords, including medical students (26), simulation (9), competency-based medical education (6) and learning (6).

In considering these clusters, it is easy to be distracted by the central key word medical education. The dominance of this keyword possibly indicates that PME is more focused on manuscripts addressing topics specific to training of doctors and less so on the broader field of HPE. However, in our experience, the term medical education is often used interchangeably with the more inclusive term, HPE, and so this finding should be interpreted with caution. Next, it is noticeable that the largest cluster (red) contains keywords that one could group as associated with assessment (e.g., assessments, workplace-based assessments, programmatic assessment), suggesting that PME has made assessment a key focus. While these keyword clusters can be interesting, these data speak to the lack of consistency in author keyword selection, which makes this type of network map less useful. This lack of consistency reflects similar findings in a case study of the multiple terms HPE researchers use for the term faculty [29].

While the number of manuscripts that PME published has remained constant, there has been a significant increase in the number of submissions, which have continued to climb during and beyond the study period. In 2021, for example, PME received over 900 submissions, which may be an artifact of the COVID-19 pandemic. But in 2022 PME submissions are on pace to exceed this number. As noted, journal pages are precious, and each article exacts a cost that demands both monetary resources (e.g., the publisher’s estimated cost to produce an article is between $ 3500–4000 [30]) and human resources (e.g., peer review and editorial oversight). At PME, we would like to grow the number of published articles to enable a broader diversity of perspectives and to raise the journal’s acceptance rate. Over the past year, we have started to address this desire by encouraging authors to streamline their manuscripts (i.e., shorter word counts and fewer figures and tables). Where appropriate, we also have suggested that authors consider depositing supplemental materials or extended methods sections on repository sites like Zenodo. Such practices provide authors with additional space to fully describe their approaches and share multiple exhibits, while also producing a digital object identifier (DOI), a persistent identifier that is searchable and citable. Such practices also support an open-science approach to medical education research, which we believe encourages the responsible conduct of research [6]. Furthermore, looking to the future, we are investigating the possibility of alternate journal models that would enable us to expand our allowable number of published pages.

While PME received and published manuscripts from around the world, the majority of those manuscripts, as well as the majority of the researchers who peer reviewed those manuscripts, were affiliated with Western countries. While this finding aligns with the broader literature [26, 28] and statistics from other journals in the field (e.g., Medical Education [31]), we feel there is an urgent need to grow our global representation and, as Kusurkar recently wrote in a PME editorial, work toward fixing the “leaky pipeline” of medical education researchers [32]. To this end, we are strategizing with our editorial team and editors of other HPE journals on how to more fully involve authors, readers, reviewers, and editors from around the globe.

Educating the medical education community of scholars has always been one of PME’s most cherished goals, and based on several of the highly cited, education-focused articles listed in Tab. 2, it seems our education goal is also valued by the community. For example, the Writer’s Craft, Insiders’ Perspective, and Statistical Pitfalls, are all publication types that take an educational focus, and these articles have been some of our most popular collections, as measured by downloads and even social media attention. We plan to continue to create novel article types that not only expand authors’ publication options but also serve as open-access educational materials that can be broadly applied across the community to improve the field’s research and communication. Additionally, we have been increasingly offering topical and skills-based workshops for the HPE community that have covered areas such as responding to peer reviewers, responsibly conducting research, and targeting the right journal for your manuscript. We will continue to seek opportunities to educate members of our HPE community. Additionally, in an attempt to offer more formal educational opportunities, we are in the early phases of creating an editorial internship program that will familiarize researchers with the editorial process and the responsibilities of being an editor.

In this study, we were fortunate to have access to the Editorial Manager’s internal report system; we also had permission from Springer to use those data. Notwithstanding this access and permission, we currently do not have standing, guaranteed, real-time access to our own data—data that are generated by our readers and editors. This lack of access is a policy hurdle, not a technical hurdle. For example, the Open Journal System, which is operated by the Public Knowledge Project, is the world’s largest open access publishing platform. It makes readily available—in real time—data on article and abstract views, editorial and review activity, and user counts by submission. And these data are readily available to its editors without the need for a special request [33]. We contend that access to a journal’s data is critical to monitoring the journal’s health and assessing the success of new initiatives. For example, if a journal implements a program to train researchers from under-represented countries, we would want to know if and how successful that initiative is at improving the representation of those countries in the scholarly conversation. Moreover, we advocate for the widespread capability to share such data so that all members of the community can learn from these efforts. If the editors have access to these data and can share the data with their constituents, then together we can work to identify what works and what does not across medical education’s publishing landscape.