When will I get my paper back? A replication study of publication timelines for health professions education research

Researchers have criticized the lengthy timeline from the submission of a manuscript to its ultimate publication, highlighting its detrimental effects to the overall progress of science [1, 2]. This criticism of publication delays may be well-founded. For example, a recent study in JAMA Oncology found that results from phase III oncology trials have a median time to publication of 350 days and even longer if reporting negative findings [3]. While such delays may negatively affect patients, scientists may suffer as well. Researchers have noted that lengthy publication timelines can be detrimental to scientists’ careers, leading to delays of promotion and tenure and/or failure to attain grant funding (e.g., due to scientists’ inability to reference their research under review) [4]. Early career researchers and trainees may be particularly negatively affected [5, 6].

Several studies have investigated publication timelines across a variety of disciplines and publishing models (e.g., open vs. subscription journals, science, technology, engineering and mathematics vs. humanities topics, high vs. low impact factor journals) [7, 8]. A recent Nature News investigation and concurrent blog post reported that Himmelstein sampled over 3 million articles from 3,475 biomedical journals present in PubMed with publication metadata between 1965–2015 and found that the average time from submission of a manuscript to its acceptance was approximately 100 days (SD unavailable) [2, 9]. Furthermore, this study found a lag of approximately 25 days between article acceptance and publication in PubMed, which was determined based on data available from 1997–2015. Specific to clinical medicine, another study investigated 781 articles published in 18 internal medicine or primary care journals and identified that the average time from submission to acceptance was 153 days (median = 123) with an average lag between acceptance and publication of 105 days (median = 68) [10]. While these studies provide two valuable benchmarks for understanding publication timelines across biomedicine and within a clinical discipline, we currently know very little about publication timelines in the field of health professions education (HPE).

We believe that lengthy publication timelines, if present, may be potentially problematic in HPE research; this underlies the impetus for the current study. Indeed, the cognitive, physical, and psychological challenges of learning medicine in demanding clinical environments should be met with timely, up-to-date, and evidence-based educational knowledge and instructional strategies [11]. Lengthy publication timelines may undermine the effectiveness of teachers who strive to deliver evidence-based content in an evidence-based manner within an optimal learning environment. Failure to disseminate the evidence that drives each of these goals may ultimately negatively impact medical learners and, indirectly, patients who rely on learners for competent care. Moreover, publication delays for work that explores important phenomena such as learner suicide or depression may negatively affect medical learners through delays in the implementation and evaluation of novel resources and support structures. In addition, HPE scientists whose work is embargoed during the publication process are unable to receive credit for pending publications in grant applications and subsequent research studies. Finally, long embargoes can hurt HPE researchers whose work in fast-moving areas like educational technology and social media may no longer be contemporary or relevant by the time it is published. Lastly, the reality that many authors must submit their manuscripts to multiple journals before they are accepted can further compound publication delays, adding additional time to the overall endeavor [2, 12].

In light of these concerns, we aimed to address the lack of data about publication timelines by replicating Himmelstein’s previous work to examine data in the HPE journals that openly provide this information. While Himmelstein’s work included articles published broadly in biomedicine from 1965–2015, we focus on HPE articles published 2008–2018. We believe this timeframe is appropriate to understand publication timelines in the rapidly growing field of HPE, as only two HPE journals provided publication history data prior to 2010. As such, Himmelstein’s prior work provides limited inclusion of HPE information. We believe these data could be a first step towards addressing potential publication delays in HPE and may spark conversations about publication timelines and ways to optimize them.

To calculate publication timelines in HPE, we replicated the bibliometric approach reported in Nature News by Himmelstein [2, 9]. We chose to replicate this particular approach because, unlike other studies reported in the literature [7, 8, 10], which relied on humans to extract the relevant data, Himmelstein’s approach utilized computer code and publicly available data. Using a computer-based rather than human-powered approach allowed us to more efficiently and objectively extract a large volume of data from articles published between 2008–2018 and to mitigate the risk of human coding errors.

We conducted this replication study with a sample of 14 journals that have been previously identified as core HPE journals, namely [13, 14]: Academic Medicine, Advances in Health Sciences Education, BMC Medical Education, Canadian Medical Education Journal, Clinical Teacher, International Journal of Medical Education, Journal of Advances in Medical Education and Practice, Journal of Graduate Medical Education, Medical Education, Medical Education Online, Medical Teacher, Perspectives on Medical Education, Teaching and Learning in Medicine, and The Journal of Continuing Education in the Health Professions. These journals were included in Himmelstein’s original study if they provided publication history metadata between 1965–2015. Thus, this study extends and replicates his earlier work by adding three additional years of data, during which the number of HPE publications steadily increased.

Following Himmelstein’s steps [2, 9], we queried PubMed on April 10, 2019 for articles published in these 14 journals between 2008–2018. PubMed was selected because it is the only publicly accessible database that provides publication timeline data. Our search yielded 19,182 citations, and we downloaded the complete, publicly accessible metadata for each citation. From this metadata, which was generated by National Library of Medicine staff, we identified the journals that make publicly available their article-level publication history (e.g., the timelines for each of the steps in the publication process). This history includes the date the article was received by the journal, the date the authors resubmitted revisions, the date the article was accepted by the journal, and the date that the article was entered into PubMed. Based on this available data, we defined the following three time periods: 1) Publication time: the time from article submission to appearance in PubMed, 2) Acceptance time: the time from article submission to acceptance by the journal, and 3) Processing time: the time from article acceptance to appearance in PubMed. These three time periods aligned with those defined in Himmelstein’s work, and, in similar fashion, we excluded from our analysis journals that did not supply this publication history metadata.

We used SAS 9.4 for analysis and data management. We ran two 2‑sided, independent sample t‑tests to determine any potential differences in publication timelines based on funding source. To increase the transparency of our work and encourage further replication, we have deposited our dataset and corresponding computer code here: https://​github.​com/​DNSchreiber-Gregory/​Publication-Timelines/​tree/​DNSchreiber-Gregory-Publication-Timelines.

Over the course of the study period (2008–2018), 19,182 articles were published in the 14 HPE journals sampled. Of these journals, publication timeline metadata were available for eight of the journals (Advances in Health Sciences Education, BMC Medical Education, International Journal of Medical Education, Journal of Advances in Medical Education and Practice, Journal of Graduate Medical Education, Medical Education, Medical Education Online, and Perspectives on Medical Education). During the study period, these eight journals published 8,681 articles. Of these articles, publication history data were available and extracted from 4,735 (55%) articles.

The mean publication time from author submission to posting on PubMed was 263.55 days (SD = 157.61; median = 228). The mean acceptance time from author submission to journal acceptance was 180.93 days (SD = 103.89; median = 163). The mean processing time from acceptance by the journal to posting on PubMed was 83.15 days (SD = 135.72; median = 23). Tab. 1 presents publication, acceptance and processing times for articles published between 2008 and 2018 in these eight HPE journals.

We have replicated and applied a previous study design within the field of HPE, and our findings suggest that, when compared with the previous study [9], which was broadly focused on 3,475 biomedical science journals with publication history metadata in PubMed, HPE may have longer publication timelines. To our knowledge, these findings represent the first and only available indicators of publication timelines in HPE. However, before discussing these findings and their implications, it is important to address the limitations of our approach.

Similar to the original study [9], our replication analysis was constrained by the incomplete publication history data made publicly available by the journals and their publishers. In our analysis, we were able to analyze only 25% of HPE articles published between 2008–2018. Furthermore, only eight of the 14 journals in our sample made data available, and none provided complete data for all of their articles. The incompleteness of the data may have skewed the results of our analysis, and this should be taken into consideration when examining the timelines for individual journals. Accordingly, we had to exclude some HPE journals, including Academic Medicine, which annually publishes the greatest number of articles in the field, as data are not publicly accessible for these journals. Furthermore, our findings represent only a coarse quantitative indicator of publication, acceptance, and processing time. In other words, we were unable to explore the publication process in a fine-grained manner, a process that includes multiple steps and multiple stakeholders (e.g. authors, reviewers, editors, and publishers), each of whom plays a role in decreasing (or extending) publishing timelines. Thus, the data are incomplete, and we are unable to draw detailed conclusions about what exactly occurs within our observed timeframes. However, despite these limitations, we believe our findings draw attention to the potential presence of publication delays and present an opportunity to spark conversation among authors, editors, reviewers and publishers in the HPE community.

Furthermore, our findings support the need for all HPE journals to publish timeline data. Indeed, if there is power in data, there is even greater power in open-access data and open-sourced data analysis [15, 16]. Unfortunately, despite our ability to conduct this study using publicly available data and a previously developed analytic method, our analysis was hampered by the lack of a complete dataset from all queried journals. Moving forward, we believe HPE journals should compile and make available the data necessary to thoroughly understand the processes governing the publication of our science and to which we are beholden. In order to develop a complete dataset, we call on all journals in HPE to make their publication timeline data publicly available in its entirety. Doing so will promote transparency and help identify the ways in which our publication timelines might be improved.

Awareness of accurate publication timelines in HPE education could benefit numerous stakeholders. For journal editors and publishers, analysis based on complete data would provide an opportunity for benchmarks within the field, critical reflection on their own timelines, and sharing of best practices from exemplars in the field. Journals that use this data to streamline their publication timelines may experience higher submission rates for higher quality articles from authors seeking a faster, more transparent publication process. At the program level, graduate programs in HPE increasingly require students to have publications accepted by peer-reviewed journals in order to graduate [17]. Knowledge of journal timelines may assist in the planning of educational programs and forecasting of graduation timelines. Related to funding, prior work has found that HPE studies supported by grant funding tend to be of higher quality [18]. Thus, our finding of shorter overall publication and processing timelines for funded research may be the result of better study quality [18]. However, this suggestion is speculative at the moment, and more sophisticated data analysis and complete data are needed to identify important funding variables and how they might influence publication timelines.

We believe that such transparency within the publication process is very important for authors. Currently, authors who submit manuscripts to HPE journals lack awareness of the timelines that govern the publication process. We believe this lack of information undermines authors’ ability to be critical about when and where to submit their manuscripts, a decision that may have real implications, especially for those authors facing funding or promotion deadlines or striving to publish contemporary research. Even more fundamental, we believe that transparency about publication timelines should be a basic courtesy afforded to all authors.

While our data are incomplete and do not include data from several key journals in our field, they do suggest that publication timelines—from the journals for which we have meaningful data—are over twice as long as the 100-day benchmark published by Himmelstein [2, 9] and longer than the timelines observed in internal medicine and primary care journals [10]. Thus, lengthy publication delays may be a challenge that needs to be confronted in the HPE community.

While we await greater clarity in the scope and nature of this challenge, which will be aided by consistent data sharing from all our journals, we have identified potential, immediate solutions—at the researcher, editor, and publisher levels—that might be explored to improve existing publication timelines.

In this study, we used publicly available data to determine publication timelines in HPE journals that make this information available. Our data, while incomplete, suggest that HPE researchers may face longer timelines than their counterparts in the biomedical sciences. Perhaps more important than this finding was our ability to use Open Science and scientific replication to answer, albeit incompletely, an important question in HPE. As a next step, we call on all HPE journals to consider sharing their publication timeline data with transparency and completeness so that all stakeholders in the publication process can have access to accurate, open information.