Does the Arc of Science Bend Towards Impact? Four Decades of Empirical Research Published in JADD Since the DSM-III

The initial search yielded 708 citations in JADD across five time points (1979, 1989, 1999, 2009, 2019). A title/abstract review identified 92 citations that did not present original empirical research (e.g., letters/commentaries, review articles, meta-analyses, and so on). The classification of the remaining 616 citations on the research roadmap is summarized in Fig. 1.

Of the 413 basic and 204 applied research studies identified, more than 90% were published since 2009 and 60% in 2019 alone. There was no evidence of a relative increase in applied research; basic research remained steady throughout this period, ranging from between 58 and 69% except for a peak of 80% in 1999. Relative to other categories, intervention research dropped from a peak of 31% and 24% of all research in 1979 and 1989, to 12% in 2009 and 2019.

This decrease in intervention research in 2009 and 2019 corresponds with an increase in other applied research. After one applied research study in 1989 (Sugiyama & Abe, 1989) and a second in 1999 (Kasari et al., 1999), other applied research studies outstripped assessment research by 2009 and intervention research by 2019. This category included a range of studies, examining shifts in diagnostic practices (Grether et al., 2009), surveys of treatment practices (Wong, 2009), the use of web-based tools and supports (Mazurek et al., 2019), and parent perspectives on assessment (Jashar et al., 2019) and intervention (Grindle et al., 2009), among other themes.

With regards to the translation of intervention research into community-based programs data from the first three time periods were collapsed together because of the low number of outcome research studies recorded. These results are summarized in Fig. 2. The proportion of intervention research studies occurring in community settings remained steady throughout this period, between 42 and 50%. While the proportion of intervention research studies occurring in more traditional research settings decreased from 53 to 33% between 2009 and 2019, this was partially due to an increase in hospital-based intervention research. Given the many hospital-based programs that are either research centers or research partners themselves, it is difficult to determine whether this reflects a true shift away from traditional research settings. Overall, only about 6% of all original research studies published in JADD clearly involved intervention research conducted in community settings, a proportion that did not increase during the study period.

Almost two-thirds of the direct intervention was carried out by researchers or their students or staff. This proportion decreased from a high of 85% during the first three time periods to 61% by 2019. This split reflects that interventions in a community setting (e.g., in the home or school) may have not only required that researchers play a critical role in training and supervising the delivery of an intervention, but sometimes also in delivering it themselves. Overall, only about 3–4% of all original research studies published in JADD clearly involved research in which community-based practitioners clearly delivered the intervention. There was no notable decrease over the course of the study period in the overall proportion of published studies in which researchers delivered the intervention.

In this study of research published since the release of the DSM-III in 1980, we sought evidence of a relative increase in the kinds of applied research more likely to immediately impact the lives of people living with ASD. After categorizing more than 600 original empirical research studies published in this journal at five different points as either involving basic or applied research, there was no clear evidence of a shift in the proportion of applied research over time. The proportion of basic research decreased following a peak of 80% in 1999, but only to points higher than had been noted at the two earliest time points in this study. While the absolute number of applied research studies also increased dramatically over these most recent time points, the number of basic research studies did so too, with basic research continuing to make up about two-thirds of all publications. The recent explosion in research publications has probably increased our general knowledge about ASD – its possible causes, characteristics, and general trajectory – but this does not appear to have spurred a relatively increased interest in the more practical knowledge about how to help people living with ASD. In other words, the arc of ASD research does not appear to have bent towards impact over the past 40 years.

To understand this explosion of publications, it is worth considering the relationship between the publication and the funding of empirical research related to ASD. This emphasis on basic research in publication patterns is certainly consistent with funding patterns described earlier in the UK (Pellicano et al., 2013) and re-analyses of research funding in the US between 2008 and 2013 (Doehring, 2019b). This explosion in research publications corresponds with passage of the Combating Autism Act in 2008 (and continued through subsequent re-authorizations), as well as the increased efforts of organizations like Autism Speaks and the Simons Foundation to raise and direct hundreds of millions of dollars from private sources.

It is also important to recognize that relationships between research funding and publication patterns may exist at multiple levels. At the system level, research funding and research publications are probably closely interwoven. While these dramatic increases in publications in 2009 and 2019 are certainly the fruits of these investments in research, it also seems likely that research published in the decade prior to this period fueled the advocacy that led to the Combating Autism Act. For example, earlier research published by the Centers for Disease Control and suggesting higher than expected prevalence (Bertrand et al., 2001) led to increased calls for more research to understand and respond to these trends. For the individual researcher, research publications and research funding are inextricably interwoven. Researchers early in their career without a strong record of research publications will struggle to secure the research funding needed to secure an academic appointment, while those unable to translate research funding into still more publications may find it difficult to build a program of study. To some extent, this explosion in research publications since 2009 might therefore be attributed to a new generation of researchers eager to quickly establish their careers, encouraged by a new influx of funding.

In this context, the relatively decreased emphasis on intervention research is surprising. Over the 40-year period studied here, the emphasis has shifted away from intervention research and towards basic research. By 2009, only one out of every eight studies tested an intervention, a decrease of 50% relative to 20 years earlier. The design of the current study sheds little light on the reasons for this disconnection. One possibility is that some types of basic research have not spawned specific interventions, only more basic research. For example, while some the evidence-based interventions identified through the systematic reviews of Odom and his colleagues (Wong et al., 2015) can trace their origins to basic research on the social, communicative and behavioral characteristics of children with ASD, it is not clear whether any of these intervention practices can be traced back to other basic research findings related to neurological or genetic differences. It is also possible that interventions resulting from basic research have not led to other intervention research studies, but this seems less likely. Systematic reviews of evidence-based practices (Reichow et al., 2011) reveal the many different types of research studies that may follow an initial publication of an effective intervention; other research studies can replicate the original finding, further specify the skill or behavior targeted, explore outcomes in other populations, test training protocols, and so on. In many respects, the initial publication of an effective intervention should trigger a new wave of related research, and result in a relative increase in related publications.

It is also possible that these differences simply reflect that intervention research can be more difficult to do. Stated simply, basic research requires identifying and then evaluating an interesting characteristic in an interesting population under very carefully controlled circumstances. Interventions add other layers of complexity to control for; the need to identify a clinical need, access a clinical population, deliver an intervention with fidelity, and so on. Both the design and the implementation of intervention research require that the researcher has either invested the time needed to become an expert in the intervention, or has identified a clinical partner with the requisite expertise. Extremely rigorous standards of research design (e.g., including double-blind, placebo controlled clinical trials) create additional barriers. It is therefore easy to imagine that researchers faced with these barriers will either publish fewer articles describing intervention research, or avoid intervention research altogether. Early career researchers under pressure to publish may hesitate to invest the hours needed to build broad clinical expertise.

Given the relative lack of focus on intervention research, the lack of a clear shift towards intervention research outside of traditional research settings is hardly surprising. Overall, only about 3% of original empirical research that was published in JADD tested interventions in community programs. About one-half of intervention research studies clearly occurred in community settings, and only about only half of these clearly relied on community-based interventionists. These proportions did not consistently increase over the time periods studied. The increases in research conducted hospital-based settings may, however, merit closer examination. While there are many examples of hospital-based professionals and programs so closely intertwined with research centers that they ae best considered a traditional research setting, the number of studies produced by hospital-based professionals who devote most of their activities to clinical care and who conduct research largely independent of research centers is less clear.

What are other possible reasons for this gap in community-based research? Conducting intervention research in community settings adds still more layers of complexity for investigators to consider. Partner agencies in the community must be identified that see the value of research and the intervention under consideration. Interventionists must become partners in delivering the intervention with fidelity. Engaging programs and training interventionists requires additional time and resources, creating more pressure for researchers. If it is easy to imagine why a researcher might turn away from pursuing intervention research, it is even easier to imagine why they would hesitate to conduct intervention research outside of a traditional research setting, especially given the upfront investments in the time needed to build relationships with community-based programs. Given the pressures to publish, an aspiring clinical researcher might hesitate to seek the kind of practical experience delivering community-based interventions needed to anticipate the challenges of conducting research in these settings.

The rise of other applied research over the most recent time periods is interesting, and worthy of additional study. Some of the studies sought to understand other gaps in early identification in the community – like the difference between clinically- and research identified cohorts (Barbaresi et al., 2009), referral biases amongst ethnic minorities (Begeer et al., 2009), or shifts in autism identification within state agencies (Grether et al., 2009). These represent a natural extension of efforts to improve identification in research setting. Otherwise, the range of topics considered was very broad, and not as easily understood.

Given this special issue on the 40th anniversary of the DSM-III, the pattern of research surrounding the release of the DSM-IV in 1994 is particularly interesting. The fact that assessment research peaked 5 years prior to and 5 years after the release of the DSM-IV may reflect how this research helped to first drive the development of a new taxonomy and then sought to understand its implications. Seminal publications introducing the Autism Diagnostic Observation Schedule (ADOS) (Lord et al., 1989) and the Autism Diagnostic Interview (ADI) (Le Couteur et al., 1989), 1989 marked a watershed in diagnosis. Basic research peaked 5 years after the release of the DSM-IV, exploring a new taxonomy and the spectrum of ASD with designs that distinguished between subtypes in relation to associated features (Craig & Baron-Cohen, 1999; Dennis et al., 1999; Klin et al., 1999; Müller et al., 1999). As a result, interest shifted away from intervention, with only one such study published in 1999 (Bolman & Richmond, 1999).

One way to determine if and how the arc of science bends towards impact is to consider patterns of publication across an entire field, as described above. It is also interesting to consider if and how specific findings from these early years have since shaped practice. For example, the emergence of the ADOS and ADI in 1989 clearly set new standards for diagnosis in research settings. No other instruments have arguably played a greater role in shaping subsequent iterations of the DSM. Their record in shaping the identification of ASD in community settings is, however, less certain. While some of the larger and more specialized community programs can justify the investments needed to build clinical expertise in the use of these instruments (Doehring & Winterling, 2011), especially given the volume and complexity of cases encountered, these instruments have not had nearly the same breadth of impact on diagnosis in most community settings. Indeed, the definitions of ASD included in many state and federal regulations that guide the identification of ASD in public schools remain woefully outdated (Doehring & Volkmar, 2016).

It is also interesting to consider the four intervention studies published in 1979. Three of these (Holman & Baer, 1979; Koegel et al., 1979; Strain et al., 1979) described specific behavioral procedures to improve the acquisition of important social and academic skills while decreasing problem behavior, presaging trends that would continue for decades. These kinds of behaviorally-based teaching strategies have since transformed home-and school-based programs across the globe. When informed by the functional assessment of problematic behavior, these techniques can also reduce the occurrence of behaviors that otherwise would have prevented countless children with ASD from accessing community-based settings.

In contrast, the fourth intervention study published in 1979 included punishment in a package of techniques intended to suppress behaviors that are highly problematic, in two institutionalized adults with very significant levels of disability (Foxx et al., 1979). On the one hand, the shift away from punishment towards positive and preventative interventions informed by the function of behavior, as captured by studies included in this review (Carr & Kemp, 1989; Duker & Rasing, 1989), demonstrates the potential for research to have tremendous impacts. On the other hand, reports in the popular press continue to document the inappropriate use of seclusion and restraint to effectively punish problematic behaviors, sometimes with deadly consequences. Nonetheless, interventions to address these types of severe behaviors are rarely studied. One systematic review of all research published over an 18 year period on interventions targeting severe behaviors identified only about 100 studies involving a total of about 150 participants (Doehring et al., 2013). More recent reviews continue to highlight the lack of interest in individuals with the most significant levels of disability (Stedman et al., 2018). Adults like those studied by Foxx and his colleagues more than 40 years ago live with the most profound consequences of their disability, and have the most to benefit from improved understanding and treatment. In these cases, the arc of research appears to bend away from those with the most to gain.

The systematic mapping of research in a specific area of assessment or intervention might provide other insights into how basic research can be translated into population-level impacts. Consider the preliminary application of a research roadmap to the CHAT and its variants (Doehring, 2019a). Basic research was first used to describe specific deficits in joint attention and other social-communicative behaviors among young children already identified with ASD (Sigman et al., 1986). Research subsequently tested whether these and other deficits could become the basis for an assessment tool, like a checklist to identify ASD amongst children at high risk but yet to be diagnosed (Baron-Cohen et al., 1992). Research then helped to confirm that such a checklist could be successfully utilized at a larger scale, outside of a traditional research setting, by community-based practitioners (Baron-Cohen et al., 1996). Next, research sought to modify this checklist to further improve its validity, reliability, and ease of use (Robins et al., 2001). Later research demonstrated that this modified checklist could be successfully translated for use across a range of languages and cultures (Albores-Gallo et al., 2012; Canal-Bedia et al., 2011; Inada et al., 2011) and incorporated into assessment protocols commonly used by community-based practitioners (Glascoe et al., 2007). Three decades after specifically relevant basic research had been first published, new studies have begun to show how such a checklist might increase access to early intervention when combined with other policy changes (Rotholz et al., 2017).

Research roadmaps like those modelled here should be situated within larger roadmaps that integrate research with services, training, policy, and advocacy, and that translate all of this into action at the local regional, state, and national level (Doehring, 2013). We have applied these roadmaps to a fictional case study of a young man in behavioral crisis, to illustrate how breakdowns in the services and supports available in local, regional, and state agencies can lead to a potentially lifelong but completely preventable placement in an institutional setting (Doehring, 2014). We have also explored the interface between research and policy at the national level (Doehring, 2019b), and found no examples of research undertaken in an intentional, coordinated manner to influence national or even state policy.

Several important limitations of the present analyses are worth noting here. For example, this review focused on publications in JADD. This was necessary given that JADD was the only journal focused on ASD during the time period studied, since the publication of the DSM-III. There are other journals that have since emerged, however, that also focus on ASD, that include research on other populations that sometimes overlap with ASD (e.g., people with I/DD), and that have published research involving people with ASD from the perspective of specific disciplines (Behavior Analysis, Pediatrics, Education, Public Health, and so on). It will be important for future research to confirm whether these same patterns hold true regardless of the journal in which this research was published.

It is legitimate to question whether the percentage of publications adequately captures the arc of science. Two publications might vary greatly with respect to the number of participants, and the significance of the results. Changing publication standards could confound comparisons across different time points, although many of the patterns noted here remained relatively consistent over the decades. The number of citations for a given class of studies might offer a different metric, capturing what researchers are interested in. To the extent that research depends heavily on funding, and that funding is generally directed towards the proposals determined to be most likely to advance understanding, analyses of patterns of funding across different types of research should reflect what researchers and policymakers believe is most important and are ready to invest in. We hope that the present study will spur a debate about each of these potential metrics for the impact of science.

These roadmaps, together with the findings of the present review, raise important questions about research publications, research funding, and the training of researchers that might be answered through future study. How often is basic research conducted with a specific goal to improve assessment or intervention? To what extent are these efforts driven by researchers with broad clinical training and/or experience? How often does such research actually result in a testable clinical practice? How often do researchers seeking to improve community-based outcomes have experience delivering services in community-based programs? Such gaps in the types of training and experience of researchers can have impacts beyond the specific research itself, by shaping the culture and the structure of the research enterprise, and perhaps contributing to the gaps in intervention research noted above. Without such experience, for example, those reviewing intervention research studies for publication might inadvertently set unreasonable expectations, because they do not recognize challenges inherent to such research, especially when it is conducted in community settings. Likewise, those reviewing proposals for funding might penalize the clinical researcher whose publication record might be less impressive because of the time invested broadening their clinical expertise outside of a traditional research setting. Assigning greater importance to intervention research publications and real-world clinical experience and outcomes may increase incentives for the next generation of researchers to design programs of study to bend the arc of science back towards impact.