Effectiveness of Responsivity Intervention Strategies on Prelinguistic and Language Outcomes for Children with Autism Spectrum Disorder: A Systematic Review and Meta-Analysis of Group and Single Case Studies

Proximal outcome measures assess skills taught directly during the intervention. Distal outcome measures assess skills beyond what was taught directly. As predicted, Yoder et al. (2013) found significantly greater probability of an effect on social communication for proximal outcome measures (63%) than distal outcome measures (39%) for children with ASD.

Boundedness of outcome measures refers to the degree to which the occurrence of the outcome behavior depends on the intervention context (e.g., same setting, materials, and/or communication partner; Yoder et al., 2013). Context-bound outcome measures are measured in situations very similar to the treatment sessions (e.g., evaluating the number of intentional communication acts during treatment sessions with the interventionist). In contrast, generalized characteristics are measured in situations that vary from the treatment context in setting, materials, and/or communication partner (e.g., number of intentional communication acts with an unfamiliar clinician during a session in which the intervention strategies are not used). Potentially context-bound outcome measures may show changes that are possibly limited to the treatment context (e.g., standardized caregiver report measure for a caregiver-implemented intervention). Yoder et al. (2013) found greater probability of a significant effect on social communication for context-bound outcome measures (82%) than generalized characteristics (33%). Boundedness also moderated the mean effect size for the effectiveness of early intervention on social communication skills of children with ASD (Fuller & Kaiser, 2020).

Correlated measurement error (CME) systematically elevates the true score for the predicted superior group or phase over the control group or phase (Yoder et al., 2018). Intervention studies are at risk for CME (a) when the outcome measure coder is not blind to treatment assignment and (b) when interventionists (including caregivers) provide the intervention and serve as the examiner when the outcome measure is assessed.

Publication bias occurs “when published research on a topic is systematically unrepresentative of the population of completed studies on that topic” (Rothstein, 2008, p. 61). We test for this known risk for meta-analyses by comparing effect sizes of published versus unpublished studies. This examination is a feature of well-designed meta-analyses.

Our comprehensive search strategy included multiple search methods. The main search utilized electronic databases. We searched PubMed on October 18, 2019 and the Education Database, ERIC, Health & Medical Collection, Linguistics and Language Behavior Abstracts, Linguistics Database, ProQuest Dissertations & Theses Global, Psychology Database, PsycINFO, and Social Science Database in ProQuest and the Cumulative Index of Nursing and Allied Health Literature (CINAHL) on October 19, 2019. See Supplementary Information 1 for an example search.

For supplementary searches, the first author hand searched table of contents for the past year for journals that contributed at least five articles to the full text screening from the main database search (i.e., Autism, Journal of Autism and Developmental Disorders, Journal of Child Psychology and Psychiatry). The first author also screened abstracts from the two prior conferences for the Gatlinburg Conference on Intellectual and Developmental Disabilities, International Meeting for Autism Research, and Society for Research in Child Development to identify findings that may not yet be in publication. Finally, the first author scanned reference lists and conducted forward searches for included studies. The supplementary searches were completed on March 28, 2020.

The primary coder (first author) screened 100% of the identified reports. Trained research assistants independently screened 25% of the reports at the title and abstract level and the full text level. The primary coder (first author) was blind to which reports would be coded for reliability. To prevent coder drift, discrepancy discussions were completed regularly. Point-by-point agreement for inclusion or exclusion (i.e., agreements divided by total number of reports) was 89% at the title and abstract level and 87% at the full text level. We used the primary coder’s decisions for inclusion.

To maintain an appropriately narrow focus, literacy, vocal stereotypy, and challenging behavior outcomes were excluded. We also excluded outcome measures that focused on the interventionist’s performance (e.g., number of adult conversational turns, prompts to the child, or use of intervention strategies). We excluded studies not written in English due to lack of translation resources. Studies were not excluded based on the language of the participants or the publication date. At the final stage of the full text screening, we excluded SCRD studies that failed to meet quality standards from the qualitative and quantitative analyses because failing to meet those standards prevents interpretation of the findings (What Works Clearinghouse, 2016). Broadly, the following criteria must be met to demonstrate an intervention effect: (a) graphical display of the data, (b) at least three attempts to demonstrate an effect and (c) a sufficient number of data points per phase (e.g., at least three data points per phase to meet with reservations and at least five data points per phase to meet without reservations for multiple baseline, multiple probe, and ABAB [reversal/withdrawal] designs; What Works Clearinghouse, 2016). Multiple baseline and multiple probe designs must also have sufficiently overlapping baselines across tiers. Failure to meet all these criteria resulted in exclusion from the qualitative and quantitative analyses. For additional details, refer to the What Works Clearinghouse Study Review Guide Instructions for Reviewing Single-Case Designs Studies (What Works Clearinghouse, 2016).

As shown in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (Fig. 1), database searches yielded 7108 records and other sources yielded 149 records. After eliminating duplicates and screening the titles and abstracts, 770 records remained. During the full text screening, independent coders eliminated studies in the order listed in Fig. 1. For the SCRD studies, the final inclusion criterion was meeting quality standards with or without reservations. The search yielded 33 RCTs that were described in 45 reports and included 294 relevant effect sizes and 42 SCRD studies that were described in 47 reports. Thirty-seven SCRD studies included sufficient graphical information for visual analysis (91 relevant opportunities to detect a functional relation) and 34 permitted extractions of at least one BC-SMD effect size (69 total BC-SMD effect sizes).

All reports were coded by the primary coder (first author) and trained research assistants using a detailed coding manual (available from first author upon request). For the RCTs, point-by-point agreement for data extraction and bias coding was 94% and 90%, respectively. For the SCRD studies, point-by-point agreement for quality coding, data extraction, and visual analysis was 80%, 92%, and 80% respectively. Discrepancies were resolved by consensus. Consensus coding was used for all analyses.

Report level features included publication status, report type, country, spoken language of the participants, and percent of participants who are monolingual. Effect size level features included sample size (for ASD group and control group for RCTs), sex, age, intervention, interventionist, time in intervention, outcome measure(s), and effect size. The total time in intervention is the number of minutes per week multiplied by the number of weeks of intervention. For caregiver-implemented interventions, the amount of time is based on structured intervention time, not all waking hours, even though a caregiver may implement at least some strategies throughout the entire day. We categorized the outcome measures as distal or proximal and context-bound, potentially context-bound, or a generalized characteristic.

For risk of bias for RCTs, we used the Revised Cochrane risk-of-bias tool for randomized trials (RoB 2; Higgins et al., 2019). We rated each study for low, moderate, or high risk of bias for randomization process, deviations from intended interventions, missing outcome data, measurement of outcome, selection of reported result, and overall. In addition, coded study quality features include risk for CME, method of handling missing data, and use of blind assessors. For quality coding for the SCRD studies we used guidelines provided by the What Works Clearinghouse Study Review Guide Instructions for Reviewing Single-Case Designs Studies (What Works Clearinghouse, 2016). Studies that did not meet quality standards with or without reservations were excluded from the meta-analysis. Remaining studies were categorized as meeting standards with versus without reservations.

For the RCTs, Tables 2 and 3 display participant characteristics and intervention features. At least 897 unique participants (accounting for possible overlap between studies) are included in at least one effect size. Participants’ mean age at study initiation was 43.01 months (SD = 17.97 months). A variety of interventions were implemented. Joint attention intervention / JASPER (8 studies) and PRT (6 studies) were most common. JASPER targets joint attention, play, and imitation through a combination of behavioral and developmental principles (Chang et al., 2016; Goods et al., 2013; Kasari et al., 2006). PRT is designed to target “pivotal” areas using ABA principles and to train caregivers in the strategies to do so (Hardan et al., 2015). Caregivers were the most common interventionist (17 studies). Table 4 displays details for effect size features, including outcome measures. The included studies used a wide variety of outcome measures and varied in the number of effect sizes per study, ranging from 1 to 78.

For the SCRD studies, Tables 5 and 6 display participant characteristics and intervention features. The studies included at least 143 unique participants with an average of 3.40 participants per study (SD = 1.17). Only participants who contributed data included for visual analysis or an effect size were included. The mean age of participants prior to intervention was 54.36 months (SD = 25.72). Table 7 displays effect size features (e.g., outcome measures and results) and visual analysis results.

For the RCTs, overall risk of bias was judged to be high for 25 studies, moderate for 7 studies, and low for only 1 study. It should be noted that a “high” risk of bias rating for any category results in an overall risk of bias rating of “high”. See Tables 8 and 9 for details. Many studies were noted to be at risk for CME which resulted in high risk of bias for “Measurement of outcome.” Only seven of the RCTs provided sufficient information to determine whether there were deviations from the intended intervention (a component of “Deviations from intended interventions”). Of those, three indicated probable risk of bias. Studies were judged to deviate from the intended intervention if the mean or median procedural fidelity value was below 80%. “Sufficient information” required that procedural fidelity data to be drawn from at least 20% of sessions or participants. Similar gaps in reporting of outcome measure reliability were also observed, as shown in Table 9. The high number of studies without sufficient information about procedural fidelity and reliability reveals an area of need for improving the quality of available studies. It inhibits quantitative analysis of the influence of procedural fidelity and reliability on intervention effects. No studies were at high risk of bias for the randomization process and only three were at high risk for deviations from the intended interventions.

For the SCRD studies, only three studies included at least one distal outcome measure (Carpenter, 2003; Ingersoll & Wainer, 2003, 2013). All outcome measures were context-bound and at risk for CME. The high proportion of proximal and context-bound outcome measures is consistent with Yoder et al. (2013). For study quality, 25 of the 47 included reports met quality standards without reservations. As shown in Fig. 1, 44 reports that otherwise would have been included were excluded due to failing to meet quality standards (listed in Supplementary Information 2). Twenty-three SCRDs provided some type of summary value for procedural fidelity of the interventionist (see Table 10). Nine additional SCRD studies provided fidelity data for the interventionists, but not in a summative form (e.g., graphically or narrative description). However, only two studies (Randolph et al., 2011; Vogler-Elias, 2009) reported procedural fidelity data for the trainers (e.g., a trainer who taught a caregiver to implement the intervention). The ten remaining SCRD studies did not report procedural fidelity data. Similar to the RCTs, the gaps in reporting of procedural fidelity reveal an area of need for improving the quality of available studies. Relative to procedural fidelity data, the SCRD studies more consistently reported interobserver agreement (IOA) data for the outcome measure. Only one study omitted IOA data, revealing an area of strength for the included studies.

We reject the null hypothesis that there is no effect of interventions using responsivity strategies on prelinguistic and language skills of children with ASD for the RCTs and SCRD studies (research question 1). For the RCTs, the mean standardized group difference is g = 0.36, 95% CI [0.21, 0.51], which is a moderate effect size. No variation in the weighted mean effect sizes were observed when we varied the p value in Stata at 0.1 increments from 0.0 to 0.9.

For the SCRD studies, the mean BC-SMD = 1.20, 95% CI [0.87, 1.54], which is large. No variation in the weighted mean effect sizes were observed when we varied the p value in Stata at 0.1 increments from 0.0 to 0.9. The difference in mean effect size between the RCTs and SCRDs may be due to methodological differences between group and SCRD studies. Thus, the effect sizes are not directly comparable between the RCTs and the SCRD studies. Relatively large effect sizes are easier to detect through visual analysis and may explain the publication bias toward studies with larger effects for SCRD studies (Shadish et al., 2015, 2016). As described in the Moderator Analyses section, we did identify evidence of publication bias. Other meta-analyses that combine group and SCRD studies have also reported relatively larger mean effect sizes for SCRD studies (Barton et al., 2017). Based on visual analysis of the SCRD studies, 41 graphs (45%) showed strong evidence, two (2%) showed moderate evidence, and 48 (53%) showed no evidence of a functional relation between the intervention with responsivity strategies and child prelinguistic and/or language skills. Opportunities to show a functional relation that showed strong evidence had a mean BC-SMD of 2.34 (SD = 2.18, range: 0.56 – 5.36). Those that showed no evidence had a mean BC-SMD of 0.67 (SD = 0.49, range: -0.20 – 2.69).

The moderator analyses address our second research question about whether particular study features account for the observed heterogeneity. RCTs and SCRD studies were analyzed separately.

The Galbraith plots (Figs. 2 and 3) and τ² values (0.19 and 0.67 for RCTs and SRCD designs, respectively) all provide evidence of substantial heterogeneity. We define heterogeneity as variation in estimated ‘true effects’ (Borenstein et al., 2009). This variation is differentiated from that due to spurious error in the computation of τ² by considering the ratio of observed to expected variation across studies. The results show that there is notable dispersion of the effect sizes that is assumed to be real rather than spurious error. The larger τ² value for SRCD studies than RCTs indicates greater dispersion in true effects for the SRCD studies than the RCTs. The Galbraith plot, which is an alternative to the forest plot for meta-analyses with a large number of effect sizes, displays more precise estimates further from the origin. The large number of effect sizes outside of the two parallel outer lines that represent that 95% confidence interval indicates substantial heterogeneity (Anzures-Cabrera & Higgins, 2010).

Based on 294 effect sizes from 33 RCTs and 69 effect sizes from 34 SCRD studies that included a total of 1040 participants, the weighted mean effect size of the effect of interventions using responsivity intervention strategies on child prelinguistic and language outcomes is moderate to large. The identified mean effect size (g = 0.36) is somewhat larger than that identified by Hampton and Kaiser (2016; g = 0.26) and Sandbank et al., (2020b; g = 0.13 for receptive language; g = 0.18 for expressive language;), which evaluated a wider variety of interventions on language outcomes. Visual analysis of 91 opportunities to demonstrate a functional relation from 37 SCRD studies provided somewhat weaker support, characterized by 45% of opportunities showing strong support, 2% showing moderate support, and 53% showing no support for the interventions improving child prelinguistic and/or language outcomes. Thus, heterogeneity in results is apparent through the effect sizes and visual analysis. Although the mean effect size for the SCRD studies was large, a nearly even split between “strong” and “no” evidence offers reason for caution in interpreting the results. Because many of the studies used a multiple baseline across participants design with three participants, the presence or absence of an effect for each participant could have a large impact on the overall judgment of a functional relation. In addition, the magnitude of effect sizes cannot be compared directly between the RCTs and SCRD studies due to methodological differences in study types.

Moderator analyses revealed that effect sizes using context-bound outcome measures had a larger mean effect size than those with potentially context-bound or generalized outcome measures for the RCTs. This finding is consistent with those reported by Yoder et al. (2013) and Fuller and Kaiser (2020) for social communication outcomes in children with ASD. In addition, RCTs at risk for CME exhibited a significant, positive effect size, but those free from CME risk did not. These results for the role of boundness and CME risk could not be replicated for the SCRD studies because all of the SCRD study effect sizes were context-bound and at risk for CME. Although some of the SCRD studies did include generalization probes (e.g., with a different communication partner or setting), in the vast majority of cases probes were not frequent enough to meet quality standards for inclusion. Similarly, very few effect sizes from the SCRD studies included distal outcome measures. Thus, proximity of the outcome measure could not be tested for the SCRD studies. Results for the RCTs and SCRD were consistent for publication bias being identified by the Egger’s test but not the moderator analysis. The relatively small number of unpublished studies for both types of studies limited the moderator analysis. For both RCTs and SCRD studies, we were unable to test for a moderating effect of time in intervention, despite attempts to use multiple intensity variables. Too few studies included key details about intensity in the included reports.

In sum, these findings provide support for the use of responsivity intervention strategies for children with ASD for improving prelinguistic and language skills. As expected, findings are more robust for context-bound outcome measures than other types of outcome measures (e.g., potentially context-bound and generalized characteristics).

Limitations for meta-analyses are influenced by primary study level characteristics as well as meta-analytic level characteristics. For the current study, imprecise reporting at the primary study level, especially for the study’s intensity, limited analyses. Despite calls for improved reporting of intervention details, only about half of the RCTs and SCRD studies provided sufficient information to determine the time in intervention, a less precise variable than the cumulative intervention intensity (Warren et al., 2007). Given the potential importance of treatment intensity for the effectiveness, cost (financial and time), and feasibility of services for children, reports of future studies would be strengthened by explicit descriptions of intensity variables. Concerns of study quality of the included studies also influenced the meta-analysis. Forty-four SCRD design studies that would have otherwise been included were excluded because they failed to meet What Works Clearinghouse standards. Risk for CME was very common across included studies and should be attended to in future studies to minimize risk for bias.

The use of meta-analytic analyses for SCRD studies is a relatively new and still developing area. As a result, only studies that used a multiple baseline across participants design were able to be included in the current quantitative meta-analysis. Future meta-analyses on responsivity intervention strategies should be considered as other analytic approaches develop. Other limitations at the meta-analytic level include the potential failure to include relevant effect sizes and only including studies written in English. The risk of missing relevant effect sizes was minimized through multiple supplementary searches and completion of reliability checks at all screening levels. Lastly, robust variance estimation is most effective with at least 40 studies. Our analyses using robust variance estimation included 33 RCTs and 34 SCRD studies.

Although our searches yielded effect sizes from fewer than 40 RCTs or SCRD studies, the use of robust variance estimation remains a strength of this meta-analysis. Robust variance estimation permits the inclusion of multiple effect sizes per study, which eliminates the loss of potentially important effect sizes. Second, we include both RCTs and SCRD studies, which is currently rare for systematic reviews and meta-analyses. This approach provides a more comprehensive review of the current literature base and opportunities for replication across the two study types. Third, we enhanced the quality of this meta-analysis by conducting interrater reliability for all screening levels and having two independent coders extract data (including risk of bias) for all included reports. Fourth, we considered the quality of the included studies through multiple avenues. We not only required studies to meet certain characteristics to be included, but also coded for study quality features including risk for bias and CME.

This systematic review and meta-analysis provides empirical support for the use of responsivity intervention strategies to improve prelinguistic and language skills of children with ASD. Because the data are more robust for context-bound outcome measures (e.g., behaviors that occur during the intervention or a very similar setting) than generalized characteristics (e.g., use of targeted skills in a novel setting with someone other than the interventionist), gains in generalized characteristics should be monitored closely during clinical practice. The observed benefits of responsivity intervention strategies were observed for a wide variety of outcome measures (e.g., joint attention, use of gestures, verbal utterances, and vocalizations), which suggests that these strategies have broad application including both prelinguistic and early language skills.

Additional, high-quality intervention studies regarding the observed benefits of responsivity intervention strategies are needed to further delineate the specific impact of these strategies and how features of such interventions can be adjusted to maximize gains. At the primary study level, future studies would be enhanced by continued improvement of study quality, especially minimizing risk for bias and CME, and more explicit reporting of putative moderators of treatment effects.

The need to report intensity data was especially apparent. Not only is such data needed to determine whether more intensive intervention is likely to have positive or negative effects on child outcomes, but also to control for intensity when investigating the role of other putative moderators, such as interventionist. Explicit reporting will improve the effectiveness of future meta-analytic moderator analyses. Primary studies that directly address the effect of intensity on intervention are also needed.

Continued inclusion of distal outcome measures in coordination with proximal measures is also warranted. Explicitly identifying outcomes as proximal versus distal will allow readers to accurately weigh the results. Because distal measures are expected to yield smaller effect sizes than proximal measures, achieving a relatively large effect size for a distal measure should be noted. SCRD studies can be used within a programmatic line of research to guide selection of outcome measures in RCTs. For example, an SCRD may include some generalization and maintenance data with sufficient data points to determine whether those dependent variables may be suitable distal and/or generalized characteristic outcome measures for a subsequent RCT. The evidence base would also benefit from the inclusion of studies that provide specific responsivity intervention strategies outside of large treatment packages as well as explicit descriptions of strategies implemented. Such evidence would facilitate ongoing efforts to identify active ingredients of interventions and inform modifications aimed at increasing effectiveness.