The Autism Impact Measure (AIM): Meaningful Change Thresholds and Core Symptom Changes Over One Year from an Online Survey in the U.S.

Autism spectrum disorder (ASD) is a neurodevelopmental condition that is categorized by deficits in core symptoms of social communication, social interaction, and restricted, repetitive behaviors (Diagnostic and Statistical Manual of Mental Disorders (DSM-5), 2013). The expression of both core and associated symptoms is broad and varied, ranging from mild to severe impairment (Diagnostic and Statistical Manual of Mental Disorders (DSM-5), 2013). The diagnosis of ASD worldwide is increasing, either because of changes in diagnostic criteria, changes in primary diagnosis, an actual increase in incidence, or a combination of the above (Polyak et al., 2015). A variety of treatments and interventions are under development. For these to be evaluated, validated outcome measures with the ability to show meaningful changes in the core characteristics of ASD are required. The commonly used measures to assess core ASD symptoms were originally designed for screening or diagnostic purposes, and, thus, are not well-suited for measuring short-term improvement or deterioration (Ghosh et al., 2013; Grzadzinski et al., 2020; Kanne et al., 2014). Development of psychometrically validated outcome measures and calculating clinically meaningful thresholds are key priorities for autism research (Bolte & Diehl, 2013).

The Autism Impact Measure (AIM) is a caregiver-reported questionnaire designed to characterize core symptoms of ASD in individuals aged 3–18 years (Kanne et al., 2014; Mazurek et al., 2018). Although the U.S. Food and Drug Administration (FDA) recommends that the individual experiencing the symptoms report these themselves, a parent or caregiver may also report on observations of the individual’s experience, if that individual is a child or lacks cognitive insight (Food & Drug Administration, 2018). The AIM measures frequency of symptom occurrence and associated functional impact, which are both important for understanding symptom severity and permitting prioritization of interventions (Kanne et al., 2014; Mazurek et al., 2018). Key advantages of the AIM include that it has been designed for use as an outcome measure specifically for ASD, is less time-consuming than other interview-administered measures, and, importantly, it does not require trained personnel for administration. Therefore, the AIM could also be a suitable tool for real-world monitoring of pediatric ASD symptomology (Houghton et al., 2019). Cross-sectional studies have demonstrated that AIM has good test–retest reliability, cross-informant reliability, and convergent validity with other measures of ASD symptoms (Houghton et al., 2019; Kanne et al., 2014). Five symptom domains have been identified in the assessment of the AIM’s structural validity: repetitive behavior, atypical behavior, communication, social reciprocity, and peer interaction (Mazurek et al., 2018). Furthermore, the AIM has been used to detect symptom improvements after short-term interventions (Mazurek et al., 2020).

Clinical outcome measures used to evaluate the effectiveness of treatments must not only be sensitive to symptom changes but should also be able to demonstrate the clinical meaningfulness of those changes and, thus, of potential treatment effects. Interpreting score changes and the clinical meaningfulness of these changes can be done either at the group level, called the clinically important difference (CID), or at the individual level, called the within-person meaningful change threshold (MCT) (Food & Drug Administration, 2018). The within-person MCT is the magnitude of change in a clinical outcome assessment that needs to be observed in order to interpret whether there has been a meaningful improvement or, in some indications, deterioration (Coon & Cappelleri, 2016; Revicki et al., 2008). Within-person MCTs can be calculated using approaches such as anchor-based methods and distribution-based methods (Coon & Cappelleri, 2016; Food & Drug Administration, 2018; Revicki et al., 2008). An advantage of the former is that it incorporates the patient or caregiver voice using data from an external indicator (anchor), whereas distribution-based methods, typically used for cross-sectional data, consider the properties of the measure itself, such as 0.5 or 0.2 standard deviations of the measure of interest at baseline or the standard error of the measurement (Coon & Cappelleri, 2016; Fayers & Hays, 2014; Food & Drug Administration, 2018; Norman et al., 2003; Revicki et al., 2008). FDA guidance references that approaches to detect meaningful change at the individual level (within-person) are preferred using anchor-based approaches (typically, the use of Global Impression of Severity or Change scales), supplemented with distribution-based approaches for cross-sectional data (Food & Drug Administration, 2018). Anchor-based approaches depend upon the quality of the anchor; therefore, when calculating the clinically meaningful threshold of an outcome measure, triangulation of multiple methods for estimating MCTs, which includes the use of multiple independent anchors, should be assessed (Revicki et al., 2008). Even following such approaches, it is worth noting that MCTs of the same measure can vary by population and context. Distribution-based MCTs for the AIM have been previously estimated from cross-sectional data from children with ASD aged 3–17 years (Houghton et al., 2019). Additionally, Mazurek et al. (Mazurek et al., 2020) looked at short-term (typically 6 weeks) changes in AIM scores after specific interventions, and their correlations to other commonly used outcome measures including the Aberrant Behavior Checklist, the Social Responsiveness Scale-2, and the Vineland Adaptive Behavior Scales-2. The primary objective of our study was to calculate preliminary within-person MCTs for the AIM using anchor-based approaches over a 12-month interval, in order to aid the interpretation of the AIM in future studies.

In order for an outcome measure to be used in a clinical trial setting as an endpoint, it is important to understand at what point an improvement or worsening is interpreted as clinically meaningful, commonly referred to as the within-person MCT (Food & Drug Administration, 2018; Grzadzinski et al., 2020; Rai et al., 2015). This study used Global Impression of Change anchors to estimate within-person MCTs of the AIM over 12 months. We found that, in relation to the overall caregiver perception of change, a reduction of 4.5 points or more in the AIM total score could be used as a threshold estimate for symptom improvement, whereas an increase of 9.9 points is the estimated threshold for symptom deterioration. Our study adds to the body of work looking to understand the meaningfulness of AIM score change over time, by examining changes in a real-world cohort over an intermediate timeframe. The AIM is a simple caregiver questionnaire designed to capture core symptoms of ASD in children that can be used to measure the effectiveness of new interventions (Mazurek et al., 2020). While estimates of meaningful change can be derived from cross-sectional data, especially when longitudinal data are absent, evaluating AIM scores over time is critical to providing a robust and comprehensive threshold estimate. Previous longitudinal studies of ASD outcome measures were limited to regional populations where outcome data were available, generally not conducted to determine minimally clinical effect sizes of a specific outcome measure (Chatham et al., 2018; Grzadzinski et al., 2020; Pickles et al., 2020).

Our results are drawn from a large sample (N = 2,761) with high between-subject variability, circumstances that translate into very small but nonetheless highly significant correlations between AIM score change and CaGI-C (r_Spearman ~ 0.10, p < 0.001). The overall pattern of median AIM score change and categories of CaGI-C is, however, indicative of a direct but variable relationship between AIM score change and CaGI-C. Change scores lay noticeably below zero for CaGI-C categories reporting ASD improvement, incrementally increasing up to categories denoting ASD deterioration (Figs. 2, S1–S4). The correlation effect size indicates that the relationship is not strongly linear, but neither would it be expected to be a priori. The median of the AIM change score for those children whose caregivers reported no change during the year was 8 points (IQR –30, 9), suggesting that the children as a group experienced an improvement in symptoms over the year, which did not translate to the less granular CaGI-C anchor.

The AIM total score thresholds calculated here are based on a real-world cohort, with typically heterogeneous treatment plans and life circumstances. Using a multivariable analysis, we identified factors in this cohort that were consistently associated with AIM score change and MCT response status, co-factors that were identified as contributing to caregiver strain by Durán-Pacheco in the same cohort (Durán-Pacheco et al., 2022). Higher ASD symptom severity, as indicated by higher baseline AIM score, was related to important reductions of AIM score over time, and therefore increased odds of MCT response for improvement and reduced odds of MCT response for deterioration. Consistent with other papers where the association between verbosity and ASD severity has been demonstrated (Mayo et al., 2013; Nevill et al., 2019), a child’s ability to speak was associated with a decrease of AIM scores over time, and thus to increased odds of being a responder for improvement. Conversely, a child straying from or eloping was related to a noticeable increase of AIM score and consequently to increased odds of experiencing a meaningful worsening. Our results are consistent with another study that has shown that severity is related to the potential for symptom improvement, with less severe cases having higher potential for long-term improvement (Mayo et al., 2013).

Our study relied on a single anchor to evaluate the potential of the AIM to capture change over intermediate time scales. In a recent study designed to assess the ability of the AIM to detect changes identified by multiple other measures of ASD, the AIM was administered repeatedly over 6-week intervals throughout the duration of various treatments, varying from two to four follow-up visits depending on the treatment (Mazurek et al., 2020). Multiple repeated measures allowed Mazurek to account for both within- and between-subject variability and found that the model estimates of slope were similar to those found in the current study. Mazurek found that the total AIM score and AIM domain scores, with the exception of repetitive behavior, were sensitive to change overall and as a function of different treatment conditions (Mazurek et al., 2020). We also found that the AIM repetitive behavior domain was less sensitive to change compared with the other domains, suggesting this concept may not be of primary consideration when caregivers respond to the global anchor, or that repetitive behaviors in the sample remained relatively stable.

Generally, we found that the magnitude of an improvement threshold was half that of the respective deterioration threshold. Sensitivity analysis that assumed equidistance (where improvement and deterioration were not modeled separately) corroborated this result; estimates yielded were of magnitudes halfway between those for improvement and deterioration MCTs. Any natural bias in the AIM score was explored by summarizing the change in AIM score for those children whose caregivers reported no change in overall ASD during the previous year. One would expect that these differences, in the event of no bias, or no overall growth, would average around 0. However, we found that change trended towards improvement. The improvement/deterioration asymmetry may be a symptom of reporter bias: parents may be more inclined to report an improvement or confirm the status quo rather than report a deterioration in their child’s ASD. Alternatively, it could be that the AIM is constructed to be more sensitive to improvement rather than to deterioration of ASD symptoms.

In order to put the estimates from our primary methodology into context, we performed a variety of sensitivity analyses. We explored the impact of a larger “effect size” and, in line with our hypothesis, the MCTs derived for “much improvement” versus “much deterioration” were larger than those derived for the primary analysis. These larger thresholds (in addition to the main thresholds) are valuable data to help inform the relevance of AIM results in future studies. Optimal thresholds derived by simultaneously maximizing the sensitivity and specificity of potential thresholds using the ROC further supported our results, by producing results in the same direction and of a similar magnitude as those derived in our main analysis (a decrease of 7.5 [4.5, 14.5] indicating improvement and an increase of 10.5 [4.5, 11.5] indicating deterioration).

In order to examine change thresholds among domains, the percent change thresholds are required, since the range of possible domain scores varies. Likewise, the percent change thresholds reported allow the utility of the primary and secondary anchors applied to AIM domain scores for communication, peer interaction, and social reciprocity to be compared. Further studies relating AIM to other ASD outcome scales over diverse time scales will be needed to explore these relationships in more detail.

Baseline characteristics and demographics of the cohort children with ASD in this study showed similarity to children with ASD in the 2009–2010 National Survey of Children with Special Health Care Needs (NS-CSHCN) (Data Resource Center for Child & Adolescent Health, 2018), indicating that this cohort is broadly representative of children with ASD in the U.S., with the exception of a higher level of education for caregivers in this study.

Designing a study to calibrate AIM score changes presents several challenges, but as Mazurek noted, the greatest challenge is that there is no “established gold standard measure of true change in ASD symptoms within or across domains” (Mazurek et al., 2020). Although the electronic surveys were designed to limit missing data, overall, around 16% of the sample data needed to be excluded due to missing or incomplete items on the AIM either at baseline or follow-up. The majority of the surveys that were excluded were missing responses to a single item on the AIM questionnaire. A consideration for future studies is to explore suitable methods to imputing missing items to avoid excluding data. Other limitations are those typical of retrospective data collection. We tried to reduce caregivers’ recall bias by restricting the recall period for the two administrations of AIM survey to the last 2 weeks. Most characteristics of the caregivers and dependents with ASD were collected at baseline, and we assumed that demographic characteristics were stable during the study period.

This study is the largest nation-wide study, to our knowledge, to assess mid-term changes in AIM scores. The digital questionnaire employed in the study minimized the extent of data that might be missing, increasing their utility. Furthermore, these data, which contribute to the expanding data on the SPARK cohort, are publicly available from the Simons Foundation for Future Research by the ASD research community.

Among a large sample of children with ASD in the United States, anchor-based, within-person MCTs for 12-month changes in AIM scores have been estimated. Whilst we acknowledge future work is needed to confirm these findings, such as triangulation of multiple approaches, this analysis represents important work in order to interpret changes of a relatively new measure. The preliminary MCTs presented will be relevant for both intervention development and, once approved, post-marketing of interventions to address core symptoms of ASD.