Although ASD symptoms are already identifiable by 18 months of age (Chericoni et al.,
2021), the diagnosis is often delayed until elementary school. For instance, in their study examining data of 2134 children included in two large family databases, Brett et al. (
2016) found that the median age at ASD diagnosis in the UK was 4.5 years, which was comparable to the situation in the US. Delayed ASD diagnosis particularly pertains to children with ASD and cognitive functioning in the normal range, who are frequently diagnosed as late as during middle school. This is illustrated in the review of Daniels and Mandell (
2014), who found that the median age at which children were diagnosed with Asperger’s disorder ranged from 7.4 to 11.2 years. Also, in many of these children, ASD may not be recognized at all. One of the studies supporting this claim was that of Kim et al. (
2011), who screened for ASD in 55,266 South-Korean children, aged 7 to 12 years, in a sample consisting of both a high-probability group (drawn from a disability registry and special education schools) and a low-probability group (drawn from regular schools). They found an estimated ASD prevalence of 1.89% in the low probability group, as opposed to 0.75% in the high-probability group. This meant that two-thirds of the ASD cases came from the mainstream population—undiagnosed and untreated. Also, in their study examining case records of 2,867 children aged 6 to 12 years, who were registered at the Maccabi Child Development Center (MCDC) in Israel, Davidovitch et al. (
2015) found that 221 children were diagnosed with ASD after the age of 6—even though their initial developmental evaluation (before the age of 6) came out negative for ASD. A delayed or missed ASD diagnosis has been associated with several factors, such as comorbid classifications (Supekar et al.,
2017), the heterogeneity of ASD symptom composition, lower severity of ASD symptoms, lower levels of impairment (e.g., less language/communication deficits, less support needed by the child; less parental concern about initial symptoms; Daniels & Mandell,
2014), children’s ability to mask symptoms with learned strategies (APA,
2013), and the lack of adequate screening practices (Self et al.,
2015). Also, detection as early as possible might prevent negative outcomes for both children (e.g., rejection by peers, harsh treatment by teachers, and inappropriate education) and their parents (e.g., frustration due to being told that there is nothing wrong with their child and not receiving appropriate support; Howlin & Asgharian,
1999). Therefore, routine-wise screening for ASD in the school-aged population is of great importance. In both community-based and clinical settings, the parent-rated school-age Child Behavior CheckList (CBCL 6-18; Achenbach & Rescorla,
2001) is an internationally used, reliable, and valid primary screening method for emotional, behavioral, and social problems in children aged 6 to 18 years (Rescorla et al.,
2007). As it has been argued that this instrument contains several items that describe problem behaviors typical for children with ASD, several researchers have begun to investigate the ability of the school-age CBCL to identify childhood and adolescent ASD.
Initially, the capability of the school-age CBCL syndrome subscales—and specifically the Withdrawn/Depressed, Thought Problems, and Social Problems subscales—were examined to differentiate between children with and without ASD (e.g., Bölte et al.
1999; Duarte et al.
2003; Hoffmann et al.,
2016). In two later studies, the discriminative power of certain combinations of syndrome subscales was investigated, which yielded the ASD profile (a combination of the Withdrawn/Depressed, Thought Problems, and Social Problems syndrome subscales; Biederman et al.,
2010) and the WTP subscale (a combination of the Withdrawn/Depressed and Thought Problems syndrome subscales; Havdahl et al.,
2016). Furthermore, two attempts have been made to develop a specific subscale, consisting of separate items to screen for ASD (Ooi et al.,
2011; So et al.,
2013). An overview of all ASD subscales derived from the school-age CBCL is presented in Table
1. Recently, Deckers et al. (
2020) validated these ASD subscales in a sample of 132 children aged 6 to 18 years, of whom 75 were diagnosed with ASD (and cognitive functioning in the normal range) and 57 had another classification. The specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013) were shown to have the best potential to distinguish between children with and without ASD.
Table 1
Items of the previously developed—i.e., the ASD profile, the WTP subscale, the specific ASD subscale by Ooi et al. (
2011), and the specific ASD subscale by So et al. (
2013)—and currently developed—i.e., the specific data-driven ASD subscale and clinician-expert ASD subscale—CBCL 6-18 subscales to screen for ASD
There is little that he/she enjoys | Cannot get his/her mind off certain thoughts; obsessions | Clings to adults or too dependent | Acts too young for his/her aged | Acts too young for his/her aged | Acts too young for his/her aged*^ | Acts too young for his/her aged*^# |
Would rather be alone than with others | Deliberately harms self or attempts suicide | Complains of loneliness | Doesn’t get along with other kidsc | Cannot get his/her mind off certain thoughts; obsessionsb | There is little that he/she enjoysa | There is little that he/she enjoysa# |
Refuses to talk | Hears sounds or voices that are not there | Doesn’t get along with other kids | Fears certain animals, situations, or places other than school5 | Daydreams or gets lost in his/her thoughtsd | Cannot get his/her mind off certain thoughts; obsessionsb^ | Bowel movements outside toiletg |
Secretive, keeps things to self | Nervous movements or twitching | Easily jealous | Would rather be alone than with othersa | Would rather be alone than with othersa | Clings to adults or too dependentc | Cannot get his/her mind off certain thoughts; obsessionsb^# |
Too shy or timid | Picks nose, skin, or other parts of body | Feels others are out to get him/her | Nervous movements or twitchinge | Poorly coordinated or clumsyc | Daydreams or gets lost in his/her thoughtsd^ | Daydreams or gets lost in his/her thoughtsd^# |
Underactive, slow moving, or lacks energy | Plays with own sex parts in public | Gets hurt a lot | Repeats certain acts over and over; compulsionsb | Repeats certain acts over and over; compulsionsb | Doesn’t get along with other kidsc,* | Doesn’t eat wellg |
Unhappy, sad, or depressed | Plays with own sex parts too much | Gets teased a lot | Speech problemc | Speech
problemc | Would rather be alone than with othersa*^ | Doesn’t get along with other kidsc*# |
Withdrawn, doesn’t get involved with others | Repeats certain acts over and over; compulsions | Not liked by other kids | Strange behaviorb | Stares blanklyd | Poorly coordinated or clumsyc^ | Fears certain animals, situations, or places other than schoole* |
| Sees things that are not there | Poorly coordinated or clumsy | Withdrawn, doesn’t get involved with othersa | Strange behaviorb | Prefers being with younger kidsc | Gets teased a lotc |
| Sleeps less than most kids | Prefers being with younger kids | | Withdrawn, doesn’t get involved with othersa | Repeats certain acts over and over; compulsionsb*^ | Would rather be alone than with othersa*^# |
| Stores up too many things he/she does not need | Speech problem | | | Too shy or timida | Not liked by other kidsc |
| Strange behavior | | | | Stares blanklyd^ | Prefers being with younger kidsc# |
| Strange ideas | | | | Strange behaviorb*^ | Repeats certain acts over and over; compulsionsb*^# |
| Talks or walks in sleep | | | | Underactive, slow moving, or lacks energya | Secretive, keeps things to selfa |
| Trouble sleeping | | | | Withdrawn, doesn’t get involved with othersa*^ | Too shy or timida# |
| | | | | | Stares blanklyd^# |
| | | | | | Stores up too many things he/she does not needb |
| | | | | | Strange behaviorb*^# |
| | | | | | Strange ideasb |
| | | | | | Stubborn, sullen, or irritablef |
| | | | | | Sudden changes in mood or feelingsf |
| | | | | | Temper tantrums or hot temperf |
| | | | | | Withdrawn, doesn’t get involved with othersa*^# |
The specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013) are comparable to the six DSM-oriented subscales (including Affective Problems, Anxiety Problems, Somatic Problems, Attention Deficit/Hyperactivity Problems, Oppositional Defiant Problems, and Conduct Problems) that have been developed by the Achenbach System of Empirically Based Assessment (ASEBA) for the school-age CBCL (Achenbach et al.,
2001). These DSM-oriented subscales contain separate items that were rated by at least 14 out of 22 internationally recruited clinicians as being very consistent (opposed to not or somewhat consistent) with the concerning DSM IV-TR classification (e.g., Oppositional Defiant Disorder) or category (e.g., anxiety disorders). In a similar way, for the preschool-age version of the CBCL (CBCL 1.5-5), ASEBA has developed a DSM-oriented Autism Spectrum Problems subscale (Achenbach et al.,
2000). However, such a DSM-oriented subscale for the school-age version of the CBCL by ASEBA is lacking. Also, the specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013) have not been implemented in clinical practice. This might be due to limitations of the previous studies examining the ASD subscales for the school-age CBCL. First, in many studies, a small sample size (e.g., Biederman et al.,
2010; Deckers et al.,
2020; Havdahl et al.,
2016) or a specific clinical control group (e.g., Ooi et al.,
2011) was used, hampering generalizability of results. Second, in most studies, an additional sample to cross-validate results (i.e., to explore whether results generalize to an independent data set) was lacking (Biederman et al.
2010; Havdahl et al.,
2016; Ooi et al.,
2011) or the sample was split instead of including a truly independent cross-validation sample (So et al.,
2013). Third, none of the studies differentiated between girls and boys or children and adolescents, nor did these include comparisons to the screening potential of ASEBA’s DSM-oriented subscales.
Discussion
In this study, we used a data-driven and a clinician-expert approach to develop a subscale for the school-age CBCL to screen for ASD, consisting of separate items. Both the specific data-driven and clinician-expert ASD subscale—along with the syndrome subscales that in prior research have been associated with ASD, the ASD subscales that have been developed in previous studies, and the widely used DSM-oriented subscales—were validated as well as cross-validated in two truly independent samples.
Overall, our results demonstrated that the currently developed ASD subscales had a better ability to identify children with ASD compared to the Withdrawn/Depressed, Thought Problems, and the Social Problems syndrome subscales, as well as combinations of these syndrome subscales [i.e., the ASD profile developed by Biederman et al. (
2010) and the WTP subscale developed by Havdahl et al. (
2016)]. This result is in line with that of Deckers et al. (
2020), who found that the specific ASD subscales developed by Ooi et al. (
2011) and So et al. (
2013) had a better capacity to differentiate between children with and without ASD compared to the (combinations of) syndrome subscales. This confirms the need for an ASD subscale based on individual school-age CBCL items, instead of relying on (combinations of) syndrome subscales. Although the currently developed ASD subscales had a higher internal consistency, they seemed to have a similar potential to discriminate between children with and without ASD as the specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013), when considering the ROC analyses. This is not surprising, given the considerable item overlap between those and the currently developed ASD subscales. However, some of the statistical AUC comparisons indicated that out of all specific ASD subscales for the school-age CBCL, the data-driven had the highest discriminative power. Moreover, our results showed that the currently developed ASD subscales performed equivalently to the DSM-oriented subscales, with comparable Cronbach’s Alpha and AUC scores. Lastly, the currently developed ASD subscales showed high sensitivity, but relatively low specificity (particularly for the subclinical range). However, high sensitivity may be considered more important—especially during middle school and adolescence, when ASD symptoms might be subtler and more heterogeneous (Bal et al.,
2019). Thus, our results suggest that the school-age CBCL seems as appropriate to screen for ASD as for other disorders (i.e., affective disorders, anxiety disorders, ADHD, ODD, and CD) and that when it comes to identifying children with ASD using this instrument, the specific data-driven subscale seems to be the best choice out of all examined ASD subscales for the school-age CBCL.
Thus, the currently developed ASD subscales for the school-age CBCL performed similarly to ASEBA’s DSM-oriented subscales. It should be noted, however, that compared to instruments that screen for ASD explicitly, their discriminative power was lower. For instance, when validating three ASD screeners—the Social Communication Questionnaire (SCQ; Berument et al.,
1999), the Social Responsiveness Scale (SRS; Constantino & Gruber
2005), and the Children’s Communication Checklist (CCC; Bishop,
1998)—in a sample of children with IQ scores higher than 70, Charman et al. (
2007) found levels of discriminative power that were somewhat higher compared to those we found for the currently developed ASD subscales in Sample 1, but clearly superior to those we found for the currently developed ASD subscales in the cross-validation samples (i.e., AUCs of at least 0.80 and sensitivity/specificity rates of at least 77%). A possible explanation for the relatively low sensitivity and/or specificity scores for the currently developed ASD subscales [as well as for the specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013)], first of all, is that these are part of a broad and general screening questionnaire, instead of developed as a stand-alone questionnaire to specifically screen for ASD (related problems), such as the SRS, SCQ, and CCC. Second, the selected school-age CBCL items for the specific ASD subscales do not cover every aspect of ASD (e.g., there are no items included on hyper- or hypo-reactivity to sensory input) or might be too vague to describe ASD symptoms (e.g., repeats certain acts over and over, compulsions). Interestingly, the DSM-oriented Autism Spectrum Problems subscale for the preschool-age version of the CBCL (CBCL 1.5-5; Appendix: Table H) includes some items that are more ASD-specific (e.g., rocks head, body) and its psychometric properties have found to be very good. For instance, when discriminating between preschoolers with ASD and typically developing preschoolers, Muratori et al. (2011) found an AUC of 0.95, a sensitivity of 85%, and a specificity of 90%. However, when comparing to preschoolers with other psychiatric disorders, results (i.e., AUC = 0.81; sensitivity = 85%; specificity = 60%) were more similar to those we found for the currently developed ASD subscales in Sample 1. Third, the presentation of ASD symptoms might change over time. That is, symptoms may differ for children during early childhood, middle childhood, and adolescence (Bal et al.,
2019). Perhaps, the items of the school-age CBCL do not reflect ASD symptom expression during middle childhood and adolescence as well as the items of the preschool-age CBCL do during early childhood. On the other hand, one could argue that because the Autism Spectrum Problems subscale (CBCL 1.5-5) is somewhat more ASD specific—with a few items being very typical for ASD or describing rather severe problem behavior—preschoolers with ASD that display subtler symptomatology and/or require less (parental) support might be missed. In the current study, we tried to account for changes in ASD symptom expression over time by—like ASEBA—considering different norms for children and adolescents. However, future (preferably longitudinal) studies are needed to explore the discriminative ability of the different specific ASD subscales for the CBCL across childhood (i.e., from the preschool to the adolescent years) and/or the lifespan—if one would be able to establish an ASD subscale for the Adult Self Report (ASR; Achenbach & Rescorla,
2003) as well. Lastly, another factor that might explain the relatively low specificity scores for the currently developed ASD subscales in particular, is that the majority of children in the first sample had an ADHD classification [as was the case in the study of Deckers et al. (
2020)]. To wit, there is high symptom overlap and comorbidity between ASD and ADHD (e.g., 59%; Stevens et al.,
2016). On the other hand, ASD might share symptom overlap and frequently co-exists with other disorders as well (i.e., the comorbidity rate between ASD and anxiety disorders is 40%; van Steensel et al.,
2011). Therefore, we recommend more direct comparisons between children with ASD and children with other classifications, in order to evaluate the discriminative ability of the currently developed ASD subscales. Particularly, future studies should include larger numbers of children with internalizing classifications in clinical control groups, as these were less well represented in our samples.
Important to note is that, even though the specific ASD subscales for the school-age CBCL seem to have less discriminative power compared to instruments developed to explicitly screen for ASD (e.g., the SRS, SCQ, and CCC), both types of instruments might serve different purposes. To wit, such ASD screeners are rarely implemented in school settings due to their administration being quite time-consuming and expensive (So et al.,
2013). Also, these disorder-specific instruments are hardly used to screen for ASD at intake in general mental health care centers (to which children are often [first] referred when they display social, emotional, and/or behavioral problems) due to intake sessions leaving little room for the administration of extra assessments focusing on merely one type of psychopathology (Deckers et al.,
2020). However, community-based pediatric services, which often use the school-age CBCL to routinely screen for a broad range of (mental) health problems in children at specified moments during their development, could profit from an ASD subscale because it can easily be incorporated in the analysis of results. Also, in clinical settings, an ASD subscale can effortlessly be added to the analysis of the screening results, if the administration of the school-age CBCL is part of the standard procedure. Thus, the ASD subscale for the school-age CBCL can be used as a first exploration into possible ASD symptomology and when children score above the corresponding cut-off, instruments to explicitly screen for ASD can be administered to zoom in further. This way, the use of the school-age CBCL ASD subscale might save time, and therefore expenses. In future research, it would be interesting to compare the performance of the currently developed
ASD subscales to that of explicit ASD screeners, to determine whether such disorder-specific screening instruments add significant value over and above a first screening with the school-age CBCL.
It was remarkable that in terms of item composition, although there was some overlap, the data-driven (i.e., based on parent reports) and clinician-expert (i.e., based on opinions of clinicians) approaches yielded different ASD subscales. Discrepancies between parent and clinician observations in the assessment of ASD symptoms, however, are not uncommon (e.g., de Bildt et al.,
2004; Lemler,
2012; Neuhaus et al.,
2018). An advantage of the data-driven approach might be that parents spend the most time with their children and are the first ones to detect certain problems, thus are the main informants with reference to their children’s (problem) behavior (Lemler,
2012). Yet, parent’s observations might be prone to over- or under-estimation, as they do not experience their children’s behavior in the school or clinical setting (Lemler,
2012), and may be biased due to perceived parenting stress (Schwartzman et al.,
2021). A strength of the clinician-expert approach might be that clinicians are more trained in observing the heterogeneous nature of the ASD symptomatology, and have seen or worked with multiple children with ASD. To illustrate, Lord et al. (
2006), who used both a parent- and a clinician-based diagnostic instrument to examine the stability of ASD diagnoses at ages two and nine, found that clinicians had a higher percentage of agreement in accurate diagnosis compared to parents. However, clinicians mainly observe children in the clinical setting, which might lead to their view on the functioning of children with ASD being somewhat one-sided (Lemler,
2012). Interestingly, Neuhaus et al. (
2018) found that disagreement between parents and clinicians was bigger when, amongst others, children had higher IQ-scores and displayed more adaptive behavior. Thus, considering the characteristics of the children included in Sample 1, following both a data-driven and a clinician-expert approach—hence constructing two different specific ASD subscales for the school-age CBCL—was the most thorough way of conducting the current study. It should be noted that our Clinicians Sample only included 15 participants, who were all employed at the same academic mental health care center. A larger-scale, international replication of the clinician-expert approach—like the one used by ASEBA when developing the DSM-oriented subscales—is needed to determine whether this or the data-driven approach is most valid for constructing a specific ASD subscale for the school-age CBCL.
A somewhat disappointing finding was that the discriminative power of not only the specific data-driven and clinician-expert ASD subscales, but also that of all other investigated syndrome subscales, previously developed ASD subscales, and ASEBA’s DSM-oriented subscales (to screen for depression, anxiety, ADHD, ODD, and CD), was lower in the cross-validation samples than in the first sample and samples used in previous validation research. For instance, Deckers et al. (
2020) found that the ability to identify children with ASD of the (combinations of) syndrome subscales ranged from poor to fair, and that this ability of the specific ASD subscales of Ooi et al. (
2011) and So et al. (
2013) ranged from fair to good. Also, Ebestuani et al. (
2010) found that the discriminative power of the different DSM-oriented subscales ranged from fair to good. The relatively low AUC, sensitivity, and specificity values in our cross-validation samples might be due to several methodological factors, such as different sample characteristics (e.g., level of intelligence, social economic status, and family situation) or different diagnostic and screening procedures applied within the participating mental health care centers. Although a range of methods (i.e., interviews, observations, questionnaires, and/or psychiatric/neuropsychological evaluations) was available to be used during the diagnostic process, there was no standardized protocol for establishing DSM classifications, thus which methods were applied could vary per mental health care center and/or per child. In addition, a standardized protocol to screen for comorbidity was lacking. This could have influenced both classification and comorbidity rates.
Strengths of the current study include the use of both a data-driven and a clinician-expert approach in constructing specific ASD subscales for the school-age CBCL, the large number of participants, the comparisons to ASEBA’s DSM-oriented subscales, and the use of two truly independent cross-validation samples. That is, we explicitly chose to use truly independent cross-validation samples over applying random subsampling (i.e., combining all data and then splitting it in half), as we wanted to explore how well a specific ASD subscale constructed within one treatment center would perform in others. In contrast, random subsampling would have led to the development/validation and cross-validation samples being rather similar, which—to our opinion—would not properly test generalizability to other treatment centers (with other sample characteristics). Although applying a resampling approach might have ensured more robustness of our specific ASD subscales in cross-validation efforts, we chose to retain the samples for development/validation and cross-validation as distinct groups, as the ultimate goal was to construct an appropriate specific ASD subscale that can be used in different clinical and pediatric settings. As such, we have chosen the sample of one treatment center as the development/validation sample (i.e., Sample 1, as this one included the most children with ASD and contained the most descriptive and clinical information), and used the other samples as truly independent cross-validation samples.
Some shortcomings—aside from the majority of children in Sample 1 having an ADHD classification, the relatively small size of the Clinicians Sample, and the varying screening and diagnostic procedures applied within the participating mental health care centers—need to be acknowledged as well. First, as Sample 1 included children that had been referred to the mental health care center between 2009 and 2020, some of them had a DSM-IV-TR instead of a DSM-5 ASD classification. However, Kulage et al. (
2020), who conducted a 5-year follow-up systematic review and meta-analysis in which they included 33 studies, found that 79% of children with a DSM-IV-TR classification still met the DSM-5 criteria for ASD. Also, in the DSM-5, the American Psychiatric Association states that individuals with a well-established DSM-IV-TR classification of autistic disorder, Asperger’s disorder, or pervasive developmental disorder not otherwise specified, should be given the DSM-5 classification of ASD (APA,
2013). Therefore, we decided not to exclude the children with a DSM-IV-TR ASD classification. Second, the perspectives of teachers were not considered in the development and validation of our specific ASD subscales. This could have been of great value, as teachers are provided with lots of opportunities to observe children in social situations, especially during interactions with their peers. It should be noted, however, that Deckers et al. (
2020) found that parents were better informants when identifying children with ASD compared to teachers. The authors argued that this may be explained by parents being able to observe their child in various settings and over time. Besides, through communication with the teacher, they might also be able to assess how their child is behaving at school.
A remaining question for future research is whether the currently and previously developed specific ASD subscales are culture-bound. To wit, previous research has found cultural differences regarding ASD symptom expression (e.g., Matson et al.,
2011) and although the school-age CBCL has been validated for use in many cultures (e.g., Rescorla et al.,
2007), the specific ASD subscales have mostly been developed and/or validated using samples predominantly consisting of Western participants.
In conclusion, the results of this study indicated that out of all developed ASD subscales for the school-age CBCL, the specific data-driven seems to have the best potential to screen for ASD during middle childhood and adolescence. Also, this subscale has a similar screening potential as the DSM-oriented subscales developed by ASEBA, which have been widely used for nearly two decades. It should be noted that all examined subscales (including ASEBA’s DSM-oriented subscales) showed rather poor discriminative power in the cross-validation samples. However, considering the possible benefits for both pediatric and clinical practice, we encourage our colleagues to continue the validation of this specific ASD subscale for the school-age CBCL.