An initial psychometric evaluation of the CBCL 6–18 in a sample of youth with autism spectrum disorders

Abstract

Individuals with an autism spectrum disorder (ASD) often present with co-occurring emotional and behavioral disorders (EBD). The Child Behavior Checklist 6–18 (CBCL; Achenbach & Rescorla, 2001) is an EBD measure that contains several norm-referenced scales derived through factor analysis of data from the general pediatric population. The psychometric properties of this widely used and well-researched measure have not been evaluated in samples of youth with ASD. This study evaluated the CBCL's internal structure, scale reliability, criterion-related validity, and diagnostic accuracy using archival data from a well-characterized sample of youth with ASD (N = 122). Confirmatory factor analyses supported the unidimensionality of the CBCL's syndrome scales and its internalizing–externalizing factor structure. Significance tests indicated that many scales discriminated between two subgroups: a group of individuals with ASD + EBD and a group with ASD alone. Diagnostic accuracy analyses indicated that the CBCL had good sensitivity but low specificity for detecting co-occurring disorders. Results supported the use of the CBCL in conjunction with other clinical data when assessing for EBD in youth with ASD.

Introduction

Individuals with an autism spectrum disorder (ASD) reportedly present with relatively high rates of co-occurring emotional and behavioral disorders¹ (EBDs; see DeBruin et al., 2006, Lainhart, 1999, Leyfer et al., 2006). Because of their age and developmental disability, many youth with ASD cannot provide sufficient information for clinicians to reliably identify the presence or specific nature of a co-occurring EBD. Thus, third party rating scales are often an important component of clinical assessment. However, few EBD measures have been validated for youth with ASD.

One widely used parent rating scale that assesses for a broad range of emotional and behavioral syndromes is the Child Behavior Checklist 6–18 (CBCL; Achenbach & Rescorla, 2001). The CBCL's norm-referenced scales were developed through factor analysis of data from the general pediatric population. These scales appear to assess for the kinds of EBDs also observed in youth with ASD such as anxiety, depression, withdrawal, social problems, attention problems, and aggression. Although the CBCL's psychometric properties have been evaluated extensively (see Berubé & Achenbach, 2010 bibliography), they have not been comprehensively evaluated in ASD samples. In this study, we evaluated the CBCL's internal structure, scale reliability, criterion-related validity, and diagnostic accuracy using archival data from a well-characterized sample of six to 18-year-olds with ASD. Such evidence is needed to justify the clinical use and interpretation of CBCL scores in the assessment of EBDs in youth with ASD.

High rates of EBD have been reported in youth with ASD. Some of the most commonly occurring disorders include depression (Ghaziuddin, Ghaziuddin, & Greden, 2002), anxiety (Kim, Szatmari, Bryson, Steiner, & Wilson, 2000), Attention-Deficit/Hyperactivity Disorder (ADHD; Gillberg & Billstedt, 2000), and Oppositional Defiant Disorder (ODD; Gadow, DeVincent, & Drabick, 2008). EBDs have also been reported to persist over time (see Gadow et al., 2004, Mash and Dozois, 2003) suggesting the need for routine screenings in this at risk group.

However, identifying co-occurring EBDs that require specific intervention in youth with ASD is often difficult and challenges in assessment have been described in the literature (e.g., see Matson and Nebel-Schwalm, 2007, Ozonoff et al., 2005). First, many of these youth exhibit impairments in social-communication, cognition, self-awareness, and insight that may limit their ability to identify and report alterations in thoughts, feelings, behaviors, levels of personal distress, and/or functional impairment. Second, developmental characteristics may moderate the symptom topography associated with those EBDs that have been observed in the general population. Initial symptom presentation of some disorders may be nonspecific, such as in anxiety disorders (e.g., see White, Oswald, Ollendick, & Scahill, 2009). Atypical presentations and nonspecific symptoms elevate the risk for diagnostic overshadowing whereby behaviors may be attributed to the child's ASD and/or development instead of a co-occurring EBD. For example, for a child who demonstrates some variability in vocalizations but most often presents with low rates, a sustained period of an increased rate of vocalizations may be attributed to development or to the ASD when, in fact, the child might be demonstrating nonspecific anxiety symptoms. Failure to detect co-occurring EBDs can forestall specific intervention and increase the risk for further functional impairment. In fact, EBDs have been associated with poorer outcomes (Howlin, Goode, Hutton, & Rutter, 2004) and may moderate response to ASD-specific treatment. To address these issues, best practices in assessment call for a multi-method multi-informant approach, which often includes third-party report such as a parent rating scale. The CBCL represents one such measure that could be included in the assessment process.

Several rating scales are available to assess for EBDs in school-aged youth; however, relatively few have been evaluated in samples of youth with ASD. Some scales were developed for children and adolescents with developmental disabilities. For example, the Aberrant Behavior Checklist (ABC; Aman et al., 1985a, Aman et al., 1985b) and the Nisonger Child Behavior Rating Form (NCBRF; Aman, Tasse, Rojahn, & Hammer, 1996) assess individuals with an intellectual disability (ID). In addition, the Autism Spectrum Disorder-Comorbid for Children (ASD-CC; Matson, LoVullo, Rivet, & Boisjoli, 2009) was developed specifically for youth with ASD. The ABC, NCBRF, and ASD-CC scales do not closely correspond to DSM-IV-TR (APA, 2000) diagnostic categories; however, they do assess for specific problems that are often a focus of clinical attention in individuals with ID and ASD. A few studies of these measures in ASD samples have produced some favorable psychometric data with regard to internal structure, reliability, and correlations with other measures (cf. Brinkley et al., 2007, Lecavalier et al., 2004, Matson et al., 2009). Two measures developed for use with the general pediatric population have also recently been evaluated in ASD samples, the Child Symptom Inventory-4 (CSI-4; Gadow & Sprafkin, 2002) and the Behavior Assessment System for Children-Second Edition (BASC-2; Reynolds & Kamphaus, 2004). They assess for a wide range of EBDs, and the CSI-4 is based directly on the DSM-IV. Studies of the BASC-2 and CSI-4 in ASD samples have focused on different psychometric characteristics that included internal structure, score profiles, and the ability of the measures to discriminate between youth with and without an ASD (see Gadow et al., 2008b, Lecavalier et al., 2009, Mahan and Matson, 2011, Volker et al., 2010). Although these initial studies on measures developed for youth with developmental disabilities and for the general population have provided important initial reliability and validity data, more research is needed to fully evaluate the psychometric properties of each measure in ASD samples.

Similarly, the current and previous versions of the CBCL preschool and school age forms have not often been investigated in ASD samples (see Pandolfi & Magyar, in press). Previous studies focused on the CBCL's ability to discriminate youth with ASD from other clinical subgroups (see Biederman et al., 2010, Bolte et al., 1999, Duarte et al., 2003, Ooi et al., 2010, Rescorla, 1988, Sikora et al., 2008) and to help determine prevalence rates and multi-informant agreement for psychiatric syndromes in youth with ASD (see Hurtig et al., 2009, Kanne et al., 2009). Collectively, these studies provided initial evidence that the CBCL may have clinical utility for the assessment of youth with ASD. However, these studies were conducted without a complete understanding of the psychometric properties of the CBCL in ASD samples. Furthermore, none of these studies investigated whether the CBCL can discriminate between youth with ASD and a co-occurring EBD (ASD + EBD) from those with ASD alone.

We know of only one study that evaluated the factor structure of a current CBCL instrument in a sample with ASD. A series of confirmatory factor analyses (CFAs) conducted by Pandolfi, Magyar, and Dill (2009) supported the factorial validity of the CBCL 1.5–5 (Achenbach & Rescorla, 2000) in a sample of preschoolers with confirmed diagnoses of ASD. Results indicated that the CBCL 1.5–5 reliably measured two broad dimensions of EBD that reflected the scale's Internalizing and Externalizing Domains, and seven more narrowly defined dimensions that reflected the CBCL syndrome scales. These findings were consistent with the factor analytic work of the test authors, and supported the use of the instrument when assessing young children with ASD. A similar study is needed for the CBCL 6–18, an instrument that covers most of the child and adolescent age range.

We utilized several methods to evaluate the extent to which psychometric evidence supports the interpretation of CBCL 6–18 scores as indicators of EBDs in youth with an ASD. First, we sought evidence based on internal structure through a series of CFAs and scale reliability analyses. Specifically, each syndrome scale was tested for unidimensionality which is important because each scale is often interpreted separately in clinical practice. We followed these tests with a CFA of the CBCL's factor structure at the scale level. We then performed significance tests across all CBCL scales to determine the extent to which scores differed between a group with ASD alone, and one with ASD + EBD. Finally, we performed diagnostic accuracy tests using ROC analyses to determine how well the CBCL discriminated individuals with ASD and a specific EBD (e.g., a depressive disorder) from the rest of the sample (those without a depressive disorder). Collectively, these analyses expanded upon the work of previous studies of the CBCL in ASD samples by providing an initial comprehensive psychometric evaluation. In addition, because the CBCL measures different constructs than the other measures reviewed here, results of this study can complement ongoing research on the other EBD measures within the ASD population which can further assist practitioners in selecting the most appropriate assessment protocol for individual clients.