The Child Behavior Checklist (CBCL; Achenbach and Rescorla
2001) is among the most widely used parent-report measures of youth emotional and behavioral problems in both clinical and research settings. Since the development of the original empirically-based CBCL scales (Achenbach and Edelbrock
1983), Achenbach and colleagues have added narrow-band syndrome scales, competence and adaptive scales, as well as broad-band internalizing, externalizing, and total problems scales to better organize the information obtained from parents on their children’s competences and behavioral problems. With well-established normative data and standardized clinical cutoffs for its various syndrome and broad-band scales, the CBCL syndrome, internalizing, externalizing, and total problems scales have demonstrated strong psychometric properties within clinical settings for discriminating between referred and nonreferred populations (Achenbach
1991; Chen et al.
1994; Drotar et al.,
1995). For instance, the CBCL internalizing scale has demonstrated the ability to discriminate between youths with and without anxiety, as well as youths with anxiety disorders and youths with externalizing disorders (Seligman et al.
2004). Significant associations with both general (e.g., anxiety and affective disorders groups) and more specific (e.g., attention-deficit/hyperactivity disorder and conduct disorder) diagnostic groups have also been established (Eiraldi et al.
2000; Kasius et al.
1997), supporting the utility of the CBCL syndrome and broad-band scales across multiple settings.
CBCL DSM-Oriented Scales
Although the CBCL’s empirically-derived syndrome and broad-band scales have evidenced particular strengths, it has been argued that a weakness is their lack of concordance with nosology from the Diagnostic and Statistical Manual of Mental Disorders (
DSM; American Psychiatric Association
2000). For example, Jensen et al. (
1993) and Kasius et al. (
1997) found that the CBCL syndrome scales appear to be sensitive to measuring general mental illness/psychopathology, but not very sensitive to measuring specific
DSM diagnoses, particularly for internalizing disorders. This may not be surprising given the factor analytically-derived syndrome scales measure dimensions not entirely consistent with
DSM nosology. For instance, items that appear specific to
DSM categories of anxiety or depression are scored on the same Anxious/Depressed syndrome scale (e.g., “too fearful or anxious”, “nervous, highstrung, or tense “, “talks about killing self”, “feels worthless or inferior”).
Scales that map closely to
DSM nosology may be useful for several reasons. First,
DSM-oriented scales have the potential to better screen for and inform subsequently administered diagnostic interviews than other scales that were designed to measure multi-disorder syndromes or other constructs not aligned with
DSM categories. Relatedly, reimbursement for mental health services in clinical settings in America relies heavily on the documentation of
DSM-based disorders (Achenbach and Dumenci
2001; Doucette
2002). The psychopathology and treatment research literature are also largely based upon the
DSM classification system. Therefore, to provide an additional perspective with closer linkage to the prevailing
DSM nosology, Achenbach et al. (
2003) developed the CBCL
DSM-oriented scales to supplement the traditional CBCL syndrome scales. Unlike the syndrome scales, the
DSM-oriented scales were not derived via factor analytic methods, but were constructed through agreement in ratings among 22 highly experienced child psychiatrists and psychologists from 16 cultures. These experts rated each pre-existing CBCL item for the degree to which each was consistent with criteria for a particular
DSM-IV diagnostic category. Items were then matched with a given diagnostic category if rated as “very consistent with the
DSM category” by at least 14 of the 22 experts (Achenbach et al.
2003), and a
DSM-oriented scale was developed for categories matching 6 or more items. The six
DSM-oriented scales (and the diagnoses they are meant to represent) are: (a) Affective Problems [Dysthymic (DYS) and Major Depressive Disorders (MDD)], (b) Anxiety Problems [Generalized Anxiety Disorder (GAD), Separation Anxiety Disorder (SAD), and Specific Phobia (SPEC)], (c) Attention Deficit/Hyperactivity Problems [Primarily Hyperactive (ADHD-PH), Primarily Inattentive (ADHD-PI) and Combined (ADHD-C) subtypes], (d) Conduct Problems [Conduct Disorder (CD)], (e) Oppositional Defiant Problems [Oppositional Defiant Disorder (ODD)], and (f) Somatic Problems (Somatization and Somatoform Disorders).
Although research supporting the psychometric properties of these CBCL
DSM-oriented scales is growing, less is known about the clinical utility of these scales with respect to their correspondence with youth clinical diagnoses. Their comparative performance with related syndrome scales regarding correspondence with
DSM diagnoses is also unknown. Knowing the degree to which the rationally-derived
DSM-oriented scales correspond with
DSM diagnoses relative to the empirically-derived syndrome scales could provide insight into the incremental utility offered by the newly derived
DSM-oriented scales as well as whether the
DSM-oriented scales achieved their intended aims (i.e., greater correspondence with
DSM categories). Among research conducted to date on the CBCL
DSM-oriented scales, Achenbach et al. (
2003) reported that the CBCL
DSM-oriented scales, compared to the syndrome scales, evidenced a similar degree of internal consistency, test-retest reliability, and cross-informant agreement. They also reported that both scales revealed similar associations between classifications of youths’ scores as “deviant” (defined as
T-score ≥ 65). The factor structure of the CBCL
DSM-oriented scales was also found to be supported in a community sample (Achenbach et al.
2003). Nakamura et al. (
2009) also found support for scale reliability, as well as convergent and discriminative validity, for all six CBCL
DSM-oriented scales using a large and ethnically diverse clinic-referred sample of 673 children and adolescents diagnosed with structured interviews.
Ferdinand (
2008) also explored the clinical utility of the CBCL
DSM-oriented Anxiety Problems and Affective Problems scales for predicting pertinent
DSM-IV internalizing diagnoses (based on either parent/child impairment ratings or clinician rated clinical severity ratings) via receiver operating characteristic (ROC) curve methodology, in a clinic-referred sample of 277 youths, ages 6–18. Ferdinand found that the CBCL
DSM-oriented Anxiety Problems scale could predict SAD, GAD and SPEC when diagnoses were based on parent/child impairment ratings, but could not predict SAD and GAD when diagnoses were based on clinician severity ratings. Similarly, Ferdinand found that the CBCL
DSM-oriented Affective Problems scale could predict MDD and DYS when diagnoses were based on parent/child impairment ratings, but could not predict DYS when diagnoses were based on clinician severity ratings.
Although studies have begun to examine the psychometric properties of some of the CBCL DSM-oriented scales, numerous questions remain regarding the clinical utility of these scales on this widely used measure. Particularly unexplored are the DSM-oriented externalizing scales (i.e., Oppositional Problems and Conduct Problems scales) as well as the Attention Deficit/Hyperactivity (ADH) Problems scale. Regarding the DSM-oriented ADH Problems scale, although the developers indicated that this scale was designed to measure both ADHD-PI and ADHD-PH, it may be useful to test empirically whether this scale performs well at identifying ADHD-PI, ADHD-PH, and ADHD-C youths as well as discriminating between ADHD and disruptive behavior related disorders. Similarly, regarding the DSM-oriented Oppositional and Conduct Problems scales, further analysis of performance in clinical settings may provide insight into questions of how best use these scales to assist in diagnostic formulations of ODD, CD and their differential diagnosis. The CBCL’s broad use in diagnostic settings makes informing answers to these questions of particular importance.
For this reason, the present study used ROC curve methodology and discriminative ANOVAs to examine the concurrent validity and correspondence of the internalizing- and externalizing-related CBCL DSM-oriented scales with related DSM diagnoses using a clinic-referred sample of children and adolescents. In general, it was predicted that the DSM-oriented scales would evidence significantly greater correspondence with DSM diagnoses than the syndrome scales given their purported closer linkage with DSM categories. Specific hypotheses for each scale are indicated below, in the Data Analytic Approach section.
Method
Participants
Youths in the present sample were 476 of 557 consecutively referred children and adolescents to two mental health clinics (the Center for Cognitive Behavioral Therapy, in Honolulu, Hawaii; and Judge Baker’s Children Hospital, in Boston, Massachusetts) for diagnostic intake evaluations. Eligibility for youths in the present study included being 6–18 years old and having a CBCL
1 and diagnostic data available. Of the 557 consecutively referred youths, three children (0.5%) were younger than 6 years old, two youths (0.4%) had missing diagnostic data and 75 youths (13.5%) did not have an available CBCL (due to their caretakers choosing to not fill out the assessment measures). These 80 youths were thus not included in the study. To help ensure that all CBCLs contained valid responses, inclusion into the study also required each CBCL form to have 90% or more completed items (i.e., fewer than 13 items missing). One participant was excluded due to having 17 items missing, leaving a final sample size of 476 youths. The final sample consisted of 320 (67.2%) boys and 156 (32.8%) girls. Information about the total number of diagnoses present in the sample (primary and anywhere in the diagnostic profile) appears in Table
1. Youths’ ages ranged from 6.55 to 18.9 years (
M = 11.4,
SD = 2.5), and primary caregivers’ ages ranged from 21 to 78 (
M = 41.4;
SD = 9.6). Additional youth and primary caregiver demographic information appears in Table
2. All children and parents were fluent in English.
Table 1
Number of diagnoses anywhere and primary in youths’ diagnostic profile (N = 476)
Anxiety Disorders |
Generalized anxiety disorder | 26 | 75 |
Separation anxiety disorder | 26 | 74 |
Specific phobia | 8 | 75 |
Social phobia | 10 | 41 |
Obsessive-compulsive disorder | 7 | 18 |
PTSD | 8 | 22 |
Panic disorder | 0 | 1 |
Anxiety NOS | 1 | 1 |
Affective Disorders |
Major depressive disorder | 27 | 69 |
Dysthymic disorder | 11 | 33 |
Depressive disorder NOS | 4 | 4 |
ADHD Disorders |
ADHD-combined type | 32 | 110 |
ADHD-predominantly inattentive type | 38 | 83 |
ADHD-predominantly hyperactive-impulsive type | 4 | 6 |
ADHD-NOS | 21 | 48 |
Disruptive Behavior Disorders |
Oppositional defiant disorder | 118 | 174 |
Conduct disorder | 57 | 97 |
Disruptive behavior disorder NOS | 5 | 6 |
Schizophrenia | 4 | 10 |
Bipolar | 1 | 4 |
PDD | 0 | 0 |
Other | 19 | 39 |
No Diagnosis | 65 | 65 |
Table 2
Youth and caregiver demographic information
Youth Ethnicity |
Multiethnic | 182 | 38.2 |
White | 169 | 35.5 |
African American | 35 | 7.4 |
Asian American | 28 | 5.9 |
Latino/Hispanic | 25 | 5.3 |
Other | 27 | 5.7 |
Missing | 10 | 2.1 |
Caregiver Type |
Biological Mother | 268 | 56.3 |
Biological Father | 83 | 17.4 |
Adoptive Mother | 22 | 4.6 |
Adoptive Father | 6 | 1.3 |
Grandmother | 21 | 4.4 |
Grandfather | 10 | 2.1 |
Other | 40 | 8.3 |
Missing | 26 | 5.5 |
Caregiver Marital Status |
Married | 220 | 46.2 |
Divorced, separated | 121 | 25.5 |
Widowed | 18 | 3.8 |
Single | 84 | 17.7 |
Missing | 33 | 6.9 |
Caregiver Highest Level of Education |
No high school | 41 | 8.6 |
High school | 163 | 34.2 |
College | 192 | 40.3 |
Graduate School | 35 | 7.4 |
Missing | 45 | 9.5 |
Family Income |
$0–$39,000 | 243 | 51.1 |
$40,000 – $79,000 | 117 | 24.6 |
$80,000 – $119,000 | 52 | 10.9 |
$120,000 or more | 25 | 5.3 |
Missing | 39 | 8.2 |
Procedure
Prior to any data collection, all participants and their legal guardians underwent standardized Institutional Review Board-approved notice of privacy and consent procedures. The youths’ primary caregivers completed the CBCL in English and also participated in structured diagnostic interviews (i.e., the P-ChIPS). Assessors consisted of Ph.D. level clinical child psychologists and doctoral students in clinical psychology
2. A small portion of diagnoses were for problem areas not assessed by the P-ChIPS diagnostic interviews (e.g., trichotillomania). In order to make such diagnostic determinations, assessors acquired information from the parents regarding these problems areas according to
DSM diagnostic criteria (APA,
2000) and also discussed diagnostic formulation with available supervisors.
Data Analytic Approach
The correspondence of the
DSM-oriented scales and syndrome scales with related diagnoses was examined via ANOVA and receiver operating characteristic (ROC) analyses, using Analyze-It for Microsoft Excel version 2.12 (Analyze-It Software Ltd.,
2008). ROC analyses result in Area Under the Curve (AUC) values, which indicate the degree to which a scale predicts a binary classification (e.g., presence or absence of a clinical diagnosis). AUC values significantly greater than .50 indicate that the scale can perform the binary classification better than chance level. AUC values may also be interpreted according to the following: 50–.70, poor; .70–.80, fair; .80–.90, good; .90–1.00, excellent (c.f. Ferdinand
2008). The comparative performance of the
DSM-oriented and syndrome scales with respect to corresponding with relevant diagnoses was evaluated via z-test comparisons of AUC values (DeLong et al.
1988). Larger AUC values are indicative of better prediction of diagnostic status. Given the number of ANOVAs and AUC z-tests conducted, consideration of the Bonferroni correction (i.e., .05/37) warranted setting the significance level to .001 to help correct for Type-I error rates in the present study. The
DSM-oriented Somatic Problems scale was not examined because no youths in the present sample had a Somatization Disorder.
Using discriminative ANOVA and ROC methodology, we applied a general analytic approach to each
DSM-oriented scale. (a) First, if the scale was designed to target a
cluster of related diagnoses, we used ANOVA and AUC analyses to evaluate whether the
DSM-oriented scale could discriminate youths with each individual diagnosis targeted by the scale from all youths without diagnoses of the related disorders (e.g., evaluating whether the
DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD from youths without any anxiety disorder). We performed this test to demonstrate that the
DSM-oriented scales corresponded significantly with
each disorder targeted by the scale. This test was applicable to the
DSM-oriented Anxiety, Affective and ADH Problems scales, as these scales were designed to target clusters of related disorders. We predicted that these
DSM-oriented scales would correspond significantly (i.e., evidence significant ANOVA and AUC values significantly greater than chance) with all individual diagnostic groups. (b) Second, we used ANOVA and AUC analyses to evaluate whether each
DSM-oriented scale could discriminate between any youths with the scale’s targeted disorder group from all youths without any of those disorders (e.g., evaluating whether the
DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD, GAD and/or SPEC from youths without any of these disorders), and whether each
DSM-oriented scale could perform this discrimination significantly better than its related syndrome scale (i.e., significantly greater AUC). These tests were applicable to all
DSM-oriented scales. We predicted that all
DSM-oriented scales would be able to discriminate the targeted diagnostic groups significantly better than the syndrome scales given that the
DSM-oriented scales were developed to more closely correspond with current
DSM categories. (c) Third, we used ANOVA and AUC analyses to evaluate whether each
DSM-oriented scale could discriminate youths with the scale’s targeted disorder group from youths with a related disorder (but without any disorder related to the targeted disorders), so as to aid in differential diagnostic formulations between related disorders
3. For example, one test of this type evaluated whether the
DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD, GAD and/or SPEC from youths with any affective disorder (and without any anxiety disorder). This type of analysis was applicable to all
DSM-oriented scales. Again, we predicted that all
DSM-oriented scales would be able to significantly discriminate pertinent diagnostic groups and would perform significantly better than the related syndrome scales (i.e., significantly greater AUC values). The one exception was the
DSM-oriented Oppositional Problems scale. We predicted that this scale would not be able to discriminate between ODD and CD youths, given that all the features of ODD are usually present in CD. Notably, we predicted the null hypothesis in this particular case. However, instead of omitting this analysis due to the prediction of the null hypothesis, we thought it was meaningful to include this analysis in the present study. That is, if significant differences were found between ODD and CD youths on the
DSM-oriented Oppositional Problems scale, this would suggest that this scale does not specifically measure ODD symptomatology given that both ODD and CD youths should have elevated scores on this scale.
Results
CBCL DSM-Oriented Anxiety Problems Scale
CBCL DSM-Oriented Affective Problems Scale
CBCL DSM-Oriented Attention Deficit/Hyperactivity Problems Scale
CBCL DSM-Oriented Oppositional and Conduct Problems Scales
Discussion
The present study evaluated and compared the degree to which the rationally-derived CBCL DSM-oriented scales and empirically-derived syndrome scales corresponded with parent-based youth DSM diagnoses. Given that expert child psychiatrists and psychologists assisted in deriving the DSM-oriented scales to be more consistent with DSM nosology, it was predicted that these scales would evidence significantly better correspondence with clinical DSM diagnoses than the syndrome scales. The present findings, however, revealed that the DSM-oriented scales generally did not evidence performance superior to that of the syndrome scales with respect to correspondence with clinical diagnoses. One explanation for this lack of increased correspondence with DSM diagnoses by the DSM-oriented scales is that these scales were derived from the limited pool of 120 items already comprising the CBCL. This likely placed a limit on the ability of the developers to create scales that align more with DSM constructs. In fact, among the six CBCL DSM-oriented scales, the Oppositional and Conduct Problems scales are the only two scales that target specific DSM disorders (i.e., ODD and CD, respectively). The remaining four DSM-oriented scales were developed to target clusters of related disorders (e.g., the DSM-oriented Anxiety Problems scale targets the cluster of GAD, SAD and SPEC anxiety disorders, as opposed to any single anxiety disorder), likely due to a limited item pool.
Interestingly, the
DSM-oriented Anxiety Problems scale was the only
DSM-oriented scale that evidenced a slight advantage over the syndrome scales. Despite being comprised of only 6-items, a concern raised by previous investigators (e.g., Ferdinand
2008; Kendall et al.
2007), the
DSM-oriented Anxiety Problems scale evidenced significantly greater AUC values than the Anxious/Depressed syndrome scale with respect to discriminating (a) the anxiety disorders targeted by the scale (i.e., SAD, GAD, and SPEC) from all other disorders, as well as (b) the anxiety disorders targeted by the scale from affective disorders. These findings are consistent with the notion that the Anxious/Depressed syndrome scale is somewhat less specific to anxiety than the
DSM-oriented Anxiety Problems scale, given that the Anxious/Depressed syndrome scale contains items also related to affective problems (e.g., “feels too guilty,” “talks about killing self,” “feels worthless and inferior”). Notably, however, the Anxious/Depressed syndrome scale also evidenced significant ANOVAs and its AUC values fell in the “fair” and “good” range, supporting this scale’s ability to also correspond significantly with anxiety problems.
Regarding assessment of disruptive behaviors, the present study demonstrated that the DSM-oriented Oppositional Problems and Aggressive Behavior syndrome scales performed similarly with respect to correspondence to ODD diagnostic status. The present study also demonstrated that the DSM-oriented Conduct Problems and the Rule-Breaking Behavior syndrome scales performed similarly with respect to corresponding with CD diagnostic status. Notably, the DSM-oriented Oppositional Problems could not discriminate between ODD and CD youths, whereas the DSM-oriented Conduct Problems scale could discriminate between these youths. This finding is consistent with the mutually exclusive diagnostic rule of ODD and CD (i.e., that a youth cannot receive both ODD and CD), as well as the notion that all the features of ODD are usually present in CD. This finding has implications pertaining to the application and interpretation of the DSM-oriented Oppositional Problems and Conduct Problems scales. Before interpreting elevations on the DSM-oriented Oppositional Problems scale as suggestive of an ODD diagnosis (or the need for further testing to determine ODD status), it is necessary to also inspect the DSM-oriented Conduct Problems scale to verify that elevations are not also present on this scale. If the DSM-oriented Conduct Problems scale is not elevated, a diagnosis of ODD may be considered; however, if the DSM-oriented Conduct Problems scale is elevated, a diagnosis of CD should instead be considered.
The gender-specific analyses revealed that the present study’s findings are generally robust to youth gender. The one potential gender-related difference is that the DSM-oriented Affective Problems scale and the Withdrawn/Depressed syndrome scale performed equally well with respect to discriminating boys with depression from boys with anxiety, whilst the DSM-oriented Affective Problems scale performed somewhat better than the Withdrawn/Depressed syndrome scale with respect to discriminating girls with depression from girls with anxiety.
Despite the DSM-oriented scales not evidencing significant advantages over the syndrome scales with respect to correspondence with clinical DSM diagnoses (except for the few instances noted above), it is notable that the 6 DSM-oriented scales are comprised of significantly fewer items than the 8 syndrome scales (i.e., 55 items versus 103 items, respectively). As public mental health systems and provider agencies move towards mandating the use of standardized measurement of youth outcomes, a shortened version of the CBCL — comprised of only the 55-itemed DSM-oriented scales — could be useful in clinical (and research) contexts, particularly when assessment battery length is a concern. More research however is needed to ensure that the DSM-oriented scales perform at least as well as the syndrome scales with respect to measurement accuracy across additional parameters and sample characteristics, such as factor structure, reliability, and age.
There were also limitations of the present study that offer directions for future research. First, there were no youths in the present sample with a somatization disorder. As a result, the
DSM-oriented Somatic Problems scale could not be evaluated. Future research evaluating the concurrent validity of the
DSM-oriented Somatic Problems scale is thus needed. There were also very few youths in the sample with a diagnosis of ADHD-PH, limiting the ability to evaluate the concurrent validity of the
DSM-oriented ADH Problems scale with respect to this diagnosis. Further, only the CBCL, as opposed to also the Teacher Report Form (TRF) and/or Youth Self-Report (YSR), was examined in the present study. Findings related to both the TRF and YSR
DSM-oriented scales would provide further understanding of the performance of these
DSM-oriented scales in clinical settings, such as their correspondence with
DSM diagnoses. Other questions pertaining to the applicability of the
DSM-oriented scales could also be examined. For instance, knowing the degree to which the
DSM-oriented Anxiety Problems scale corresponds with other anxiety disorders beyond the anxiety disorders targeted by the scale (i.e., SAD, GAD, and SPEC) would help inform its usage. Further, knowing the degree to which other parent-report measures (e.g., the Revised Child Anxiety and Depression Scale — Parent Version; Ebesutani et al.
2009) designed to map onto specific
DSM constructs correspond with clinical diagnoses compared to the CBCL
DSM-oriented scales may also inform whether the CBCL
DSM-oriented scales achieved their aims.
Overall, although the present findings support the concurrent validity of the recently derived CBCL
DSM-oriented scales, the present study did not identify significant advantages of the
DSM-oriented scales over the syndrome scales with respect to correspondence with
DSM diagnoses. Given that in creating the
DSM-oriented scales, the developers likely sought to increase correspondence with
DSM diagnoses relative to that of the syndrome scales, additional attention may be needed to explore why increased correspondence was not achieved and whether increased correspondence may be obtained via modifications to the
DSM-oriented scales. Kendall et al. (
2007), for example, recently attempted to improve the 6-item
DSM-oriented Anxiety Problems scale by adding an additional 10 anxiety-related CBCL items to the scale. Although this did not lead to clear improvement of the scale (i.e., their new 16-item anxiety scale predicted anxiety disorders better than the original 6-item CBCL
DSM-oriented Anxiety Problems scale when based on mother reports, but predicted anxiety disorders worse when based on father reports), these findings are promising with respect to the potential for enhancing the clinical utility and concurrent validity of the CBCL
DSM-oriented scales via scale/item modifications. Additional similar efforts would be valuable. In the meantime, findings of the present study demonstrated that the CBCL
DSM-oriented scales evidenced significant correspondence with related
DSM diagnoses and are thus clinically useful alongside the syndrome and broad-band scales to incorporate into the assessment of youth emotional and behavior problems. It is encouraged that future efforts be made to increase correspondence of the
DSM-oriented scales with
DSM diagnoses for enhanced utility in clinical and research settings.
Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (
https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.