Introduction
Anxiety disorders are among the most common mental health problems experienced by children and adolescents [
1,
2]. Accordingly, the past decade has brought forth a multitude of studies examining the etiology, phenomenology, and prognoses of these anxiety disorders [
3]. Despite this empirical attention, however, research on the validity of our current classification system for childhood anxiety disorders continues to yield conflicting findings [
4‐
11]. On the one hand, arguing for the accuracy of our current system, a large body of literature has shown that children with anxiety disorders can be reliably distinguished from those without anxiety disorders [
12‐
14] and that specific anxiety disorders can be disentangled from each other into discrete yet correlated entities [
10,
11]. On the other hand, arguing against the accuracy of our current system, existing childhood anxiety research and clinical work is plagued by diagnostic difficulties and inter-rater disagreement [
15,
16], drawing into question the system delineated in the
Diagnostic and Statistical Manual of Mental Disorders (
DSM-
IV) [
17] and the utility of retaining this system for future modifications of the manual. Approaches to understanding and addressing this conflict have been varied, yet a potentially pivotal question has remained unaddressed: Do the diagnostic difficulties and issues of inter-rater disagreement act in such a way that they refute the current classification system? Or are these diagnostic difficulties a complicating—yet distinct—issue in an otherwise accurate system?
Supporting evidence for the current classification system of childhood anxiety disorders may fall into two of the components of construct validity [
18‐
20]: (a) discriminant validity (i.e., the extent to which anxiety disorders in children can be distinguished from each other and from disorders unrelated to anxiety) and (b) convergent validity (i.e., the extent to which different means of assessing the same anxiety construct converge with one another). If the current diagnostic system, which distinguishes among major anxiety syndromes of childhood, accurately categorizes anxiety disorders in children, there should be evidence of good discriminant and convergent validity for measures of these syndromes. For example, measures of separation anxiety should aggregate together but be psychometrically distinct from measures of social anxiety. Of course, discriminant validity does not require the complete independence of separate constructs, so long as each disorder has a significant degree of unique variance. Furthermore, the measurement of each anxiety disorder should reflect the trait in question more than the general reporting pattern of each informant [
19,
20]. The present study uses a clinical sample of children and adolescents to examine the construct validity of the
DSM-
IV child anxiety classification system, with a focus on SoP, SAD, GAD, and PD.
Previous efforts to demonstrate the construct validity of specific anxiety disorders in children have found support in the area of discriminant validity. Though promising, these studies have often been limited by their focus on a restricted range of anxiety disorders (e.g., only SoP versus non-disordered children) or employment of a narrow range of methodology (e.g., only child-report). Some studies have demonstrated that children with any anxiety disorder (not a specific diagnosis) differed from their normative peers on various measures of anxiety [
12,
14]. Studies that have aimed to distinguish among specific anxiety disorders in childhood have produced mixed results. Large questionnaire studies have used factor analytic techniques to identify classes of child psychopathology [
21] yet the limited number of items assessing specific anxiety symptoms have often precluded detecting particular anxiety syndromes that may cohere together. Spence, using a more-detailed pool of anxiety self-report items, found support for PD, SAD, SoP, GAD, obsessive–compulsive disorder, and physical fears (analogous to specific phobia) as distinct entities that are related by a higher-order factor [
10]. Spence’s findings are encouraging, yet their generalizability to the construct validity of child anxiety disorders is limited by the use of only child-report data and a community sample. Ferdinand and colleagues [
11] reported that in referred and general population samples of children and adolescents, SAD and SoP are only distinct constructs in the sample of referred children. The present paper expands upon Ferdinand’s work by including multiple informants and employing a multitrait–multimethod matrix, described below, to assess the convergent and discriminant validity of these taxa.
Evidence for the convergent validity of these four major child anxiety disorders is scant. Agreement between diagnosticians, parents, and children has been found to be moderate in clinical samples, but varies by the specific anxiety disorder in question [
9]. Most studies show moderate convergence between children’s self-report measures and diagnoses on the clinical/non-clinical dichotomy but show poorer agreement at the level of specific anxiety disorders [
4]. Convergence between parent-report questionnaires and diagnostician-assigned diagnoses has also been fairly weak in some studies. For example, Boyle and colleagues [
22] found modest “case” agreement when mother-report questionnaire scores were dichotomized to clinical/non-clinical and compared to diagnoses derived from a structured interview (κ’s = .31 to .37). Moderate to poor concordance has also been shown in several studies comparing parent and child self-report measures of anxiety [
23,
24]. A recent study by Comer and Kendall reported stronger parent–child agreement at the symptom level than at the diagnosis level, particularly for observable symptoms, yet even this relatively stronger agreement was weak overall (κ < .35) [
8]. In short, convergent validity of the child anxiety disorder syndromes is not yet well established.
Campbell and Fiske’s multitrait–multimethod matrix (MTMM) design [
18] offers an approach that permits the simultaneous evaluation of discriminant and convergent validity that could be used to address unresolved questions about the construct validity of the current conceptualization of child anxiety disorders. The MTMM design has proven to be influential in the past half century of psychological research [
19]. However, the original analytic strategy proposed by Campbell and Fiske for conducting an MTMM analysis has been shown to have multiple limitations, including ambiguity about what constitutes satisfactory results and how to extract underlying trait and method factors from the correlation matrix [
25]. Of the several proposed corrections to the original simple correlational approach to MTMM analyses, covariance structure modeling has gained the most prominence [
19], with the general confirmatory factor analysis model (CFA; a type of structural equation modeling) being the method of choice [
25,
26]. The present study uses CFA to evaluate and compare a series of nested models, following the guidelines of Widaman [
27] and Byrne [
19,
28], to test an MTMM model of child anxiety. This analytic approach has clear criteria for hypothesis testing and allowed us to probe the construct validity of child anxiety disorders in a manner distinct from previous, less conclusive studies.
Results
A correlation matrix of all measures included in the study is displayed in Table
1, with descriptive statistics for each measure provided at the bottom of the table. The reporters’ ratings of the SAD, SoP, and PD are related to each other in a pattern that supports their construct validity; reporter ratings are significantly correlated within each disorder across informants, evidencing convergent validity by traditional MTMM criteria [
18]. However, ratings of SAD, SoP, and PD are typically not significantly correlated with each other outside of ratings within the child and parent informants, evidencing discriminant validity by traditional criteria with significant informant bias. The sole exception to this finding was a small but significant relationship between diagnosticians’ ratings of PD and parents’ ratings of SAD. The general reporting bias shown by the parents and children in the sample was not evident in the diagnostician ratings, where ADIS-CRS scores for specific disorders did not significantly correlate with each other, with the exception of SoP and GAD. Overall, the obtained pattern of correlations provided moderate evidence of convergent and discriminant validity for SAD, SoP, and PD, but less so for GAD.
Table 1
Correlation matrix of study measures
DGAD | .202** | | | | | | | | | | | |
DSAD | .066 | .066 | | | | | | | | | | |
DPD | .041 | .103 | .067 | | | | | | | | | |
ChSoc | .218** | .152* | .070 | .009 | | | | | | | | |
PaSoc | .331** | .132 | .012 | −.089 | .222** | | | | | | | |
ChGAD | −.084 | .020 | .330** | .098 | .180* | −.044 | | | | | | |
PaGAD | .076 | .147 | .239** | .250** | −.037 | .198** | .208** | | | | | |
ChSAD | −.004 | .052 | .420** | .042 | .224** | .017 | .570** | .347** | | | | |
PaSAD | .102 | .145 | .359** | .165* | −.069 | .251** | .279** | .664** | .564** | | | |
ChPD | .036 | .117 | .040 | .207** | .503** | .072 | .284** | .058 | .340** | .103 | | |
PaPD | .093 | .128 | .065 | .275** | .039 | .324** | −.03 | .430** | .083 | .404** | .277** | |
Mean | .18 | .29 | .14 | .05 | 12.49 | 14.98 | 16.26 | 15.77 | 9.56 | 10.51 | 13.52 | 12.29 |
SD | .38 | .45 | .35 | .21 | 6.08 | 5.82 | 4.47 | 4.39 | 4.94 | 5.32 | 6.61 | 6.53 |
Goodness-of-fit indices for the models are displayed in Table
2. The chi-square test of model fit measures the degree to which the data depart from the specified model. The larger the chi-square relative to the degrees of freedom (a measure of the number of unspecified parameters), the poorer the model fit. The comparative fit index (CFI) is a measure of fit that accounts for the complexity of the model in its calculation. The CFI ranges from 0 to 1, with a CFI above .90 indicating an acceptable fit. Lastly, the normed fit index (NFI) is an additional measure of fit that ranges from 0 to 1, with values above .90 indicating acceptable fit.
Table 2
Summary of goodness-of-fit indexes for MTMM models
1 Freely correlated traits; Freely correlated methods | 33.531 | 33 | .999 | .941 |
2 No traits; Freely correlated methods | 214.456 | 51 | .673 | .621 |
3 Perfectly correlated traits; Freely correlated methods | 82.623 | 39 | .913 | .854 |
4 Freely correlated traits; Perfectly correlated methods | 51.848 | 36 | .968 | .908 |
More importantly for the present purposes is the comparison of goodness-of-fit indices between models. Table
3 presents differences in chi-square values, degrees of freedom, the CFI, and the NFI. In terms of the test of the convergent validity of the proposed structure of anxiety syndromes as illustrated in Fig.
1, the statistically significant difference between the fit of models 1 and 2 is supportive of the convergent validity of this conceptualization of child anxiety. The existence of traits in model 1 significantly improves the fit. The statistically significant difference between the fit of models 1 and 3 provides support for the discriminant validity of the traits; allowing the four putative anxiety syndromes to correlate freely significantly improved model fit (the alternative was a correlation of 1.0 among all the traits), showing that the four anxiety traits are not perfectly correlated with one another, and thus, diverge meaningfully. Finally, discriminant validity of method (reporters) was also supported by the significant difference between the fit of models 1 and 4; when the model distinguished between methods, model fit improved significantly compared to the alternative (analogous to a lack of independence among methods), showing that each of the three methods employed—child report, parent report, and diagnostician rating—provide nonredundant information about the traits in question.
Table 3
Differential goodness-of-fit indexes for MTMM nested model comparisons
Test of convergent validity |
Model 1 vs. Model 2 (traits) | 180.925 | 18 | <.0001 | .326 | .320 |
Tests of discriminant validity |
Model 1 vs. Model 3 (traits) | 49.092 | 6 | <.0001 | .086 | .087 |
Model 1 vs. Model 4 (methods) | 18.317 | 3 | =.0004 | .031 | .033 |
Discussion
The aim of this study was to evaluate the construct validity of four child anxiety disorders using MTMM. In line with previous research, the present study’s observed correlation matrix displayed modest interrater agreement [
8,
34,
38] and significant intrarater correlations for parent and child informants [
7,
10,
34,
39]. Nonetheless, most of the traditional MTMM criteria for convergent and discriminant validity were met for SAD, SoP, and PD based on the simple correlation matrix. When confirmatory factor analyses (CFAs) were used to test hypotheses about construct validity using a model fitting approach applied to the same matrix, results were supportive of both convergent and discriminant validity, even with the addition of GAD in the model (which had not shown strong evidence of validity by MTMM criteria in the simple correlation matrix). The advantages of distinguishing among these four child anxiety syndromes outweighed interrater disagreement and other sources of error, such that the standard model of child anxiety disorders (as distinguishable but related entities) was a significantly better fit to the data than any alternative model (e.g., models excluding the specific disorders or constraining the specific disorders to be perfectly correlated with one another).
If specific anxiety disorders are phenomenologically distinct from one another, the more accurate model (i.e., the model that fits the data better) should be one in which specific anxiety syndromes can be distinguished from one another and do not provide entirely overlapping information, even if they are partially interrelated. In the present study, the specific anxiety disorder syndromes were related to each other, yet the model assuming perfect concordance between anxiety syndromes fit the data significantly worse than the model that allowed the magnitude of each syndrome to vary independently from the others. Thus, the present study supports the current conceptualization of these anxiety disorders as related, yet distinct entities. In line with previous research [
4,
22‐
24], these findings also suggest that each of the methods used to assess the child anxiety syndromes offers a related, yet unique perspective. This supports discriminant validity for the methods; each method (i.e., reporter) is significantly independent from the others.
If specific anxiety disorders are specified correctly in the current DSM-IV nosology, the model in which specific anxiety disorders exist and are adequately defined by converging reports of different informants will fit the data better than the model in which specific anxiety disorders are excluded and variation in the data is explained exclusively by individual differences among children and by method variance, not by meaningful patterns or subtypes of anxiety. In the present sample, the inclusion of specific anxiety disorders in the model significantly improved model fit and child-, parent-, and diagnostician-ratings converged on the specific anxiety disorders under evaluation. Although parents, children, and diagnosticians at times painted a different picture of the children’s symptom severity (illustrated by modest inter-rater convergence at the bivariate correlation level), they tended to be more in line with one another when rating a specific anxiety syndrome than when rating different syndromes. This pattern of findings offers support for the convergent validity of the four anxiety syndromes according to the CFA-based MTMM criteria employed in this study.
The present research extends previous findings by Spence [
10], Chorpita and colleagues [
40,
41], and Ferdinand and colleagues [
11] by using a multi-method approach with a clinical sample, affording greater symptom variability than could typically be found in community settings and allowing greater understanding of the aspects of construct validity of these anxiety syndromes among children who were clinically referred. Similar to the present study, Chorpita’s work, which found that the variability observed in specific anxiety disorders could be substantially explained by a set of higher-order factors (negative affectivity, physiological hyperarousal, etc.), highlights the way in which specific anxiety disorders are related yet distinct entities that share different combinations of the same underlying dimensions. Although the scope of this paper is limited to an examination of the construct validity of specific child anxiety disorders, the findings do not imply that higher-order factors would not partially account for the relationships between disorders. A logical next step in this line of research will be to integrate the present MTMM methodology, offering support for the current categorization of child anxiety disorders, with models investigating the dimensions that underlie these categories.
Limitations
Several methodological limitations of this study warrant discussion and attention in future research. First, the children and parents participating in this study presented to a clinic specializing in childhood anxiety treatments. Although not all children received a diagnosis, overall anxiety levels of children in the current sample are greater than that of the population. Such a sample of children grants the variability needed to explore interrelationships among the constructs of interest, yet it is possible that estimates of these interrelationships may change in a population with a significant proportion of children without anxiety disorders [
39]. This is an important future direction for study.
Second, the size of the current sample, though large enough to conduct confirmatory factor analyses, was not large enough to examine the relations between the child’s developmental status, gender, and the construct validity of the anxiety syndromes. Testing for model differences between groups requires multiple sample analysis, which in the case of gender, for example, would reduce the subsamples to
n = 94 and
n = 80, providing inadequate power for the number of free parameters in some of the models. Some research indicates that the structure of anxiety is stable in multiple age groups [
42], but most agree that a child’s developmental status is a pivotal component in the assessment of child psychopathology (e.g., the declining incidence of SAD as children transition to adolescence) [
40,
43,
44]. Gender has been shown to strongly relate to the prevalence rates anxiety syndromes with females demonstrating almost twice the risk of males in some studies [
45], warranting additional research on the construct validity of anxiety syndromes within each gender.
The present analytic procedure goes beyond standard approaches to studying childhood anxiety disorders by evaluating the construct validity of these disorders at a model level, assessing discriminant and convergent validity with state of the art structural equation procedures that provide clear-cut methods for hypothesis testing. In response to the question posed at the beginning of the present paper regarding whether modest interrater agreement and other diagnostic difficulties refute the current classification system of child anxiety syndromes, the present study suggests that the answer is no. The current nosology appears to reflect the data reasonably well using a comparative CFA approach.
Despite their differences, each informant provided a unique perspective that converged into four distinct anxiety disorders. This finding echoes that of Phillips and colleagues, who found that the perspectives offered by multiple informants contribute to the diagnostic picture beyond what can be captured by combining informants’ reports using the ‘or’ rule [
46]. The modest interrater agreement suggests that not only may diagnoses be most accurate when information from multiple informants is considered—a common truism in clinical assessment [
47]—but that the nature of the anxiety construct may change depending on which informant(s) the assessment is based. This theoretical approach, the
emergent variable model [
48], holds that the construct under study, in this case child anxiety, is a composite of different measures rather than a latent variable that exists independently of the methods used to estimate it. Similar to standard diagnostic practice for assessing attention deficit/hyperactivity disorder [
17], a complete assessment of anxiety may necessitate multiple informants. Future psychometric research efforts in this area may wish to consider whether empirically-based approaches to combining data from multiple informants can be derived to yield more accurate diagnostic decisions in research and clinical practice.
Summary
This study found moderate support for the convergent and discriminant validity of four main anxiety disorders: SoP, SAD, GAD, and PD. Results were particularly strong for SoP, SAD, and PD. Strengths of this study include its use of CFA, a model-based approach, which showed that the standard model of child anxiety disorders (as distinguishable but related entities) was a significantly better fit to the data than any alternative model. These results suggest that specific anxiety disorders should be distinguished from each other, and that a complete assessment of anxiety may necessitate multiple informants.