Introduction
The Autism Diagnostic Observation Schedule (ADOS; Lord et al. 2000) is a semi-structured, standardized observational instrument to assess the social and communicative abilities of individuals with possible autism spectrum disorder (ASD). The ADOS consists of four modules intended for use in children, adolescents, and adults with different developmental and language levels. Items are scored from 0 (not abnormal) to 2 or 3 (most abnormal), and a diagnosis of autism or ASD is established if the individual assessed has scores higher than the established cut-off values in the Communication domain, the Social domain, and a sum of the two. The current ADOS diagnostic algorithm does not include items related to repetitive behaviors and restricted interests, although these behaviors are coded if they occur (Lord et al. 2001). Initially, this choice was based on limited available time to notice these kinds of behaviors in the context of the assessment. In research and clinical practice, the ADOS used in combination with the Autism Diagnostic Interview-Revised (ADI-R; Lord et al. 1994) is considered the “gold standard” for diagnosing autism. The ADI-R is a comprehensive, standardized, and semi-structured parent interview that includes items covering all three major domains of dysfunction in autism, namely, quality of reciprocal social interaction, communication, and repetitive, restricted, and stereotyped patterns of behavior.
In 2007, Gotham and colleagues proposed revised algorithms for Modules 1 through 3 to improve the sensitivity and specificity of the ADOS. In the revised algorithms, the original ADOS domains and cut-off values for Social and Communication items have been collapsed into a single factor consisting of 10 items that describes social and communication domain items: the Social Affect factor (SA). In addition, 4 items from a second factor, restricted, totals repetitive behavior (RRB), have been included because RRB domain items may contribute to the diagnosis of autism or ASD, even in the limited context of the ADOS (Lord et al. 2006). There are two diagnostic cut-off scores for the combined SA&RRB domain total, one for autism and one for ASD. In order to reduce ceiling effects in communication items, the revised algorithms distinguish between “Some words” and “No words” in Module 1, on the basis of the item A1 score (overall level of non-echoed language). To reduce the difference between younger more rapidly developing children and older children, the revised algorithms distinguish between age younger and older than 5 years in Module 2.
The new algorithms have important advantages over the original algorithms: they are more homogeneous, which makes comparisons between and within cases across all three modules easier; the effects of age and IQ are probably reduced; and they make direct comparison with ADI-R scores possible. In addition, on the basis of these revised ADOS algorithms, Gotham et al. (2009) recently proposed a calibrated autism severity metric that could prove very useful for identifying trajectories of autism severity in clinical, genetic, and neurobiological research.
To the best of our knowledge, four studies have replicated the revised ADOS algorithms. In the first, small study (N = 26), Overton et al. (2008) reported inconsistent results, finding slightly more accurate results for Module 1, although revised algorithm scores for Modules 2 and 3 were similar to those for the original algorithms. In the second study, Gotham et al. (2008) replicated their own study in a large and independent sample (N = 1282) and presented sensitivity and specificity data that were similar or better than those for the original algorithms, with the exception of scores for young children with phrase speech and a clinical diagnosis of pervasive developmental disorder not otherwise specified (PDD-NOS). In the third replication study (N = 195), Gray et al. (2008) found improved sensitivity with slightly reduced specificity for Module 1/No Words, and improved sensitivity and specificity for Module 1/Some Words. Additional indices of diagnostic accuracy were taken into consideration; however, it was difficult to compare findings with those of the other studies due to dissimilarities in definition of diagnostic groups. In the fourth study, De Bildt et al. (2009) evaluated the revised algorithms in a Dutch, low-functioning sample (N = 558). The balance between sensitivity and specificity for Module 2 and 3 was better with the revised algorithms, without there being changes in efficiency of classification. The sensitivity and specificity of Module 1 showed a more modest improvement, possibly because of the low-functioning sample.
Although these replication studies generally support the use of the revised algorithms, additional research is warranted because the revised algorithms could not be applied to some developmental cells due to limited data (De Bildt et al. 2009; Gotham et al. 2008; Gray et al. 2008). The sample studied by Overton et al. (2008) was too small in general to draw firm conclusions from the results. The sample in the current study did allow for filling up some gaps in the replication literature so far, especially for Module 1/Some Words and both Module 2 cells. In addition, as age and diagnostic groups represented in samples will influence outcome measures at all times, it is of interest to replicate Gotham et al. (2007) findings in an independent sample with a slightly different make-up.
In the current study we sought to augment the findings of earlier studies in a large independent Dutch sample (N = 532). The main aim was to examine whether the revised algorithms improved the diagnostic validity of the ADOS. We administered Modules 1 and 2 but not Module 3, because of the limited data available for this module. Additional aims were to study the effects of age and IQ on the revised algorithms, evaluated against the original algorithms, and to verify the factor structure of the revised algorithms.
Methods
Participants
Data for 426 participants (aged 15 months to 12 years; 78% male; 95% Dutch Caucasian background) were re-evaluated using the revised algorithms. We included 106 participants with repeated assessments (separated by a mean of 30.6 ± 7.0 months) because Gotham et al. (2007) reported that repeated assessments did not influence findings. Taking the repeated assessments into account, the total sample included 532 cases. Data were provided by three Dutch University Centers for Child and Adolescent Psychiatry, namely, Nijmegen (n = 490), Utrecht (n = 23), and Groningen (n = 19). At least 90% of all participants had been referred for suspected ASD and/or other developmental disorders, most within the context of an extensive early screening project for ASD (see Oosterling et al. 2009). A minority was recruited for research projects (De Bildt et al. 2005; IMGSAC 2001).
On the basis of their use of expressive language, the participants had been assessed with either Module 1 (60%) or Module 2 (40%). They were divided into four developmental cells as suggested in the revised algorithms proposed by Gotham et al. (2007) and as summarized in Table 1. In order to enhance readability of this paper, the four developmental cells will further be referred to as: cell A, B, C or D (see Table 1).
Table 1
Revised algorithm developmental cells
Age | Module 1 | Module 2 | |
|---|---|---|---|
No speech | Five or more words - simple phrases | Flexible three words phrases - verbally fluent | |
<5 | Module 1 No words | Module 1 Some words | Module 2 Younger then 5 years (C) n = 108 |
(A) | (B) | ||
5–12 | n = 137 | n = 184 | Module 2 Five years and older (D) n = 103 |
The clinical diagnoses of the 532 cases were autism (214 cases; 40% of the entire sample), PDD-NOS (134 cases; 25%), Asperger (10 cases; 2%), non-ASD developmental disorders (156 cases; 30%), and no psychiatric disorder (18 cases; 3%). Among the cases with non-ASD developmental disorders, 15 (10%) had non-specific mental retardation, 36 (23%) had language disorders, 52 (33%) had externalizing disorders (Attention Deficit Hyperactivity Disorder or Oppositional Defiant Disorder), 17 (11%) had internalizing disorders (mood or anxiety disorder), and 36 (23%) had other developmental problems. Table 2 provides information on verbal/nonverbal IQ or ratio IQ based on the Psycho-Educational Profile-Revised (PEP-R; Schopler et al. 1990), and on ADOS and ADI-R domain totals.
Table 2
Description of sample characteristics
Module 1 | Module 2 | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
No words (A) | Some words (B) | Younger than 5 years (C) | Five years and older (D) | |||||||||||||
n
| Mean | SD | Range |
n
| Mean | SD | Range |
n
| Mean | SD | Range |
n
| Mean | SD | Range | |
Autism
| ||||||||||||||||
Age | 104 | 35.6 | 12.9 | 18–73 | 60 | 43.2 | 14.4 | 22–78 | 34 | 52.3 | 7.6 | 30–60 | 16 | 79.6 | 26.0 | 60–151 |
PEP-R Ratio IQ | 76 | 45.2 | 14.8 | 11–85 | 15 | 59.6 | 8.1 | 47–70 | – | – | – | – | – | – | – | – |
VIQ | 15 | 43.1 | 20.2 | 13–83 | 33 | 69.7 | 14.9 | 30–102 | 15 | 85.0 | 17.4 | 44–110 | 13 | 89.5 | 13.0 | 69–106 |
NVIQ | 19 | 65.6 | 21.7 | 35–107 | 33 | 77.0 | 17.4 | 50–113 | 26 | 89.9 | 22.0 | 56–131 | 16 | 87.3 | 20.0 | 50–123 |
ADI Social | 88 | 16.7 | 4.7 | 4–26 | 47 | 14.2 | 4.4 | 5–22 | 25 | 15.9 | 5.2 | 5–25 | 9 | 13.9 | 6.7 | 5–21 |
ADI Comm-V | 1 | 12.0 | – | – | 24 | 12.1 | 3.8 | 4–19 | 25 | 12.2 | 5.3 | 3–23 | 9 | 10.3 | 3.7 | 6–16 |
ADI Comm-NV | 87 | 10.9 | 2.7 | 3–14 | 23 | 8.3 | 2.8 | 4–13 | – | – | – | – | – | – | – | – |
ADI RR | 88 | 3.8 | 1.9 | 0–9 | 47 | 3.7 | 2.3 | 0–8 | 25 | 5.1 | 3.0 | 0–10 | 9 | 3.9 | 3.2 | 0–9 |
ADOS Social | 104 | 11.8 | 2.6 | 2–14 | 60 | 7.5 | 3.2 | 0–14 | 34 | 8.0 | 3.6 | 2–16 | 16 | 5.9 | 3.4 | 0–12 |
ADOS Comm | 104 | 5.9 | 1.4 | 2–8 | 60 | 4.0 | 2.1 | 0–9 | 34 | 4.4 | 2.4 | 0–10 | 16 | 3.6 | 1.9 | 1–8 |
ADOS SA | 104 | 17.0 | 3.2 | 4–20 | 60 | 11.4 | 4.7 | 0–20 | 34 | 10.5 | 4.8 | 2–18 | 16 | 8.0 | 4.3 | 0–15 |
ADOS RRB | 104 | 3.4 | 1.8 | 0–7 | 60 | 2.6 | 1.9 | 0–7 | 34 | 1.8 | 1.5 | 0–5 | 16 | 2.7 | 2.0 | 0–7 |
Non-Autism ASD
| ||||||||||||||||
Age | 20 | 30.1 | 7.8 | 20–54 | 40 | 36.0 | 6.6 | 26–69 | 41 | 50.5 | 7.2 | 25–59 | 45 | 73.2 | 13.3 | 60–113 |
PEP-R Ratio IQ | 10 | 56.4 | 12.1 | 36–77 | – | – | – | – | – | – | – | – | – | – | – | – |
VIQ | 8 | 73.4 | 19.6 | 50–104 | 36 | 85.6 | 22.1 | 54–178 | 19 | 98.3 | 15.4 | 70–126 | 35 | 96.6 | 21.6 | 50–133 |
NVIQ | 9 | 88.6 | 10.3 | 73–103 | 37 | 93.1 | 17.7 | 50–133 | 29 | 104.2 | 17.3 | 70–148 | 38 | 91.6 | 22.9 | 50–132 |
ADI Social | 15 | 9.6 | 4.6 | 3–18 | 36 | 8.9 | 4.5 | 0–18 | 22 | 13.6 | 6.7 | 2–29 | 21 | 9.9 | 5.0 | 3–22 |
ADI Comm-V | 2 | 5.5 | 7.8 | 0–11 | 27 | 8.7 | 3.9 | 2–16 | 22 | 9.6 | 5.2 | 0–18 | 21 | 6.9 | 3.2 | 2–15 |
ADI Comm-NV | 13 | 7.2 | 3.2 | 3–14 | 9 | 6.4 | 2.6 | 2–10 | – | – | – | – | – | – | – | – |
ADI RR | 15 | 2.5 | 2.1 | 0–7 | 36 | 2.7 | 1.9 | 0–8 | 22 | 4.2 | 3.4 | 0–11 | 21 | 1.9 | 2.1 | 0–8 |
ADOS Social | 20 | 7.0 | 2.8 | 2–13 | 40 | 4.6 | 2.9 | 0–11 | 41 | 4.1 | 2.8 | 0–11 | 45 | 5.7 | 2.5 | 1–11 |
ADOS Comm | 20 | 4.2 | 1.3 | 2–6 | 40 | 2.5 | 1.6 | 0–7 | 41 | 2.8 | 1.9 | 0–9 | 45 | 2.6 | 1.6 | 0–7 |
ADOS SA | 20 | 11.0 | 3.7 | 3–18 | 40 | 7.3 | 4.2 | 1–17 | 41 | 5.8 | 3.7 | 1–15 | 45 | 7.2 | 3.2 | 1–14 |
ADOS RRB | 20 | 2.0 | 1.5 | 0–5 | 40 | 1.1 | 1.1 | 0–4 | 41 | 1.0 | 1.3 | 0–5 | 45 | 1.4 | 1.5 | 0–6 |
Non-ASD
| ||||||||||||||||
Age | 13 | 26.8 | 7.6 | 15–38 | 84 | 34.5 | 6.7 | 22–66 | 33 | 50.7 | 6.9 | 35–39 | 42 | 71.1 | 8.2 | 60–98 |
PEP-R Ratio IQ | 4 | 63.5 | 11.2 | 50–77 | 1 | 69.0 | – | – | – | – | – | – | – | – | – | – |
VIQ | 8 | 66.7 | 25.5 | 18–91 | 80 | 87.2 | 18.3 | 38–134 | 21 | 96.6 | 11.5 | 75–121 | 34 | 99.5 | 21.6 | 53–145 |
NVIQ | 8 | 89.4 | 14.9 | 56–107 | 81 | 93.1 | 17.3 | 49–140 | 27 | 98.6 | 15.3 | 66–121 | 38 | 95.6 | 18.3 | 60–133 |
ADI Social | 10 | 7.2 | 4.3 | 1–13 | 67 | 6.8 | 4.5 | 0–18 | 11 | 6.7 | 4.4 | 1–17 | 19 | 5.7 | 4.6 | 0–15 |
ADI Comm-V | 2 | 2.5 | 2.1 | 1–4 | 49 | 6.2 | 3.3 | 1–13 | 10 | 4.8 | 3.6 | 1–13 | 19 | 3.8 | 3.0 | 0–13 |
ADI Comm-NV | 8 | 8.3 | 4.0 | 3–14 | 18 | 4.7 | 3.1 | 0–10 | 1 | 2.0 | – | – | – | – | – | – |
ADI RR | 10 | 1.8 | 1.7 | 0–5 | 67 | 2.3 | 1.8 | 0–8 | 11 | 1.8 | 1.5 | 0–5 | 19 | 1.0 | 1.2 | 0–4 |
ADOS Social | 13 | 4.8 | 4.1 | 0–13 | 84 | 2.3 | 1.8 | 0–8 | 33 | 2.3 | 2.1 | 0–8 | 42 | 2.1 | 1.9 | 0–6 |
ADOS Comm | 13 | 2.9 | 1.7 | 0–6 | 84 | 1.4 | 1.2 | 0–5 | 33 | 1.5 | 1.4 | 0–5 | 42 | 1.0 | 1.3 | 0–6 |
ADOS SA | 13 | 7.4 | 5.5 | 1–18 | 84 | 3.8 | 2.6 | 0–12 | 33 | 3.3 | 2.7 | 0–11 | 42 | 3.2 | 2.4 | 0–9 |
ADOS RRB | 13 | 1.0 | 1.4 | 0–4 | 84 | 0.6 | 1.0 | 0–4 | 33 | 0.5 | 1.0 | 0–3 | 42 | 0.9 | 1.3 | 0–6 |
Compared with the samples of Gotham et al. (2007, 2008) and De Bildt et al. (2009), our sample was younger and higher functioning, and ADI-R and ADOS mean scores were lower or comparable. In addition, compared to the other samples, our sample included relatively more non-spectrum and non-autism ASD cases and relatively fewer autism cases. Details about differences in age, (non)verbal IQ, ADI-R, and ADOS mean scores by developmental cell and by diagnostic category are presented in Tables 5 and 6 (Appendices). Our samples appeared to be similar, in terms of age, developmental level, and ADOS and ADI-R mean scores, to the sample administered Module 1 in the study of Gray et al. (2008).
Measures and Procedure
The diagnostic protocol at the three participating centers included: (a) a psychiatric evaluation, (b) assessment with the ADOS (Lord et al. 2001), (c) a standardized parent interview with the ADI-R (Rutter et al. 2003; available for 70% of the sample), and (d) evaluation of IQ and language skills. In addition, parent–child play was observed for most children aged 0–6 years. A multidisciplinary team comprising minimally a child psychiatrist and a psychologist established the diagnosis, based on DSM-IV-TR (APA 2000) criteria; the child psychiatrist had final responsibility for the diagnosis. Psychologists who met standard requirements for research reliability administered the ADOS and ADI-R. Cognitive abilities were assessed with a variety of instruments, mainly the PEP-R (Schopler et al. 1990; used for 22% of the sample), the Mullen Scales of Early Learning (MSEL; Mullen 1995; used for 31% of the sample), the Snijders-Oomen nonverbal (Dutch) intelligence test (SON-R; Tellegen et al. 1998; used for 29% of the sample), and the Wechsler tests (Wechsler 1997, 2002; used for 9% of the sample). Language abilities were assessed using the Dutch modification of the Reynell Developmental Language Scales (Van Eldik et al. 1995; used for 44% of the sample) and the Schlichting Test for language production (Schlichting et al. 1995; used for 45% of the sample). An indication of verbal IQ was based on either the verbal scale of the MSEL (expressive and receptive language) or the verbal scale of the Wechsler tests or, if these measures were not available, on the mean score on the receptive and expressive scales of the Reynell and Schlichting tests. An indication of nonverbal IQ was based on either the nonverbal scale of the MSEL (visual reception and fine motor scales), or on the SON-R.
Design and Analyses
For each case, the original algorithms were used to compute Communication and Social domain totals (Lord et al. 2001), and the revised algorithms were used to compute Social Affect (SA) and restricted, repetitive behavior (RRB) domain totals. For both the original and revised algorithms, item scores of 3 were recoded to 2 and item scores of 7 and 8 to 0 as they appear on the algorithm. Diagnostic classifications were then generated using the original or revised threshold cut-offs, both with SA alone and SA&RRB combined (Gotham et al. 2007, p. 621).
To determine whether the revised algorithms improved diagnostic accuracy, we compared sensitivity, specificity, efficiency (correct classification rate), Positive Predictive Value (PPV), and Negative Predictive Value (NPV) of the original and revised algorithms in relation to the clinical classification of each case in the four developmental cells. In this analysis, cell A was further divided by mental age as indicated by the PEP-R (≤ 15 months cell A-I and ≥ 15 months cell A-II), because Gotham et al. (2007) found that the ADOS is of limited value for children with a mental age of less than 16 months as indicated by a very high percentage of false positive results in this cell (specificity = 0.19). The clinical significance of the indices of diagnostic accuracy were evaluated as: < 0.70 = poor; 0.70–0.79 = fair; 0.80–0.89 = good; 0.90–1.00 = excellent (Cicchetti et al. 1995). We compared the ability of the old and new algorithms to distinguish between: (a) autism and non-spectrum cases, (b) non-autism ASD (PDD-NOS and Asperger’s Syndrome) and non-spectrum cases, and (c) ASD and non-spectrum cases. In total 516 valid cases were available for calculation of diagnostic accuracy; 16 cases were excluded because they had missing data for items included in the original or revised algorithms e.g., an ‘8’ on the ‘Stereotyped/Idiosyncratic use of words or phrases’ item. We also decided to exclude diagnostic categories with fewer than 10 cases per diagnostic category as a small sample size may substantially influence the outcomes. Hence, cell A-I with only 3 non-spectrum cases was removed from this analysis.
In order to accurately interpret ADOS outcomes, we also calculated indices of diagnostic accuracy for the ADI-R, calculated the agreement between ADOS and ADI-R classifications based on Cohen’s Kappa (Cicchetti and Sparrow 1981), determined which variables influenced an incorrect ADOS classification, using logistic regression modeling, and determined the distribution of scores for ADOS item A1 (Overall level of Non-echoed Language), to check whether participants received the appropriate module.
To compare the effects of age and (non)verbal IQ on the original and revised algorithm totals, we used T-tests to evaluate the difference between two dependent correlations with n − 3° of freedom and alpha set at 0.05 (Chen and Popovich 2002). In these analyses the correlations between the new algorithms and age/(non)verbal IQ were compared to the correlation between the old algorithms and age/(non)verbal IQ, while being corrected for the correlation between the new and old algorithms. Cell A was left out of this analysis as verbal IQ and nonverbal IQ could not be established for the majority of the cases in this cell.
To verify the goodness-of-fit of the factor structure of the revised algorithms (SA alone vs. SA&RRB), we applied confirmatory factor analyses (CFA) for categorical data by developmental cell. In these analyses, scores of 8 were marked as missing values and excluded. Items with more than 80% missing values were also omitted. Mplus Version 4.1 software (Muthen and Muthen 1995) was used for these analyses.
Results
Improved Diagnostic Validity of the Revised Algorithms?
Autism Versus Non-Spectrum Disorders
The revised algorithms based on the combined SA&RBB domain total had a clearly better balance between sensitivity and specificity, and between PPV and NPV than the original algorithms, and more cases were correctly classified in cells B, C, and D but not in cell A-II (see Table 3). However, the sensitivity of both the original and revised algorithms was (unacceptably) low in cells B, C, and D (Cicchetti et al. 1995).
Table 3
Pearson correlation and t-tests for correlations between age/(N)VIQ and ADOS totals for the revised and original algorithms
Non-Autism ASD Versus Non-Spectrum Disorders
The diagnostic validity of the revised algorithms based on the SA domain total alone was better than that of the original algorithms in cell A-II and D, whereas it was comparable in cell B and C (see Table 3). In general, the balance between sensitivity and specificity seemed to be better with the revised algorithms (based on SA domain total alone and on the combined SA&RRB domain total); however, in most cells the sensitivity of both the original and revised algorithms was unsatisfactory. In cell C, the PPV of the revised algorithm was considerably lower than that of the original algorithm although it remained above the 0.70 threshold (Cicchetti et al. 1995).
ASD Versus Non-Spectrum Disorders
The revised algorithms based on SA domain total alone distinguished between ASD (autism and non-autism ASD) cases and non-spectrum cases better (improved sensitivity and efficiency) than the original algorithms. Although the specificity of the SA or combined SA&RRB domain totals of the revised algorithms was lower than that of the original algorithms, it was still higher than 0.70 in all cells. The PPV and NPV were not substantially different between the two algorithms. Only in cell D, the PPV increased a lot with the revised algorithm (SA and SA&RRB combined), and the NPV of the revised algorithm was lower than that of the original algorithm in cell A-II, especially with SA&RRB combined.
Additional Analyses to Explain ADOS Outcomes
As the diagnostic sensitivity of the ADOS was surprisingly low in all cells regardless of the algorithm used (old or new), we performed additional analyses. First, we calculated the predictive validity of the ADI-R, using cut-off scores for autism and ASD (see Risi et al. 2006, p. 1100). In general, compared to the ADOS, the sensitivity of the ADI-R (either for autism vs. non-spectrum or for non-autism ASD vs. non-spectrum) was comparable to lower in cell A and B, and comparable to higher in cell C and D. The specificity of the ADI-R was comparable or slightly lower than that of the ADOS. Second, analysis showed the agreement between the ADOS and the ADI-R classifications to be poor (κ < 0.40). Third, we ran a binary logistic regression to determine which variables affected incorrect ADOS classification of cases clinically diagnosed with ASD. The independent variables clinician responsible for final diagnosis and psychologist coding the ADOS did not have a significant influence and were therefore left out from the model. Remaining independent variables that were entered in the model were age at ADOS administration and verbal IQ. Overall, for the old algorithms, the model reached significance (χ2(2) = 17.10, p < .001), with a significant positive result for ‘verbal IQ’ (Odds Ratio = 1.03, CI = 1.01–1.05, p < .001). For the revised algorithms with SA alone and SA&RRB combined the models were also significant (with χ2(2) = 8.38, p < .05 and χ2(2) = 25.05, p < .001 respectively), and with a significant positive effect of verbal IQ (Odds Ratio = 1.02, CI = 1.01–1.04, p < .01, and Odds Ratio = 1.04, CI = 1.02–1.06, p < .001 respectively). Fourth, in order to check that participants had been administered the appropriate module, we looked at the distribution of scores (0, 1, and 2) for ADOS item A1 (Overall Level of Non-echoed Language) in the different developmental cells: it was 0, 0, 58 in cell A-II; 49, 30, 21 in cell B; 95, 5, 0 in cell C; and 95, 4, 1 in cell D. These results indicate that there might have been a ceiling effect in cells C and D (most participants scored 0 on this item), with some cases administered a too easy module.
Age and Effects of Verbal and Nonverbal IQ
With the revised algorithms (SA alone and SARRB combined) no correlation was found between age and ADOS total for cell C. Correlations in all other cells were only low (≤ 0.37; see Table 4). Correlations between the revised ADOS total (SA alone and SARRB combined) and (non)verbal or verbal IQ were low to moderate for all developmental cells (−0.24 to −0.47). In general, age or (non)verbal effects were not diminished using the new algorithms with SARRB compared to the old algorithms. With the SA factor alone, however, for some developmental cells reduced effects of age or (non)verbal IQ were found (see Table 4). See Table 7 (Appendix) for more detailed correlations, also with the domain totals of the ADI-R (Lord et al. 1994).
Table 4
Pearson correlation and t-tests for correlations between age/(N)VIQ and ADOS totals for the revised and original algorithms
Correlation between revised and original algorithm totals | Revised algorithm | Original algorithm |
t
(SA vs. CommSoc)
|
t
(SARRB vs. CommSoc)
| |||
|---|---|---|---|---|---|---|---|
r
SA—Comm-Soc
|
r
SARRB—Comm-Soc
|
r
SA—age
|
r
SARRB—age
|
r
Comm-Soc—age
| |||
Age | |||||||
Cell A-I | 0.97 | 0.94 | 0.23 | 0.32* | 0.33* | −3.17** | −0.05 |
Cell A-II | 0.98 | 0.94 | 0.35** | 0.37** | 0.37** | −0.95 | 0.00 |
Cell B | 0.98 | 0.98 | 0.29** | 0.32** | 0.30** | −1.15 | 0.71 |
Cell C | 0.97 | 0.96 | −0.07 | −0.07 | −0.08 | 0.24 | 0.22 |
Cell D | 0.94 | 0.94 | 0.22* | 0.31** | 0.30** | −2.49* | 0.27 |
r
SA—Comm-Soc
|
r
SARRB—Comm-Soc
|
r
SA—(N)VIQ
|
r
SARRB—(N)VIQ
|
r
Comm-Soc—(N)VIQ
|
t
(SA vs. CommSoc)
|
t
(SARRB vs. CommSoc)
| |
|---|---|---|---|---|---|---|---|
VIQ | |||||||
Cell B | 0.98 | 0.98 | −0.33** | −0.36** | −0.36** | 2.13* | 0.24 |
Cell C | 0.97 | 0.96 | −0.38** | −0.41** | −0.44** | 2.12* | 0.92 |
Cell D | 0.94 | 0.94 | −0.24* | −0.34** | −0.29** | 1.39 | −1.50 |
NVIQ | |||||||
Cell B | 0.98 | 0.98 | −0.36** | −0.41** | −0.40** | 2.33* | 0.55 |
Cell C | 0.97 | 0.96 | −0.32** | −0.36** | −0.34** | 0.77 | −0.58 |
Cell D | 0.94 | 0.94 | −0.35** | −0.47** | −0.42** | 2.07* | −1.54 |
Verification of Factor Structure
In confirmatory factor analysis (CFA), a Comparative Fit Index (CFI; Bentler 1990) between 0.9 and 1 indicates good fit. In addition, a root mean square error approximation (RMSEA) of 0.08 or lower is considered satisfactory (Browne and Cudeck 1993). Using these criteria, we found the two-factor model (SA&RRB) of the revised algorithms to fit the data better than the one-factor model (SA) in cells A, B, and C, with CFI values ranging between 0.96 and 1.00 and RMSEAs ranging between 0.04 and 0.08. The goodness-of-fit rating for cell D was slightly poorer (CFI = 0.87, RMSEA = 0.14). Note that in cell A item A3 ‘Intonation’ was scored in only a few cases (11.7%), and was excluded from the analysis.
In their analyses, Gotham et al. (2007) allowed a third factor, called “Joint Attention”, which was consistently confirmed across Modules 1 and 2 in their study. This factor comprised response to joint attention, gesturing, showing, initiating joint attention, and unusual eye contact in cell A, and pointing, gesturing, showing, initiating joint attention, and unusual eye contact in cells B, C, and D. CFA of a three-factor model in the current study was satisfactory for cell A (CFI = 1.00, RMSEA = 0.03), cell B (CFI = 0.96, RMSEA = 0.08), and cell C (CFI = 1.00, RMSEA = 0.05) but less satisfactory for cell D (CFI = 0.89, RMSEA = 0.13). In general, outcomes with the three-factor model were only marginally better than with the two-factor model.
Across modules, 5 items or less had communalities (the percentage of variance in a given item explained by all of the factors) less than 0.4, and only 1 or 2 items had communalities below 0.2, except for cell D that included 4 items with moderate communalities (0.2–0.4) and 4 items with low communalities (< 0.2). The communalities for gestures and hand mannerisms were consistently low or moderate across modules. The number of items with low or moderate communalities did not differ much between the two- or three-factor models. See Tables 8 and 9 (Appendices) for correlations of the 14 items with the two- and three-factor models.
Discussion
This study replicates previous findings on the diagnostic validity of the revised algorithms for ADOS Modules 1 and 2, proposed by Gotham et al. (2007), in a relatively large independent Dutch sample. Overall, the children in our sample were younger and higher functioning than those included in previous replication studies and with a different distribution of clinical diagnoses (autism, non-autism ASD, and non-spectrum disorders). For some specific cells, our study fills up some gaps in the literature, namely for the Modules 1/Some Words cell and for the Module 2/Younger as well as the Module 2/Older than 5 years cell.
Regarding our main aim, to investigate whether the revised algorithms improved the diagnostic validity of the ADOS compared to the original algorithms, we found: (a) increased predictive validity for autism cases with the revised algorithms (combined Social Affect and Restricted, Repetitive Behavior domains—SA&RRB) except for participants who were administered Module 1/No words and had a mental age of > 15 months (cell A-II), (b) less consistent improvement in predictive validity for non-autism ASD cases, and (c) increased predictive validity for ASD cases as a whole, especially when the revised algorithms based on the SA domain total were used. Previous studies generally indicated that the sensitivity, specificity, and/or efficiency of the revised algorithms were the same or better than those of the original algorithms (De Bildt et al. 2009; Gotham et al. 2007, 2008; Gray et al. 2008; Overton et al. 2008). Our findings support the use of the more homogeneous revised algorithms, with similar items used across developmental cells to allow for easier comparison of ADOS scores within and between individuals.
In previous research, the only exception to the overall finding of improved predictive validity of the revised algorithms was the performance of the ADOS revised algorithm for non-autism ASD in young children administered Module 2 (cell C). Gotham et al. (2008) reported a marked decrease in sensitivity for non-autism ASD with the revised algorithm based on combined SA&RRB domain total compared with the original algorithm in this cell (0.65 vs. 0.88). In cell C (with 17 non-autism ASD and 18 non-spectrum participants), they reported a high mean score (4) for the ADI-R restricted, repetitive and stereotyped behavior domain but a low ADOS mean score for this domain (RRB score of 1.3), which suggests that the ADOS may miss this type of behavior. We found a similar pattern in cell C, which consisted of 41 individuals with non-autism ASD and 30 individuals with non-spectrum disorders. However, compared to the original algorithms, we did not observe the decrease in sensitivity in our substantially larger sample. This suggests that the revised algorithm also has improved diagnostic validity in children with non-autism ASD who have phrase speech and are younger than 5 years.
The sensitivity of the revised algorithms was much lower in the different developmental cells than that reported in previous studies, whereas the specificity was comparable or even better (De Bildt et al. 2009; Gotham et al. 2007, 2008; Gray et al. 2008). As most cases, including the non-spectrum cases, had been referred for suspected ASD, we would have expected the opposite, namely, a high sensitivity with a lower specificity. The high specificity of the ADOS found in our study supports the usability of the ADOS for research purposes in order to select narrowly defined groups of participants. However, the low sensitivity is a problem, because it is important not to miss cases. There are a number of possible explanations for this discrepancy in sensitivity with earlier studies:
(a)
In some cells, the sample size per diagnostic group was limited, which may have influenced the results. Gotham et al. (2007, 2008) excluded cells with fewer than 15 participants per diagnostic group, whereas we included some diagnostic groups with only 10 cases (Module 1/No Words, mental age > 15 months) or 14 cases (Module 2, > 5 years) per cell.
(b)
Flaws in ADOS coding could distort results. However, this is improbable because our teams were well trained, and five members are recognized as ADOS trainers by leading institutions in the UK or USA. The trainers supervised less experienced colleagues in order to ensure reliability.
(c)
Sample variation may play a role. Most children in this study were clinically referred within the context of an extensive early screening project (see Oosterling et al. 2009). The fact that our sample was younger, higher functioning, and included fewer core cases of autism than the samples in other studies may have given rise to misclassification. Higher functioning children often show milder ASD symptoms or they may “cover up” their weaknesses in a semi-structured one-to-one context. As the disabilities of higher-functioning ASD children are often more evident in complex situations of daily life, we would expect the ADI-R to be more sensitive, which proved to be the case, although only modestly so, for participants who were administered Module 2.
(d)
Another point concerns how diagnoses were established. As the agreement between ADOS and ADI-R classifications is poor, clinicians must have often relied on only one of the instruments in assigning a DSM-IV diagnosis (APA 2000) in addition to observations from other assessments. As the clinicians who made the final diagnoses were highly experienced and the independent variable ‘clinician responsible for final diagnosis’ was not a predictor of the ADOS false negative cases, there is little reason to question the quality of clinical diagnoses made in this study. However, it is a matter of speculation to what extent differences in ADOS outcomes between the sites included in this study and those in the original study performed in the USA are related to differences in assigning clinical diagnoses (clinician based vs. DSM-IV based). The definition of non-autism ASD might be broader in Europe than in the USA, or the distinction between autism and non-autism ASD might be slightly different in clinical practice. For this reason, we added the ASD versus non-spectrum disorders comparison in the analyses, which improved diagnostic validity relative to the more strict comparison (autism vs. non-spectrum disorders and non-autism ASD vs. non-spectrum disorders), especially when the revised algorithms based on SA alone were used.
(e)
Finally, the choice of module may have influenced results. Guidelines for module choice are clearly described (Lord et al. 2001) but in clinical practice the choice of whether to administer Module 1 or Module 2 is sometimes difficult and arbitrary. In this regard, Klein-Tasman et al. (2007) found that administration of an ‘easier’ module instead of an appropriate module resulted in under classification of autism in participants administered Module 1/2, and in under classification of ASD in participants administered Module 2/3. Therefore, we sought to verify whether the choice of module could have been responsible for the low sensitivity in our study. Despite careful consideration by the diagnostic team of which module to administer, the distribution of item A1 (Level of Non-echoed Language), with a disproportionately large number of 0 scores, suggests that the choice of module resulted in under classification of at least some cases. This notion is supported by the observation that verbal IQ was higher in incorrectly classified cases than in correctly classified cases.
To recapitulate, while a combination of factors could have led to the relatively low sensitivity in our study, which is a problem, this does not detract from our overall finding that the diagnostic validity of the revised algorithms is better than that of the original algorithms.
Interestingly, the revised algorithms based on a single factor (SA) were better than revised algorithms based on two factors (combined SA&RRB) for identifying cases of non-autism ASD in the youngest/lowest functioning group (cell A-II) and in the oldest/highest functioning group (cell D). Although these findings are not in line with Gotham et al. (2007) and are based on a limited amount of data, at least for cell A-II, they might indicate that restricted and repetitive behavior is of less diagnostic value in very young or low-functioning children (cell A-II) and/or older or higher functioning children at the milder ends of the spectrum (cell D) than in other children.
One of the aims of the revising the ADOS algorithms was to minimize the effect of verbal and nonverbal IQ on ADOS totals (Gotham et al. 2007). However, we found a diminished effect of IQ (verbal and nonverbal) on revised algorithm totals in only a few cells and only with regard to the SA domain total. This suggests that RRB items are (partly) responsible for the influence of verbal/nonverbal IQ on the combined SA&RRB domain total. This is contrary to previous research (De Bildt et al. 2009; Gotham et al. 2007, 2008), which generally found verbal and nonverbal IQ not to affect the ADOS total calculated with the revised algorithms. The fact that our sample was higher functioning than the samples in previous studies could play a role in this difference.
We found that age did not affect the revised algorithm (SA&RRB) domain total in cell C and had only a minor effect in the other cells; the magnitude of this effect was not smaller with the revised algorithm than with the original algorithm, consistent with the findings of other studies (Gotham et al. 2007, 2008; De Bildt et al. 2009).
We found the two-factor model (SA&RRB) of the revised algorithms to fit the data better than the one-factor model (SA), with a slightly unsatisfactory fit for cell D only. In both independent samples of Gotham et al. (2007, 2008), the two-factor model provided a better fit than the one-factor model in all developmental cells. While variation in factor structure is to be expected across samples, the slightly unsatisfactory fit of the model in cell D may be due to the difference in the distribution of diagnoses across samples, which was most apparent in this cell. In our sample, cell D included relatively few autism cases, while autism was overrepresented in cell D in the two studies by Gotham et al. In addition to the two-factor model, Gotham et al. (2007) found a three-factor model, with ‘joint attention’ as the third factor, to fit the data better for both cells A and B, but they decided to use the two-factor model due to its greater consistency across developmental cells. In the current study, the three-factor model was not better in any developmental cell, supporting Gotham et al.’s decision to use the two-factor model.
Limitations
There are some limitations to this study. We did not include any participants assessed with Module 3, which limited the comparison with previous studies. However, we did fill up gaps in the literature on other modules. Moreover, the small sample size of some cells (A-II and D) limited the interpretation of some results and prevented analysis of algorithm performance in cell A-I. However, in this regard, a new Toddler Module of the ADOS with novel tasks for infants and toddlers has been reported by Luyster et al. (2009) to meet the diagnostic conditions applicable to very young and/or severely delayed children. A final limitation is that clinical diagnoses were made on the basis of the old algorithms, which may have led to biased outcome measures for the original algorithms. Given this bias, one would have expected the diagnostic agreement to be better with the original algorithms than with the revised algorithms, which was not the case, thereby emphasizing the improved diagnostic validity of the revised algorithms.
Clinical Implications
In general, the current replication study of ADOS Modules 1 and 2 in a large, independent and well-defined population shows that the revised ADOS algorithms (Gotham et al. 2007) improve diagnostic validity compared with the original algorithms. The improvement in predictive validity was most apparent for autism; the sensitivity and specificity of the revised algorithms for non-autism ASDs were only marginally better than the original algorithms. The low sensitivity of the revised algorithms may be problematic for some developmental cells or diagnostic subgroups since this reflects increasing discrepancy between the ADOS algorithm and clinical diagnosis. Although the source of the lower sensitivity merits further study, it does once again emphasis that a diagnosis of ASD requires a multidisciplinary approach that includes a variety of assessment measures.
We confirmed the factor structure proposed by Gotham et al. (2007), and the revised algorithms were minimally influenced by age; however, they were not entirely independent of (non)verbal IQ. Overall, our study continues to support the use of the more homogeneous new ADOS algorithms for clinical and research purposes.
Acknowledgements
We gratefully acknowledge all children and parents who participated in our research project and all clinicians who contributed to the data gathering, especially Nicky de Waal, Tim Woudenberg, Sammy Roording-Ragetlie and Kina Potze. This study was supported by a grant from the Korczak Foundation.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
