Skip to main content
SpringerLink
Log in

The objectivity of the Autism Diagnostic Observation Schedule (ADOS) in naturalistic clinical settings

  • Original Contribution
  • Published:
European Child & Adolescent Psychiatry Aims and scope Submit manuscript

Abstract

The Autism Diagnostic Observation Schedule (ADOS) is a first-choice diagnostic tool in autism spectrum disorder (ASD). Excellent interpersonal objectivity (interrater reliability) has been demonstrated for the ADOS under optimal conditions, i.e., within groups of highly trained “research reliable” examiners in research setting. We investigated the spontaneous interrater reliability among clinically trained ADOS users across multiple sites in clinical routine. Forty videotaped administrations of the ADOS modules 1–4 were rated by five different raters each from a pool of in total 15 raters affiliated to 13 different clinical sites. G(q,k) coefficients (analogous to intraclass correlations), kappas (ĸ) and percent agreement (PA) were calculated. The median interrater reliability for items across the four modules was G(q,k) = .74–.83, with the single ADOS items ranging from .23 to .94. G(q,k) for total scores was .85–.92. For diagnostic classification (ASD/non-spectrum), PA was 64–82 % and Fleiss’ ĸ .19–.55. Objectivity was lower for pervasive developmental disorder not otherwise specified and non-spectrum diagnoses as compared to autism. Interrater reliabilities of the ADOS items and domain totals among clinical users across multiple sites were in the same range as previously reported for research reliable users, while the one for diagnostic classification was lower. Differences in sample characteristics, rater skills and statistics compared with previous studies are discussed. Findings endorse the objectivity of the ADOS in naturalistic clinical settings, but also pinpoint its limitations and the need and value of adequate and continuous rater training.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Klin A, Lang J, Cicchetti DV, Volkmar FR (2000) Brief report: interrater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: results of the DSM-IV autism field trial. J Autism Dev Disord 30(2):163–167

    Article  CAS  PubMed  Google Scholar 

  2. Volkmar F, Siegel M, Woodbury-Smith M, King B, McCracken J, State M, American Academy of C, Adolescent Psychiatry Committee on Quality I (2014) Practice parameter for the assessment and treatment of children and adolescents with autism spectrum disorder. J Am Acad Child Adolesc Psychiatry 53(2):237–257

    Article  PubMed  Google Scholar 

  3. Autism: Recognition, Referral, Diagnosis and Management of Adults on the Autism Spectrum (2012). The British Psychological Society & The Royal College of Psychiatrists. Leicester, UK

  4. Swedish Council on Health Technology Assessment (SBU) (2013) Autismspektrumtillstånd. Diagnostik och insatser, vårdens organisation och patientens delaktighet—En systematisk litteraturöversikt, vol 215. Swedish Council on Health Technology Assessment

  5. American Psychiatric Association (APA) (2013) Diagnostic and statistical manual of mental disorders DSM-5, 5th edn. American Psychiatric Association, Arlington

    Google Scholar 

  6. Lord C, Rutter M, DiLavore P, Risi S (1999) Autism Diagnostic Observation Schedule (ADOS). Western Psychological Publishing, Los Angeles

    Google Scholar 

  7. Lord C, Rutter M, DiLavore P, Risi S, Gotham K, Bishop S (2012) Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) Manual (Part I). Western Psychological Services, Torrance

    Google Scholar 

  8. Rutter M, Le Couteur A, Lord C (2003) Autism Diagnostic Interview Revised (ADI-R). Western Psychological Services, Los Angeles

    Google Scholar 

  9. Molloy CA, Murray DS, Akers R, Mitchell T, Manning-Courtney P (2011) Use of the Autism Diagnostic Observation Schedule (ADOS) in a clinical setting. Autism 15(2):143–162

    Article  PubMed  Google Scholar 

  10. Akshoomoff N, Corsello C, Schmidt H (2006) The role of the autism diagnostic observation schedule in the assessment of autism spectrum disorders in school and community settings. Calif School Psychol 11:7–19

    Article  PubMed  PubMed Central  Google Scholar 

  11. Ashwood KL, Buitelaar J, Murphy D, Spooren W, Charman T (2015) European clinical network: autism spectrum disorder assessments and patient characterisation. Eur Child Adolesc Psychiatry 24(8):985–995

    Article  PubMed  Google Scholar 

  12. Wolff JJ, Gu H, Gerig G, Elison JT, Styner M, Gouttard S, Botteron KN, Dager SR, Dawson G, Estes AM, Evans AC, Hazlett HC, Kostopoulos P, McKinstry RC, Paterson SJ, Schultz RT, Zwaigenbaum L, Piven J (2012) Differences in white matter fiber tract development present from 6 to 24 months in infants with autism. Am J Psychiatry 169(6):589–600

    Article  PubMed  PubMed Central  Google Scholar 

  13. Bryson SE, Zwaigenbaum L, Brian J, Roberts W, Szatmari P, Rombough V, McDermott C (2007) A prospective case series of high-risk infants who developed autism. J Autism Dev Disord 37(1):12–24

    Article  PubMed  Google Scholar 

  14. Ozonoff S, Young GS, Belding A, Hill M, Hill A, Hutman T, Johnson S, Miller M, Rogers SJ, Schwichtenberg AJ, Steinfeld M, Iosif AM (2014) The broader autism phenotype in infancy: when does it emerge? J Am Acad Child Adolesc Psychiatry 53(4):398–407

    Article  PubMed  PubMed Central  Google Scholar 

  15. Bölte S, Poustka F (2004) Diagnostic Observation Scale for Autistic Disorders: initial results of reliability and validity. Z Kinder Jugendpsychiatr Psychother 32(1):45–50

    Article  PubMed  Google Scholar 

  16. Zander E, Sturm H, Bölte S (2015) The added value of the combined use of the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule: Diagnostic validity in a clinical Swedish sample of toddlers and young preschoolers. Autism 19(2):187–199

    Article  PubMed  Google Scholar 

  17. Gotham K, Risi S, Pickles A, Lord C (2007) The Autism Diagnostic Observation Schedule: revised algorithms for improved diagnostic validity. J Autism Dev Disord 37(4):613–627

    Article  PubMed  Google Scholar 

  18. Gotham K, Risi S, Dawson G, Tager-Flusberg H, Joseph R, Carter A, Hepburn S, McMahon W, Rodier P, Hyman SL, Sigman M, Rogers S, Landa R, Spence MA, Osann K, Flodman P, Volkmar F, Hollander E, Buxbaum J, Pickles A, Lord C (2008) A replication of the Autism Diagnostic Observation Schedule (ADOS) revised algorithms. J Am Acad Child Adolesc Psychiatry 47(6):642–651

    Article  PubMed  PubMed Central  Google Scholar 

  19. de Bildt A, Sytema S, van Lang ND, Minderaa RB, van Engeland H, de Jonge MV (2009) Evaluation of the ADOS revised algorithm: the applicability in 558 Dutch children and adolescents. J Autism Dev Disord 39(9):1350–1358

    Article  PubMed  PubMed Central  Google Scholar 

  20. Bastiaansen JA, Meffert H, Hein S, Huizinga P, Ketelaars C, Pijnenborg M, Bartels A, Minderaa R, Keysers C, de Bildt A (2011) Diagnosing autism spectrum disorders in adults: the use of Autism Diagnostic Observation Schedule (ADOS) module 4. J Autism Dev Disord 41(9):1256–1266

    Article  PubMed  Google Scholar 

  21. Gray KM, Tonge BJ, Sweeney DJ (2008) Using the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule with young children with developmental delay: evaluating diagnostic validity. J Autism Dev Disord 38(4):657–667

    Article  PubMed  Google Scholar 

  22. Hus V, Lord C (2014) The autism diagnostic observation schedule, module 4: revised algorithm and standardized severity scores. J Autism Dev Disord 44(8):1996–2012

    Article  PubMed  PubMed Central  Google Scholar 

  23. Kim SH, Lord C (2012) Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. J Child Psychol Psychiatry 53(2):143–151

    Article  PubMed  Google Scholar 

  24. Klein-Tasman BP, Risi S, Lord CE (2007) Effect of language and task demands on the diagnostic effectiveness of the autism diagnostic observation schedule: the impact of module choice. J Autism Dev Disord 37(7):1224–1234

    Article  PubMed  Google Scholar 

  25. Le Couteur A, Haden G, Hammal D, McConachie H (2008) Diagnosing autism spectrum disorders in pre-school children using two standardised assessment instruments: the ADI-R and the ADOS. J Autism Dev Disord 38(2):362–372

    Article  PubMed  Google Scholar 

  26. Lord C, Risi S, Lambrecht L, Cook EH Jr, Leventhal BL, DiLavore PC, Pickles A, Rutter M (2000) The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord 30(3):205–223

    Article  CAS  PubMed  Google Scholar 

  27. Oosterling I, Roos S, de Bildt A, Rommelse N, de Jonge M, Visser J, Lappenschaar M, Swinkels S, van der Gaag RJ, Buitelaar J (2010) Improved diagnostic validity of the ADOS revised algorithms: a replication study in an independent sample. J Autism Dev Disord 40(6):689–703

    Article  PubMed  PubMed Central  Google Scholar 

  28. Lord C, Rutter M, Goode S, Heemsbergen J, Jordan H, Mawhood L, Schopler E (1989) Autism diagnostic observation schedule: a standardized observation of communicative and social behavior. J Autism Dev Disord 19(2):185–212

    Article  CAS  PubMed  Google Scholar 

  29. DiLavore PC, Lord C, Rutter M (1995) The pre-linguistic autism diagnostic observation schedule. J Autism Dev Disord 25(4):355–379

    Article  CAS  PubMed  Google Scholar 

  30. de Bildt A, Sytema S, Meffert H, Bastiaansen JCJ (2015) The Autism Diagnostic observation schedule, module 4: application of the revised algorithms in an independent, well-defined, Dutch sample (n = 93). J Autism Dev Disord 30 August 2015. doi:10.1007/s10803-015-2532-4

  31. McClure I, Mackay T, Mamdani H, McCaughey R (2010) A comparison of a specialist autism spectrum disorder assessment team with local assessment teams. Autism 14(6):589–603

    Article  PubMed  Google Scholar 

  32. Kim YS, Leventhal BL, Koh YJ, Fombonne E, Laska E, Lim EC, Cheon KA, Kim SJ, Kim YK, Lee H, Song DH, Grinker RR (2011) Prevalence of autism spectrum disorders in a total population sample. Am J Psychiatry 168(9):904–912

    Article  PubMed  Google Scholar 

  33. Fombonne E, Marcin C, Bruno R, Tinoco CM, Marquez CD (2012) Screening for autism in Mexico. Autism Res 5(3):180–189

    Article  PubMed  Google Scholar 

  34. Schendel DE, Diguiseppi C, Croen LA, Fallin MD, Reed PL, Schieve LA, Wiggins LD, Daniels J, Grether J, Levy SE, Miller L, Newschaffer C, Pinto-Martin J, Robinson C, Windham GC, Alexander A, Aylsworth AS, Bernal P, Bonner JD, Blaskey L, Bradley C, Collins J, Ferretti CJ, Farzadegan H, Giarelli E, Harvey M, Hepburn S, Herr M, Kaparich K, Landa R, Lee LC, Levenseller B, Meyerer S, Rahbar MH, Ratchford A, Reynolds A, Rosenberg S, Rusyniak J, Shapira SK, Smith K, Souders M, Thompson PA, Young L, Yeargin-Allsopp M (2012) The Study to Explore Early Development (SEED): a multisite epidemiologic study of autism by the Centers for Autism and Developmental Disabilities Research and Epidemiology (CADDRE) network. J Autism Dev Disord 42(10):2121–2140

    Article  PubMed  PubMed Central  Google Scholar 

  35. Putka DJ, Le H, McCloy RA, Diaz T (2008) Ill-structured measurement designs in organizational research: implications for estimating interrater reliability. J Appl Psychol 93(5):959–981

    Article  PubMed  Google Scholar 

  36. Hallgren KA (2012) Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 8(1):23–34

    PubMed  PubMed Central  Google Scholar 

  37. Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88(2):322–328

    Article  Google Scholar 

  38. Cicchetti DV, Sparrow SA (1981) Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 86(2):127–137

    CAS  PubMed  Google Scholar 

  39. Shavelson RJ, Webb NM (2006) Generalizability theory. Handbook of complementary methods in education research. Lawrence Erlbaum Associates Publishers, Mahwah, pp 309–322

    Google Scholar 

  40. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382

    Article  Google Scholar 

  41. Cohen J (1960) A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  42. de Vet HC, Terwee CB, Knol DL, Bouter LM (2006) When to use agreement versus reliability measures. J Clin Epidemiol 59(10):1033–1039

    Article  PubMed  Google Scholar 

  43. Cicchetti DV (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 6(4):284–290

    Article  Google Scholar 

  44. Cicchetti DV, Volkmar F, Klin A, Showalter D (1995) Diagnosing Autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychol 1(1):26–37

    Article  Google Scholar 

  45. Volkmar FR, Klin A, Siegel B, Szatmari P, Lord C, Campbell M, Freeman BJ, Cicchetti DV, Rutter M, Kline W et al (1994) Field trial for autistic disorder in DSM-IV. Am J Psychiatry 151(9):1361–1367

    Article  CAS  PubMed  Google Scholar 

  46. Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, Gabbard GO, Gau SS, Javitt DC, Oquendo MA, Shrout PE, Vieta E, Yager J (2013) The initial field trials of DSM-5: new blooms and old thorns. Am J Psychiatry 170(1):1–5

    Article  PubMed  Google Scholar 

  47. Charman T, Gotham K (2013) Measurement Issues: Screening and diagnostic instruments for autism spectrum disorders—lessons from research and practice. Child Adolesc Ment Health 18(1):52–63

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We thank the Child and Adolescent Psychiatry of Stockholm County Council, Region Västra Götaland and Region Skåne, PRIMA Child and Adult Psychiatry, Tiohundra Norrtälje and Astrid Lindgren Children’s Hospital and Sachs’ Children and Youth Hospital, Stockholm County Council, for the generous cooperation in realizing this study. We also thank our funders Frimurare Barnhuset, Swedish Research Council, Stockholm County Council and finally Karolinska Institutet. Finally, we are especially grateful toward all the children and parents that participated in the project.

Author information

Authors and Affiliations

  1. Pediatric Neuropsychiatry Unit, Department of Women’s and Children’s Health, Center of Neurodevelopmental Disorders (KIND), Karolinska Institutet, Gävlegatan 22B, 113 30, Stockholm, Sweden

    Eric Zander, Charlotte Willfors, Steve Berggren, Nora Choque-Olsson, Christina Coco & Sven Bölte

  2. Child and Adolescent Psychiatry, Center for Psychiatry Research, Stockholm County Council, Stockholm, Sweden

    Eric Zander, Charlotte Willfors, Steve Berggren, Nora Choque-Olsson, Christina Coco & Sven Bölte

  3. Neurodevelopmental Psychiatry Unit South East, Child and Adolescent Psychiatry, Stockholm County Council, Stockholm, Sweden

    Eric Zander

  4. BUP Södertälje, Child and Adolescent Psychiatry, Stockholm County Council, Södertälje, Sweden

    Nora Choque-Olsson

  5. Neuropediatric Unit, Sachs’ Children and Youth Hospital, Stockholm County Council, Stockholm, Sweden

    Christina Coco & Viviann Nordin

  6. PRIMA Child and Adolescent Psychiatry, Stockholm, Sweden

    Anna Elmund & Lina Poltrago

  7. Citypsykologhus, Stockholm, Sweden

    Anna Elmund

  8. BUP Sollentuna, Child and Adolescent Psychiatry, Stockholm County Council, Sollentuna, Sweden

    Åsa Hedfors Moretti

  9. Astrid Lindgrens Children’s Hospital, Karolinska University Hospital Solna, Stockholm County Council, Solna, Sweden

    Anette Holm

  10. Astrid Lindgrens Children’s Hospital, Karolinska University Hospital Huddinge, Stockholm County Council, Huddinge, Sweden

    Ida Jifält

  11. Pupil Health Unit, Tiohundra, Norrtälje, Sweden

    Ida Jifält

  12. BUP Malmö, Child and Adolescent Psychiatry, Region Skåne, Malmö, Sweden

    Renata Kosieradzki

  13. Pupil Health Unit, Resource Team for Learning Disabled, City of Malmö, Sweden

    Renata Kosieradzki

  14. Södra Älvsborg Hospital (SÄS), Child and Adolescent Psychiatry, Region Västra Götaland, Borås, Sweden

    Jenny Linder

  15. BUP Lund, Child and Adolescent Psychiatry, Region Skåne, Lund, Sweden

    Karin Olafsdottir

Authors
  1. Eric Zander
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charlotte Willfors
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Steve Berggren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nora Choque-Olsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christina Coco
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anna Elmund
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Åsa Hedfors Moretti
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Anette Holm
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ida Jifält
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Renata Kosieradzki
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jenny Linder
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Viviann Nordin
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Karin Olafsdottir
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Lina Poltrago
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Sven Bölte
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric Zander.

Ethics declarations

Conflicts of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 133 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zander, E., Willfors, C., Berggren, S. et al. The objectivity of the Autism Diagnostic Observation Schedule (ADOS) in naturalistic clinical settings. Eur Child Adolesc Psychiatry 25, 769–780 (2016). https://doi.org/10.1007/s00787-015-0793-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00787-015-0793-2

Keywords

Search

Navigation