Top

Journal of Autism and Developmental Disorders

Gepubliceerd in:

01-12-2013 | Original Paper

Reliability Generalization for Childhood Autism Rating Scale

Auteurs: Jonathan Breidbord, Tim J. Croudace

Gepubliceerd in: Journal of Autism and Developmental Disorders | Uitgave 12/2013

Abstract

The Childhood Autism Rating Scale (CARS) is a popular behavior-observation instrument that was developed more than 34 years ago and has since been adopted in a wide variety of contexts for assessing the presence and severity of autism symptomatology in both children and adolescents. This investigation of the reliability of CARS scores involves meta-analysis and meta-regression of empirical data from reports of original research that made use of CARS between 1980 and 2012. Findings of good internal consistency (.896, 95 % CI .877–.913) and good interrater reliability (.796, 95 % CI .736–.844) support use of CARS at least in early-phase, exploratory research. Evidence of heterogeneity among literature data indicates that reliability is a property of CARS scores and is not intrinsic to the instrument itself. As the first of its kind pertaining to autism, this investigation provides guidance for reviews of other instruments’ ratings.

vorige artikel Children’s Compliance with American Academy of Pediatrics’ Well-Child Care Visit Guidelines and the Early Detection of Autism

volgende artikel Cognitive Enhancement Therapy for Adults with Autism Spectrum Disorder: Results of an 18-month Feasibility Study

Arbelle, S., Sigman, M. D., & Kasari, C. (1994). Compliance with parental prohibition in autistic children. Journal of Autism and Developmental Disorders, 24, 693–702.PubMedCrossRef

Arizona Software. (2008). GraphClick graph-digitization software: Release 3.0 [computer software]. Neuchâtel: Arizona Software.

Bagby, R. M., Ryder, A. G., Schuller, D. B., & Marshall, M. B. (2004). The Hamilton Depression Rating Scale: Has the gold standard become a lead weight? American Journal of Psychiatry, 161, 2163–2177.PubMedCrossRef

Baghdadli, A., Assouline, B., Sonié, S., Pernon, E., Darrou, C., Michelon, C., et al. (2012). Developmental trajectories of adaptive behaviors from early childhood to adolescence in a cohort of 152 children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 1314–1325.PubMedCrossRef

Baker, W. L., White, C. M., Cappelleri, J. C., Kluger, J., & Coleman, C. I. (2009). Understanding heterogeneity in meta-analysis: The role of meta-regression. International Journal of Clinical Practice, 63, 1426–1434.PubMedCrossRef

Bebko, J. M., Konstantareas, M. M., & Springer, J. (1987). Parent and professional evaluations of family stress associated with characteristics of autism. Journal of Autism and Developmental Disorders, 17, 565–576.PubMedCrossRef

Bernabei, P., Cerquiglini, A., Cortesi, F., & D’Ardia, C. (2007). Regression versus no regression in the autistic disorder: Developmental trajectories. Journal of Autism and Developmental Disorders, 37, 580–588.PubMedCrossRef

Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One, 5, 1–10.

Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 4, 386–397.CrossRef

Bristol, M. M. (1987). Mothers of children with autism or communication disorders: Successful adaptation and the Double ABCX model. Journal of Autism and Developmental Disorders, 17, 469–486.PubMedCrossRef

Campbell, J. S. (2003). Meta-analysis and reliability generalization of the Psychopathy Checklist applied to youth. Unpublished doctoral dissertation from University of Northern Colorado.

Chen, G. M., Yoder, J. J., Ganzei, B. L., Goodwin, M. S., & Belmonte, M. K. (2012). Harnessing repetitive behaviours to engage attention and learning in a novel therapy for autism: An exploratory analysis. Frontiers in Psychology, doi:10.3389/fpsyg.2012.00012.

Chez, M. G., & Buchanan, C. P. (2000). Comment: Reply: Reply to B. Rimland’s “Comments on ‘Secretin and autism: A two-part clinical investigation’”. Journal of Autism and Developmental Disorders, 30, 97–98.CrossRef

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.CrossRef

Connelly, B. S. (2008). The reliability, convergence, and predictive validity of personality ratings: An other perspective. Unpublished doctoral dissertation from University of Minnesota.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.CrossRef

Duhachek, A., & Iacobucci, D. (2004). Alpha’s standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 5, 792–808.CrossRef

Fein, D., Skoff, B., & Mirsky, A. F. (1981). Clinical correlates of brainstem dysfunction in autistic children. Journal of Autism and Developmental Disorders, 11, 303–315.PubMedCrossRef

Freeman, N. L., Perry, A., & Factor, D. C. (1991). Child behaviours as stressors: Replicating and extending the use of the CARS as a measure of stress: A research note. Journal of Child Psychology and Psychiatry, 32, 1025–1030.PubMedCrossRef

Hakstian, A. R., & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219–231.CrossRef

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.CrossRef

Hobson, R. P., Lee, A., & Brown, R. (1999). Autism and congenital blindness. Journal of Autism and Developmental Disorders, 29, 45–56.PubMedCrossRef

Howell, R. T., & Shields, A. L. (2008). The file drawer problem in reliability generalization: A strategy to compute fail-safe N with reliability coefficients. Educational and Psychological Measurement, 68, 120–128.CrossRef

Hranilovic, D., Bujas-Petkovic, Z., Vragovic, R., Vuk, T., Hock, K., & Jernej, B. (2007). Hyperserotonemia in adults with autistic disorder. Journal of Autism and Developmental Disorders, 37, 1934–1940.PubMedCrossRef

Kasari, C., & Sigman, M. (1997). Linking parental perceptions to interactions in young children with autism. Journal of Autism and Developmental Disorders, 27, 39–57.PubMedCrossRef

Konstantareas, M. M., & Homatidis, S. (1989). Assessing child symptom severity and stress in parents of autistic children. Journal of Child Psychology and Psychiatry, 30, 459–470.PubMedCrossRef

Konstantareas, M. M., & Homatidis, S. (1999). Chromosomal abnormalities in a series of children with autistic disorder. Journal of Autism and Developmental Disorders, 29, 275–285.PubMedCrossRef

Konstantareas, M. M., Homatidis, S., & Plowright, C. M. S. (1992). Assessing resources and stress in parents of severely dysfunctional children through the Clarke modification of Holroyd’s Questionnaire on Resources and Stress. Journal of Autism and Developmental Disorders, 22, 217–234.PubMedCrossRef

Konstantareas, M. M., & Stewart, K. (2006). Affect regulation and temperament in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 36, 143–154.PubMedCrossRef

Kurita, H., Miyake, Y., & Katsuno, K. (1989). Reliability and validity of the Childhood Autism Rating Scale-Tokyo version (CARS–TV). Journal of Autism and Developmental Disorders, 19, 389–396.PubMedCrossRef

Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202–220.CrossRef

Laushey, K. M., & Heflin, L. J. (2000). Enhancing social skills of kindergarten children with autism through the training of multiple peers as tutors. Journal of Autism and Developmental Disorders, 30, 183–193.PubMedCrossRef

LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.CrossRef

Li, A., & Bagger, J. (2007). The balanced inventory of desirable responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525–544.CrossRef

Lord, C., Schopler, E., & Revicki, D. (1982). Sex differences in autism. Journal of Autism and Developmental Disorders, 12, 317–330.PubMedCrossRef

Matese, M., Matson, J. L., & Sevin, J. (1994). Comparison of psychotic and autistic children using behavioral observation. Journal of Autism and Developmental Disorders, 24, 83–94.PubMedCrossRef

Matson, J. L., Smiroldo, B. B., & Hastings, T. L. (1998). Validity of the autism/pervasive developmental disorder subscale of the Diagnostic Assessment for the Severely Handicapped-II. Journal of Autism and Developmental Disorders, 28, 77–81.PubMedCrossRef

Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., et al. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research, 18, 313–333.PubMedCrossRef

Nassen, L. M. (2008). An internal consistency reliability generalization of the Child Behavior Checklist. Unpublished doctoral dissertation from University of Northern Colorado.

Njardvik, U., Matson, J. L., & Cherry, K. E. (1999). A comparison of social skills in adults with autistic disorder, pervasive developmental disorder not otherwise specified, and mental retardation. Journal of Autism and Developmental Disorders, 29, 287–295.PubMedCrossRef

Nordin, V., Gillberg, C., & Nyden, A. (1998). The Swedish version of the Childhood Autism Rating Scale in a clinical setting. Journal of Autism and Developmental Disorders, 28, 69–75.PubMedCrossRef

Pereira, T. V., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010). Critical interpretation of Cochran’s Q test depends on power and prior assumptions about heterogeneity. Research Synthesis Methods, 1, 149–161.CrossRef

Perry, A., Condillac, R. A., Freeman, N. L., Dunn-Geier, J., & Belair, J. (2005). Multi-site study of the Childhood Autism Rating Scale (CARS) in five clinical groups of young children. Journal of Autism and Developmental Disorders, 35, 625–634.PubMedCrossRef

Ponterotto, J. G., & Charter, R. A. (2009). Statistical extensions of Ponterotto and Ruckdeschel’s (2007) reliability matrix for estimating the adequacy of internal consistency coefficients. Perceptual and Motor Skills, 108, 878–886.PubMedCrossRef

Reichler, R. J., & Schopler, E. (1971). Observations on the nature of human relatedness. Journal of Autism and Childhood Schizophrenia, 1, 283–296.PubMedCrossRef

Rodriguez, M. C., & Jung, T. (2003). The replicability of reliability generalization. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. Accessed May 27, 2011 from http://www.edmeasurement.net/aera/papers/rg%20replication%20aera.pdf.

Rodriguez, M. C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306–322.PubMedCrossRef

Rousson, V., Gasser, T., & Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine, 21, 3431–3446.PubMedCrossRef

Ryngala, D. J., Shields, A. L., & Caruso, J. C. (2005). Reliability generalization of the revised Children’s Manifest Anxiety Scale. Educational and Psychological Measurement, 65, 259–271.CrossRef

Sæmundsen, E., Magnússon, P., Smári, J., & Sigurdardóttir, S. (2003). Autism Diagnostic Interview-Revised and the Childhood Autism Rating Scale: Convergence and discrepancy in diagnosing autism. Journal of Autism and Developmental Disorders, 33, 319–328.PubMedCrossRef

Samtani, A., Sterling-Lewis, K., Scholten, R. J. P. M., Woolfenden, S., Hooft, L., & Williams, K. (2011). Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database for Systematic Reviews Collaboration,. doi:10.1002/14651858.cd009044.

Sawilowsky, S. S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase’s “reliability generalization” method and some Epm [sic] editorial policies. Educational and Psychological Measurement, 60, 157–173.CrossRef

Schlosser, R. W., Wendt, O., & Sigafoos, J. (2007). Not all systematic reviews are created equal: Considerations for appraisal. Evidence-Based Communication Assessment and Intervention, 1, 138–150.CrossRef

Schopler, E., Andrews, C. E., & Strupp, K. (1979a). Do autistic children come from upper-middle-class parents? Journal of Autism and Developmental Disorders, 9, 139–152.PubMedCrossRef

Schopler, E., & Reichler, R. J. (1971). Parents as cotherapists in the treatment of psychotic children. Journal of Autism and Childhood Schizophrenia, 1, 87–102.PubMedCrossRef

Schopler, E., & Reichler, R. J. (1972). How well do parents understand their own psychotic child? Journal of Autism and Childhood Schizophrenia, 2, 387–400.PubMedCrossRef

Schopler, E., Reichler, R. J., DeVellis, R. F., & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10, 91–103.PubMedCrossRef

Schopler, E., Reichler, R. J., & Renner, B. R. (1988). The Childhood Autism Rating Scale (CARS). Los Angeles, CA: Western Psychological Services.

Schopler, E., Rutter, M., & Chess, S. (1979b). Editorial: Change of journal scope and title. Journal of Autism and Developmental Disorders, 9, 1–10.PubMedCrossRef

Sevin, J. A., Matson, J. L., Coe, D. A., Fee, V. E., & Sevin, B. M. (1991). A comparison and evaluation of three commonly used autism scales. Journal of Autism and Developmental Disorders, 21, 417–432.PubMedCrossRef

Sloan, J. L. (1978). Differential development of autistic symptoms in a pair of fraternal twins. Journal of Autism and Childhood Schizophrenia, 8, 191–202.PubMedCrossRef

Sloan, J. L., & Marcus, L. (1981). Some findings on the use of the adaptive behavior scale with autistic children. Journal of Autism and Developmental Disorders, 11, 191–199.PubMedCrossRef

Stanley, G. C., & Konstantareas, M. M. (2007). Symbolic play in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 37, 1215–1223.PubMedCrossRef

StataCorp. (2012). Stata statistical software: Release 12.1 [computer software]. College Station, TX: StataCorp LP.

Sun, S. (2011). Meta-analysis of Cohen’s kappa. Health Services Outcomes Research Methodology, 11, 145–163.CrossRef

Teal, M. B., & Wiebe, M. J. (1986). A validity analysis of selected instruments used to assess autism. Journal of Autism and Developmental Disorders, 16, 485–494.PubMedCrossRef

Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174–195.

Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for Depression: A meta-analysis over a period of 49 years. Psychiatry Research, 189, 1–9.PubMedCrossRef

Travis, L., Sigman, M., & Ruskin, E. (2001). Links between social understanding and social behavior in verbally able children with autism. Journal of Autism and Developmental Disorders, 31, 119–130.PubMedCrossRef

Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509–522.CrossRef

Vacha-Haase, T., & Thompson, B. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies. Measurement and Evaluation in Counseling and Development, 44, 159–168.CrossRef

Van Bourgondien, M. E., Marcus, L. M., & Schopler, E. (1992). Comparison of DSM-III-R and Childhood Autism Rating Scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22, 493–506.PubMedCrossRef

Vassar, M., & Hale, W. (2007). Reliability reporting practices in youth life satisfaction research. Social Indicators Research, 83, 487–496.CrossRef

Vassar, M., Ridge, J. W., & Hill, A. D. (2008). Inducing score reliability from previous reports: An examination of life satisfaction studies. Social Indicators Research, 87, 27–45.CrossRef

Viding, E., & Blakemore, S. J. (2007). Endophenotype approach to developmental psychopathology: Implications for autism research. Behavioral Genetics, 37, 51–60.CrossRef

Wiggins, L. D., Robins, D. L., Adamson, L. B., Bakeman, R., & Henrich, C. C. (2012). Support for a dimensional view of autism spectrum disorders in toddlers. Journal of Autism and Developmental Disorders, 42, 191–200.PubMedCrossRef

Wilkinson, L., & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.

Wilson, R., Pascalis, O., & Blades, M. (2007). Familiar face recognition in children with autism: The differential use of inner and outer face parts. Journal of Autism and Developmental Disorders, 37, 314–320.PubMedCrossRef

Woolfenden, S., Sarkozy, V., Ridley, G., & Williams, K. (2012). A systematic review of the diagnostic stability of autism spectrum disorder. Research in Autism Spectrum Disorders, 6, 345–354.CrossRef

Yamashita, Y., Fujimoto, C., Nakajima, E., Isagai, T., & Matsuishi, T. (2003). Possible association between congenital cytomegalovirus infection and autistic disorder. Journal of Autism and Developmental Disorders, 33, 455–459.PubMedCrossRef

Zheng, Y. P., Zhao, J. P., Phillips, M., Liu, J. B., Cai, M. F., Sun, S. Q., et al. (1988). Validity and reliability of the Chinese Hamilton Depression Rating Scale. British Journal of Psychiatry, 152, 660–664.PubMedCrossRef

Titel: Reliability Generalization for Childhood Autism Rating Scale
Auteurs: Jonathan Breidbord
Tim J. Croudace
Publicatiedatum: 01-12-2013
Uitgeverij: Springer US
Gepubliceerd in: Journal of Autism and Developmental Disorders / Uitgave 12/2013
Print ISSN: 0162-3257
Elektronisch ISSN: 1573-3432
DOI: https://doi.org/10.1007/s10803-013-1832-9

Andere artikelen Uitgave 12/2013

Functional Neuroimaging of Social and Nonsocial Cognitive Control in Autism

Original Paper

Asperger’s Disorder will be Back

Original Paper

Implementation Challenges in Translating Pivotal Response Training into Community Settings

Brief Report

The Effects of a Multi-Component Higher-Functioning Autism Anti-Stigma Program on Adolescent Boys

Original Paper

Initial Efficacy of Project ImPACT: A Parent-Mediated Social Communication Intervention for Young Children with ASD

Original Paper

A Cost of Illness Study of Children with High-Functioning Autism Spectrum Disorders and Comorbid Anxiety Disorders as Compared to Clinically Anxious and Typically Developing Children

Original Paper