Abstracts

ORIGINAL ARTICLE

Validity of Autism Behavior Checklist (ABC): preliminary study

Validade do Inventário de Comportamentos Autísticos (ICA): estudo preliminar

Márcia Regina Fumagalli Marteleto; Márcia Regina Marcondes Pedromônico

Department of Speech Therapy, Federal University of São Paulo (UNIFESP), São Paulo (SP), Brazil (Psychologists)

^{Correspondence} Correspondence Márcia Regina Fumagalli Marteleto Disciplina de Distúrbios da Comunicação Humana, Departamento de Fonoaudiologia Rua Botucatu, 802 - Vila Clementino 04023-900 São Paulo, SP, Brazil Phone/Fax: (11) 5549-7500. E-mail: marcia.marteleto@terra.com.br

ABSTRACT

OBJECTIVE: To examine the concurrent and criterion validity of the Autism Behavior Checklist (ABC).

METHODS: Three groups, comprising 38 mothers of children previously diagnosed with autism (DSM IV-TR, 2002), 43 mothers of children with language disorders other than autism, and 52 mothers of children who had no linguistic or behavioral complaints, were interviewed. In order to minimize the effect of maternal level of education, the questionnaire was completed by the researcher. To determine the concurrent validation, ANOVA and discriminant analysis were used. The ROC curve was used to establish the cutoff score of the sample and to examine the criterion validity.

RESULTS: The mean total score was significantly higher in the group of mothers of autistic children than in the other groups. The ABC correctly identified 81.6% of the autistic children. The ROC curve cutoff score was 49, and the sensitivity was 92.1%, higher than the 57.89% found when a cutoff score of 68 was used. The specificity was 92.6%, similar to the 94.73% obtained with a cutoff score of 68.

CONCLUSIONS: The ABC shows promise as an instrument for identifying children with autistic disorders, both in clinical and educational contexts, especially when a cutoff score of 49 is used.

Keywords: Autistic disorder; Diagnostic and statistical manual of mental disorders; Diagnosis, differential; Validation studies [Publication type]; Questionnaires

RESUMO

OBJETIVO: Examinar a Validade Concorrente e a Validade de Critério do Inventário de Comportamentos Autísticos (ICA).

MÉTODOS: Foram entrevistadas, com a escala, mães de crianças com diagnóstico de transtorno autista, previamente estabelecido por especialistas na área. Para comparação, foram também entrevistadas mães de crianças com transtorno de linguagem e mães de escolares sem queixas de problemas de linguagem e comportamento social. Os três grupos foram assim constituídos: GTA: 38 mães de crianças com transtorno autista (DSM IV-TR, 2002), GTL: 43 mães de crianças com transtorno de linguagem (DSM IV-TR, 2002) e GET: 52 mães de crianças escolares típicas. O questionário foi preenchido sob forma de entrevista para minimizar os efeitos da escolaridade materna. ANOVA e análise discriminante foram usadas para examinar a Validade Concorrente. A curva ROC foi usada para estabelecer o ponto de corte da amostra e para examinar a Validade de Critério.

RESULTADOS: O Inventário de Comportamentos Autísticos identificou corretamente 81,6% das crianças com autismo, sendo o escore médio total do GTA significantemente

(p < 0,001) maior que os outros dois grupos de crianças. O Inventário de Comportamentos Autísticos mostrou baixa sensibilidade (57,89%) e alta especificidade (94,73%) quando se usou a nota de corte 68 pontos; diminuída a nota de corte para 49 pontos obtida pela curva ROC, a sensibilidade da escala aumentou (92,1%) e a especificidade se manteve alta (92,6%).

CONCLUSÕES: O Inventário de Comportamentos Autísticos é um instrumento promissor para identificar crianças com autismo, especialmente com ponto de corte 49, tanto na clínica como em contextos educacionais.

Descritores: Transtorno autístico; Manual diagnóstico e estatístico de transtornos mentais; Diagnóstico diferencial; Validade; Questionários

Introduction

Autism is a severe, chronic development disorder, involving marked retardation of aptitudes for social interaction, communication and play.¹

The detection of autism and other general developmental disorders in very young children is quite difficult since delayed development may not be identified until the child is given the opportunity to interact in social environments other than the family setting. In addition, at the most severe levels, the differential diagnosis between autism and mental retardation is more difficult, especially among children of preschool age.²

Early diagnosis is very important since the sooner the recommended orientation of procedures is carried out, the more likely it is that such children will develop social and communicative skills, and the less stereotyped their behavior will be.

Regarding psychometric scales and psychological tests, there are few instruments that have been validated for the evaluation of Brazilian children under 6 years of age, and still fewer that have been validated for the evaluation of Brazilian children with autism or similar disorders. Therefore, to further research and clinical treatment, it is necessary that protocols for the evaluation of these groups of children be systematized.

In 2003, the validity of the Child Behavior Checklist (CBCL)³ for identifying children with autism was examined.⁴ The authors used the CBCL to interview 38 mothers of children with autism (mean age, 7.4 years), 31 mothers of children with other psychiatric disorders (mean age, 7.8 years) and 34 mothers of typical students (mean age, 7.0 years). Using logistic regression models, those authors identified combinations of CBCL scales that distinguished the groups. The results suggested that the CBCL is a promising instrument for identifying children with autism. In 2004,⁵ another study involving use of the CBCL identified the most frequent behavior problems in autistic children, discussing differences between speaking and nonspeaking children. In general, speaking and nonspeaking autistic children both presented behavior problems and more complaints of thinking problems.

The second edition of the Autism Screening Instrument for Educational Planning study (ASIEP-2 study)^6-7 was initiated in 2001 and presented in 2002.⁸

The ASIEP-2 is a screening instrument used to evaluate the autistic profile of children suspected of having this pathology and to create educational plans for such children. It consists of five subscales: the Autism Behavior Checklist (ABC), the Sample of Vocal Behavior, Interaction Assessment, Educational Assessment and Learning Rate Prognosis. It was designed in the 1980s and revised in 1993 in most of the United States and Canada. The material was administered by a multiprofessional team (psychologists, speech therapists, teachers and pediatricians) in special education schools whose students presented autism, hearing impairment, mental retardation, visual impairment or other disorders.

The ABC consists of a list of atypical behaviors characteristic of the pathology and is designed for the triage of children suspected of having this disorder, contributing to the differential diagnosis and the referral of such children to educational intervention. Due to its easy application and low cost, it has been used by health professionals in various countries, in research and in clinical practice. Some studies have used the questionnaire in interviews with the parents of such children.^9-10

The psychometric properties of the ABC have been investigated and evaluated for some years. Although the scale presents little psychometric power, it has been considered useful in the screening of the children suspected of having autism.

The standardization of the ABC occurred in three stages. Initially, the authors⁶ selected the atypical behaviors most frequently presented by the autistic patients from other scales used in the diagnosis of autism. Second, they mailed the list of behaviors to experienced professionals in the area of autism, who returned it to the authors after modifying and excluding behaviors from the list. The modified questionnaires were sent to 3000 professionals in the area of special education, who were asked to apply the scale through the observation of the children, identifying them by gender, age, diagnosis, and whether they were living in the institution or with the family. A total of 1049 questionnaires were returned to the authors. Statistical analyses determined the weight (score: 1 to 4) of each behavior, according to the degree to which it correlated with the pathology, a score of 4 representing the strongest correlation. The final version of the questionnaire addresses 57 atypical behaviors related to five areas: 1) sensory stimuli sensorial; 2) relating; 3) body and object use; 4) language; and 5) social self-help. The behaviors of the five areas are distributed at random on a registry form on which the observer notes the current behavior of the child. The weights of the behaviors identified are totaled by area, and those are in turn totaled to obtain the overall score. The authors proposed that children presenting overall scores equal to or greater than 68 points be classified as autistic.^6-7 Scores between 54 and 67 points are considered to represent moderate probability of classification as autistic, and scores between 47 and 53 points are considered inconclusive. When the score is below 47 points, the child is not considered autistic.

Since the creation of the ASIEP-2, many studies using the ABC have been carried out in attempts to check its validity and reliability.^10-21

The capacity of ABC to correctly evaluate children suspected of having autism using the cutoff point of 68 has been questioned.^13-14,16-19 The results of such studies, however, vary significantly regarding the sample selection criteria and methodology.

The validity of the scale was also examined in a study carried out by Miranda-Linné and Melin, who compared the total score of the scale, by area and by item, between speaking and nonspeaking autistic individuals.²⁰ The sample was composed of the members of the National Society for Autistic Children in Sweden (NSACS). Initially, cover letters and two questionnaires were sent to 2052 NSACS members, who were asked to administer the questionnaires to parents who volunteered to participate in the study. The first questionnaire addressed the age of the individuals, gender, laterality, diagnosis, age at time of diagnosis, age at onset of symptoms, presentation form of symptoms, cognitive disorders, verbal communication and family history of neuropsychiatric pathologies. The second questionnaire was a modified version of the ABC and included the list of behavioral symptoms frequently presented by individuals with autism. A total of 1596 NSACS members returned the questionnaires, and 497 were fond to be correctly filled out. Based on data collected in these questionnaires, the authors classified the language level of the children and adolescents. Individuals who had never spoken, or who had once spoken but were currently mute, were classified as mute, and those who presented spontaneous utterances were classified as speaking. The children and adolescents were then divided into two groups: 155 nonspeaking individuals (66% boys and 34% girls), and 335 speaking individuals (70% boys and 30% girls). The authors found a total mean score of 59.90 for nonspeaking individuals and 54.86 for speaking individuals. There was no significant difference between the two groups in mean score. The mean scores obtained in the two groups were both lower than the 68-point cutoff proposed by Krug et al.^6-7 In their study, Miranda-Linné and Melin²⁰ confirmed the proposals of other authors to decrease the 68-point cutoff score, considering it too high to correctly identify children with autism, and suggested a cutoff score of 54.

In one study, children with autism and children with mental retardation were compared using the ABC together with two other ASIEP-2 subscales (Interaction Assessment and Educational Evaluation).¹³ The authors correctly classified 100% of the children with autistic disorder and 95% of the children with mental retardation.

In general, only approximately 50% of children with autism are identified when the cutoff point proposed by the authors of the ABC is used, whereas the discriminative capacity of the questionnaire increases significantly when the cutoff point is lowered.^14,16-19

Based on the questions arising from these studies, the present article proposes to examine the concurrent and criterion validity of the ABC.

Methods

1. Participants

This study involved mothers of children previously diagnosed with autism, from two institutions of the city of São Paulo that work exclusively with programs of behavioral and educational intervention for autism: the Friends of Autism Association and the Speech Disorders Outpatient Clinic of the Universidade Federal de São Paulo (UNIFESP). The children were diagnosed by a multiprofessional team specialized in interviewing parents, were submitted to clinical evaluation and were diagnosed with autism.¹ Since language problems constitute one of the criteria for the diagnosis of autism, mothers of children diagnosed with speech disorders¹ but without autism, all undergoing speech therapy at the UNIFESP Speech Disorders Outpatient Clinic, were also invited to participate in the study. With the objective of having a comparison group without speech problems or autism complaints, mothers of school children from an educational program in a school associated with UNIFESP were also invited to participate. This study involved 38 mothers of children with autistic disorder, designated the autistic disorder group (ADG), 43 mothers of children with language disorder, designated the language disorder group (LDG), and 52 mothers of children without autism or language problems, designated the typical schoolchild group (TSG).

2. Adaptation of the instrument

For the prevalidation of the ABC, the questionnaire was initially translated from English into Portuguese. A back-translation was then carried out by a professional proficient in the English language. Subsequently, the questionnaire was administered to 6 mothers, 1 of mother of a child diagnosed with autism and 5 mothers of typical students. This test was administered with the objective of determining whether the translation needed to be adapted; these protocols were not included in the validation study.

During the administration of the test, items 13, 16, 19, 20, 21, 22, 23, 25, 29, 37, 38, 48, 49, 51 and 52 required further explanation for the mothers to understand their meaning. In order to eliminate the need for such elaboration, those items were adapted to colloquial language, without changing the meaning of the question. After these changes in the questionnaire, we considered the ABC translated and adapted to Brazilian culture. We entitled it the Inventário de Comportamentos Autísticos (a direct translation of "Autism Behavior Checklist").

We present herein the registry form, adapted and translated into Brazilian Portuguese⁸ (Table 1).

3. Procedure

In order to minimize the effect of maternal level of education, the ABC was administered to all of the mothers as an interview conducted by the psychologist responsible for the study. The mothers answered yes or no regarding the presence of a given behavior. The partial scores were then calculated and totaled to obtain the overall score for each child.

4. Data analysis

The concurrent validity refers to making comparisons between individuals previously diagnosed with autism and those diagnosed with other pathologies, with the aim of demonstrating that the test distinguishes the behavior of an individual presenting the chosen behavior from that of other individuals in different groups.²² We used analysis of variance (ANOVA) to compare averages of independent samples and discriminant analysis to evaluate how well the score would be able to distinguish among the three study groups.

The criterion validity refers to the effectiveness of a test to predict the behavior of the individual in predetermined situations. In this type of validation, indices are calculated using standard mathematical formulas. Sensitivity is the capacity of the test to detect the disease, and specificity refers to the capacity of the test to detect the absence of the disease.²²

In the present study, the initial criterion for the test of positivity was that established by the authors of the instrument^6-7 (all children obtaining a score of 67/68 points were considered positive). We later built the ROC curve with the aim of finding the cutoff point for our sample.

This study was approved by the UNIFESP Ethics in Research Committee (process no. 316/01).

Results

1. Description of the sample

Within the study sample, the ADG consisted of 32 boys (84.21%) and 6 girls (15.79%), the LDG of 29 boys (67.44%) and 14 girls (32.56%) and the TSG of 25 boys (48.08%) and 27 girls (51.92%).

Mean age was 7 years and 5 months, with a standard deviation (SD) of 2.8 in the ADG, 6 years and 9 months (SD = 2.3) in the LDG, and 7 years and 7 months (SD = 1.8) in the TSG. There were no significant differences among the three groups in terms of age.

In the ADG, 81.58% of the children were in preschool, whereas 78.85% of the TSG children were in elementary school. Among the LDG children, 41.86% were in preschool, and 41.86% were in elementary school.

In the ADG and TSG, the mean maternal level of education was 9 years of schooling, compared with 6 years in the LDG. In the LDG, mean maternal level of education was significantly lower (p < 0.001) than that found for the other two groups.

A significant difference was found among the groups with regard to mean monthly income, and the ADG was identified as the group presenting the highest such income (R$ 2,251.59).

2. Concurrent validity

After it had been confirmed that the data followed a normal distribution, the mean and SD of the total score were calculated for each group of participants. The percentage of children correctly identified by discriminant analysis was also calculated for each group.

Table 2 presents the ANOVA means and SDs found for the total ABC scores, as well as the discriminant analysis showing the percentage of children correctly classified in each of the groups.

We observed that the total mean score for the ADG children was significantly higher than that for children in the other two groups (p < 0.001), as well as being higher than the 68-point cutoff proposed by the authors of the test.^6-7 Total mean score for the LDG children was significantly higher than that for the TSG children (p < 0.001) and lower than that for the ADG children.

In the ADG and TSG, 81.6% and 84.6% of the children, respectively, were correctly classified. However, only 44.2% of the children in the LDG were correctly classified.

3. Criterion validity

Figure 1 shows the ROC curve of the ABC. The area under the curve was found to be 0.937 (95% CI = 0.892-0.982), which indicates that children with autism can be satisfactorily identified using the total score. The suggested cutoff point was 49 points.

Table 3 shows the validity indices for the ABC, considering the cutoff points: 67/68 and 48/49.

At cutoff point 67/68, we observed that the instrument presented low sensitivity (57.89%) and high specificity (94.73%) in identifying children with autistic disorder in the population. At cutoff point 48/49, the sensitivity of the instrument increased (92.11%), and the specificity remained high (92.63%).

Discussion

This study showed the validity of the ABC in distinguishing children with autism from children with language disorders and from those without complaints.

This instrument correctly classified 81.6% of the children with autism, and the total mean score was 72.736 (19,548), similar to that proposed by the authors of the instrument.^6-7 These data suggest that the ABC is useful and should be included in the protocol for investigating children suspected of having autism. Other studies have found similar results. Two studies^13-21 correctly classified 100% and 85%, respectively, of the children evaluated. However, such results were not obtained in other studies,^14-17 in which only 50% of the children were correctly classified. Neither speaking nor nonspeaking autistic children were correctly classified at cutoff point 68.²⁰

When we tested the criterion validity of the instrument using the 67/68 cutoff point, we found that the sensitivity (57%) and specificity (94.73%) were the same as those obtained in the validation study developed in 1988.¹⁴ Using the 48/49 cutoff point indicated in our sample, the sensitivity of the ABC significantly increased to 92.11% and the specificity of the instrument remained high, meaning that the ABC is capable of identifying the children who do not have autistic disorder, even among children with other pathologies.

We observed that the instrument correctly classified 44% of the children in the LDG. This suggests that there is a need to precisely characterize the language phenotypes of children with autism.²³ In children with developmental problems, the younger the child, the more difficult the establishment of the differential diagnosis is since delayed language development is a common characteristic of many childhood mental disorders. We emphasize that the ABC should be used for its intended purpose (the screening of children suspected of having autism), together with instruments used to evaluate the other various mental aspects, such as language, cognition and social skills.

In this study, the power of the instrument in screening children presenting a high probability of having autism increased when a lower cutoff point was used. Previous studies^16-19 have called into question the cutoff point suggested by the authors of the instrument.

In the studies previously mentioned,^14,16-20 the great variation in the age of the participants may be suggested as an explanation for the different cutoff points obtained. From our point of view, the behavioral phenotype of the child with autism suffers the effect of the development process. As the child grows, there is a change in the course of the disease, and the child tends to present other behaviors characteristic of the pathology,²⁴ even when in special treatment programs. In our study, we tried to reduce the age bracket of the children in order to minimize the variability in the behavioral manifestation. The fact that we found the appropriate cutoff point to be lower than that suggested by the authors of the instrument led us to consider another important factor: the level of severity of autism in that child. For example, speaking is considered a sign of less severity of the pathology in the child with autism, even with changes of form and content characteristic of the verbal language of this group of children.¹ In the ABC, seven of the thirteen behaviors described in the area of language are those atypical of the profile: repetition of sounds, words and phrases, inappropriate use of pronouns and prosodic alterations; and these were the ones with the highest score (3 or 4). We understand that the speaking child with autism will receive more points, increasing the probability of being identified and benefiting from the interventions offered in specialized educational programs. However, the nonspeaking child with autism, with more nonverbal stereotypies, may be incorrectly classified as being at a lower risk for autism. Therefore, further studies are necessary. Such studies should examine the ABC in groups of children with different degrees of severity, and even within the variations of the profiles included in the autistic spectrum.

In conclusion, despite the limitations of our study (the small sample size, the variability in the ages of the subject, and the fact that the severity of the disorder was no determined in the study group), our results suggest that the ABC is a useful screening instrument in the identification of children with autism, in clinical and educational contexts, with higher probability when a cutoff point of 49 is used.

Acknowledgments

We thank the Associação Amigos do Autista (AMA-SP), the Ambulatório dos Distúrbios da Comunicação Humana da UNIFESP, the Escola Paulistinha de Educação, the Centro de Recreação da UNIFESP and Capes/CNPq. We also thank Prof. Dr. Jacy Perissinoto and Prof. Me. Ana Carina Tamanaha, partners in studies on autism, and Prof. Me. Ellen Osborn for the back-translation.

Financing: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Conflict of interests: None

Submitted: 18 August 2004

Accepted: 31 January 2005

Academic thesis conducted with financial support provided by the Conselho Nacional de Pesquisa e Desenvolvimento/Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CNPq/CAPES) and entitled: Validity and Reliability of the Autism Behavior Checklist (ABC): Preliminary study, March 2003, presented to the Pediatric Department of the Universidade Federal de São Paulo.

Publication Dates

History