Elsevier

Research in Autism Spectrum Disorders

Volume 6, Issue 3, July–September 2012, Pages 1091-1098
Research in Autism Spectrum Disorders

Agreement in multi-informant assessment of behaviour and emotional problems and social functioning in adolescents with Autistic and Asperger's Disorder

https://doi.org/10.1016/j.rasd.2012.02.008Get rights and content

Abstract

There is a paucity of evidence concerning the patterns of multi-informant agreement in populations with Autism Spectrum Disorders (ASD). This study compared ratings of behaviour and emotional problems and social functioning provided by 45 adolescents aged 12–18 years, with Autistic or Asperger's Disorder with ratings by their parents and teachers. Behaviour and emotional problems were measured using the Child Behaviour Checklist and social functioning was measured with the Social Skills Rating System. Moderate agreement was found between adolescent and parent ratings of psychopathology and social functioning. In general, parent and teacher ratings and adolescent and teacher ratings differed significantly. These findings suggest that behaviour varies across environmental contexts. Patterns of agreement between multiple informants observed in the current study frequently differed from those previously found in non-ASD populations. Clinicians and researchers need to consider a multi-informant approach to obtain a detailed picture of behaviour and emotional problems in adolescents with autism and Asperger's Disorder.

Highlights

► Adolescents and parents generally had moderate levels of agreement on ratings of behaviour and emotional problems and social functioning. ► Little agreement was found between adolescent and teacher ratings of behaviour and emotional problems and social functioning, whereas agreement between parent and teacher ratings was variable. ► Patterns of agreement between multiple informants frequently varied to those previously observed in non-ASD populations.

Introduction

The ability of informants to provide a meaningful report of phenomena is limited by their capacity to observe individuals in specific settings (Fagan & Fantuzzo, 1999) and certain behaviours may be absent or evident with varying extent depending on the environment in which they occur (Renk & Phares, 2004). Different informants are also likely to colour their observations according to different biases, expectations and standards (Achenbach & Rescorla, 2001). Multi-informant assessment can capture the observations of numerous sources to provide a comprehensive and balanced assessment of an individual's behaviour, experiences and functioning (Macintosh & Dissanayake, 2006). Despite this advantage, few studies have examined the properties and characteristics of multi-informant assessment in populations with Autism Spectrum Disorders (ASD). This includes broad measures of behaviour and emotional problems and social functioning that are routinely used in research and clinical practice. Such information is valuable for those seeking to fully and accurately assess the wide variety of behavioural and emotional difficulties young people with ASD experience and the qualities of their social function (Matson and Nebel-Schwalm, 2007, Simonoff et al., 2008).

Consistent evidence of the discrepancy between self, parent and teacher report of behaviour and emotional problems, in children and adolescents from various populations, suggests that multi-informant assessment is preferable to obtaining information from a single informant (Bird et al., 1992, De Los Reyes and Kazdin, 2005). For example, children and adolescents from clinical, paediatric hospital and typically developing samples tend to report having more behaviour and emotional problems than other informants, whilst parents report more problems than teachers (Clay et al., 2008, Montano et al., 2011, Sawyer et al., 1992, van der Ende and Verhulst, 2005, van der Ende et al., 2011). Correlations between different informants are also generally low as reported by Achenbach, McConaughy and Howell's (1987) meta-analysis of 119 studies which found a mean correlation of 0.28 between informant ratings in young people. However, agreement varies depending on the rater dyads being compared. Ratings from informants who have witnessed behaviour in the same context (such as fathers and mothers) tend to produce moderate agreement whereas lower agreement is found between raters across contexts (Achenbach et al., 1987, Duhig et al., 2000, van der Ende et al., 2011). Low to moderate correlations between parent and teacher ratings have been found using a range of instruments in young people from general, clinical and intellectually disabled populations (Aman et al., 1996, Clay et al., 2008, Dekker et al., 2002, Freund and Reiss, 1991, Salbach-Andrae et al., 2009b, Stone et al., 2010). Weak to moderate correlations between self and parent and self and teachers ratings have been observed when assessing behaviour and emotional problems in adolescents with and without Axis I disorders (e.g. substance, mood, personality and conduct disorders) (Salbach-Andrae et al., 2009a, Salbach-Andrae et al., 2009b, van der Ende and Verhulst, 2005). Agreement between self and teacher ratings are generally lower than those between self and parents ratings (Salbach-Andrae et al., 2009b, van der Ende and Verhulst, 2005).

Preliminary evidence specific to the agreement of multi-informant ratings in young people with ASD is also available. Hurtig et al. (2009) assessed self, parent and teacher reported behaviour and emotional problems in 46 adolescents with Autistic or Asperger's Disorder, aged 11–17 years. Use of the Youth Self Report (YSR), Teacher Report Form (TRF) and parent-rated Child Behaviour Checklist (CBCL) allowed comparison of ratings across 11 broad and specific problem scales. Weak to moderate correlations were found between self and parent ratings with nine of the scales correlating significantly (r = .27–.51). Some agreement was also found between self and teacher ratings, and parent and teacher ratings, with significant correlations found on seven (r = .36–.66) and five scales (r = .39–.58), respectively. Agreement within both dyads tended to occur on scales concerning externalised behaviour problems. Self and parent agreement was also examined in typically developing controls where significant correlation was found on all but one scale. However, correlations were often weaker than self-parent correlations in the ASD sample. Kanne, Abbacchi, and Constantino (2009) compared levels of agreement between parent and teacher ratings of 177 children and adolescents aged 3–18 years with ASD to that found in their typically developing siblings (n = 148) who were aged 1–16 years. Significant, moderate to strong correlations (r = .37–.73) between parent and teachers ratings were observed on six of seven DSM-oriented subscales in the sibling sample. In contrast, only four subscales in the ASD sample correlated significantly, at a low to moderate level (r = .26–.49). Such findings suggest that there is more variability in inter-rater agreement in ASD compared to typically developing or intellectually disabled populations. Yet, Hurtig et al. findings suggest inter-rater agreement is roughly comparable and, in some instances, higher in young people with ASD compared to typically developing individuals. Agreement may vary according to the measures used and specific problems examined.

Multi-informant assessment of social functioning is also preferable given further evidence of low agreement between self, parent and teacher report (Kalyva, 2010). Renk and Phares (2004) conducted a meta-analysis of 74 studies that assessed multi-informant agreement of social competence in children and adolescents. Typically developing, clinical and physically and developmentally disabled samples were included. Low to moderate mean inter-rater correlations were found; self and parent ratings (r = .21) had the lowest correspondence, while parent–teacher agreement (r = .38) was significantly higher than that of self-teacher agreement (r = .25). Other examinations of children and adolescents from the general population have found weak to moderate correlations between self, parent and teacher ratings of specific areas of social skills (r = .02–.39) such as cooperation, empathy and engagement (Gresham, Cook, Vance, Elliot, & Kettler, 2010). Assessment of social functioning in young people with intellectual disability is limited to parents and teachers given the difficulty of children and adolescents with ID to provide self report (Bielecki & Swender, 2004). Low correlations have generally been found between parent and teacher ratings on scales of social competence and social relating problems (Aman et al., 1996, Dekker et al., 2002, Tasse and Lecavalier, 2000). Evidence of low agreement in these populations may reflect variability in social behaviour across contexts or perhaps the differing perspectives of informants (Renk & Phares, 2004).

Although social function impairments represent a core component of ASD diagnoses, there is a lack of evidence concerning multi-rated and cross-contextual social functioning in young people with ASDs (Murray, Ruble, Willis, & Molloy, 2009). Available data are inconsistent with inter-rater agreement appearing to vary according to the instruments and samples examined. Murray et al. (2009) used the TRIAD Social Skills Assessment to examine 45 children and adolescents aged 5–14 years with Autistic Disorder, Asperger's Disorder or Pervasive Developmental Disorder-Not Otherwise Specified, who were referred for social skills intervention. Correlations between parent and teacher ratings of positive social behaviour ranged from low to weak and statistically non-significant (r = .15–.38). Macintosh and Dissanayake (2006) found no significant correlations between parent and teacher rated total and subscale social skill scores (r = −.28 to .20) on the Social Skill Ratings System (SSRS). Their samples consisted of 20 children aged 4–10 years with Autistic Disorder and 17 children aged 5–10 years with Asperger's Disorder. The Assertion subscale in the Asperger's Disorder sample (r = .60) was an exception, demonstrating good inter-rater agreement between parents and teachers. In contrast, Kalyva (2010) reported moderate agreement between parent and teacher (r = .51–.59) ratings on the Matson Evaluation of Social Skills with Youngsters in 21 children and adolescents with Asperger's Disorder, aged 9–16 years.

Further inconsistency is evident when the self-ratings of adolescents with Autism Spectrum Disorders are examined. Kalyva (2010) reported strong agreement between self and teacher ratings (r = .79), whereas self-father agreement (r = .67) was moderate and self-mother agreement (r = .28) was not significant. However, Vickerstaff, Heriot, Wong, Lopes, and Dossetor (2007) found no significant correlation between the total SSRS scores of children, parents and teachers when examining 22 young people with Autistic Disorder, Asperger's Disorder and Atypical Autism, aged 7–13 years. Children rated their social functioning as better than did their parents and teachers, whereas parent and teacher ratings did not significantly differ. Similar results were found by Koning and Magill-Evans (2001) who compared self, parent and teacher SSRS report in 21 adolescent males aged 12–15 years with Asperger's Disorder. Adolescents also rated themselves as more socially skilled than did their parents or teachers and correlations between all informants were weak (r = −.01 to −.10). Thus, some evidence suggests poor agreement occurs in ratings of social functioning in young people with ASD, with a level of correspondence below that reported in non-ASD populations.

Together, these findings demonstrate that agreement between multi-informant ratings of behaviour and emotional problems and social functioning in young people from general community, intellectually disabled, clinical and ASD populations are often low. However, some instances of moderate agreement have also been recorded. Few studies have examined agreement between multiple informants in young people with ASD populations. Hurtig et al. (2009) is the only published study in which self, parent and teacher ratings on broad measures of behaviour and emotional problems have been examined in adolescents with ASD. Furthermore, only three studies have compared self, parent and teacher ratings of social functioning in young people simultaneously (Kalyva, 2010, Koning and Magill-Evans, 2001, Vickerstaff et al., 2007). However, the low mean age and age range of their sample limits the ability to generalise findings to older adolescents with ASD. In view of these limitations and the paucity of available evidence, this study aimed to examine ratings (by parents, teachers and adolescents) of behaviour and emotional problems and social skills in a sample of adolescents with ASD.

Section snippets

Participants

Participants were 45 adolescents who were diagnosed with either Autistic or Asperger's Disorder. Demographic details are presented in Table 1. Participants were recruited from schools, support services, and support groups from Melbourne and greater Victoria. Families who had participated in previous research at the Monash University Centre for Developmental Psychiatry and Psychology and had indicated that they were interested in being contacted about future research projects, were mailed

Results

The mean score and standard deviations of YSR, TRF and CBCL ratings are presented in Table 2 whilst correlations are reported in Table 3. Moderate to strong correlations were found between all broad and subscale YSR and CBCL scores. No significant correlations were found between YSR and TRF broad or subscales scores, with the exception of ratings on the Rule-breaking Behaviour scale. Low to moderate but significant correlation was found between the majority of CBCL and TRF scale scores, with

Discussion

The results from the current study suggest that self, parent and teacher ratings of behaviour and emotional problems and social functioning, in adolescents with Autistic and Asperger's Disorder, are frequently discrepant. However, different patterns of agreement are apparent between the various dyads. Adolescents and parents generally had moderate–strong levels of agreement when reporting upon behaviour and emotional problems and social functioning. Such agreement in the current study suggests

References (58)

  • E. Simonoff et al.

    Psychiatric disorders in children with Autism Spectrum Disorders: Prevalence, comorbidity, and associated factors in a population-derived sample

    Journal of the American Academy of Child & Adolescent Psychiatry

    (2008)
  • S. Van der Oord et al.

    A psychometric evaluation of the Social Skills Rating System in children with Attention Deficit Hyperactivity Disorder

    Behaviour Research and Therapy

    (2005)
  • T. Achenbach et al.

    Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity

    Psychological Bulletin

    (1987)
  • T. Achenbach et al.

    Manual for the ASEBA school-age forms and profiles

    (2001)
  • American Psychiatric Association

    Diagnostic and statistical manual of mental disorders

    (2000)
  • T. Barry et al.

    Examining the effectiveness of an outpatient clinic-based social skills group for high-functioning children with autism

    Journal of Autism and Developmental Disorders

    (2003)
  • S. Berument et al.

    Autism screening questionnaire: diagnostic validity

    British Journal of Psychiatry

    (1999)
  • J. Bielecki et al.

    The assessment of social functioning in individuals with mental retardation

    Behaviour Modification

    (2004)
  • H. Clay et al.

    Assessing emotional and behavioural problems in children: Factors associated with multiple informant consistency in New Zealand

    New Zealand Journal of Psychology

    (2008)
  • C. Corsello et al.

    Between a ROC and a hard place: Decision making and making decisions about using the SCQ

    Journal of Child Psychology and Psychiatry

    (2007)
  • A. De Los Reyes et al.

    Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study

    Psychological Bulletin

    (2005)
  • M. Dekker et al.

    Psychometric properties of the revised Developmental Behaviour Checklist scales in dutch children with intellectual disability

    Journal of Intellectual Disability Research

    (2002)
  • M. Demaray et al.

    Social skills assessment: A comparative evaluation of six published rating scales

    School Psychology Review

    (1995)
  • A. Duhig et al.

    Interparental agreement on internalizing, externalizing, and total behavior problems: A meta-analysis

    Clinical Psychology: Science and Practice

    (2000)
  • D. Flanagan et al.

    Contemporary intellectual assessment: Theories, tests and issues

    (2005)
  • F. Gresham et al.

    Cross-informant agreement for ratings for social skill and problem behavior ratings: An investigation of the Social Skills Improvement System-Rating Scales

    Psychological Assessment

    (2010)
  • F. Gresham et al.

    Social Skills Rating System, manual

    (1990)
  • T. Hurtig et al.

    Multi-informant reports of psychiatric symptoms among high-functioning adolescents with Asperger syndrome or autism

    Autism

    (2009)
  • M.Y. Ivanova et al.

    Testing the 8-syndrome structure of the Child Behavior Checklist in 30 societies

    Journal of Clinical Child and Adolescent Psychology

    (2007)
  • Cited by (0)

    View full text