Elsevier

Research in Developmental Disabilities

Volume 33, Issue 5, September–October 2012, Pages 1343-1351
Research in Developmental Disabilities

Agreement among physical educators, teachers and parents on children's behaviors: A multitrait–multimethod design approach

https://doi.org/10.1016/j.ridd.2012.03.015Get rights and content

Abstract

The study examines the agreement among raters on children's problematic behaviors. A multitrait–multimethod (MTMM) matrix was applied to a normative sample of elementary school-aged children (N = 841). The participants were rated by their physical educators, using the Motor Behavior Checklist for children (MBC; Efstratopoulou, Janssen, & Simons, 2012). Teachers and parents rated the same students using the Teacher Report Form (TRF; Achenbach, 1991b), the Child Behavior Checklist (CBCL; Achenbach, 1991a) and the ADHD Rating Scale-IV (DuPaul, Power, Anastopoulos, & Reid, 1998). The resulting matrix revealed significant correlations for the Rules Breaking, Lack of Attention, Hyperactivity/Impulsivity, Lack of Social interaction problem scale and for the Internalizing, Externalizing and Total scores. Convergent validity of the specific MBC subscales was supported by significant correlations with the corresponding subscales of TRF, CBCL and ADHD Rating Scale-IV. Findings underscore the importance of taking child's settings and observer influences into account and suggest that MBC is a new promising instrument that can provide valid ratings on externalizing behavior and social problems in children when used by physical educators in school settings.

Highlights

Convergent validity of the MBC and agreement among raters on children's behavior was examined. ► A MTMM matrix was applied to data from students rated by Physical educators, Teachers and parents. ► Validity of the MBC scales was supported by significant correlations with TRF, CBCL and ADHD RS-IV. ► MBC can provide valid ratings on children's behavior in school settings.

Introduction

Detection efforts for students with emotional, behavioral and developmental disorders are particularly critical during the early educational years, when these children are most amenable to change in behavioral, social, and academic arenas and before experience negative outcomes within and beyond the school setting (Landrum et al., 2003, Lane, 2003, Volkmar et al., 2004).

Given the costs associated with these disorders to students themselves, their families, and society as a whole, it is not surprising that reducing the incidence of emotional, behavioral and developmental disorders through systematic screening and comprehensive intervention efforts is a growing area of interest in educational research (Kauffman and Landrum, 2009, Lane, 2007, Nelson et al., 2003).

Despite the usefulness of rating instruments for screening children's deviant behaviors, the relatively modest agreements among rating sources it is a problem concerning the validity of the information and the importance of context or setting effects on children's behavior. Information on children's behavior can be gathered by a number of informants who each have their own point of view. Parents can observe their child in a wide range of situations; nonetheless, information from the parents is not always reliable. The accuracy of parents as raters may vary depending on such factors as education, the amount of stress associated with the child's behaviors, and hidden agenda's that parents may have when rating a child (De Los Reyes & Kazdin, 2005). Some parents tends to follow a pattern of idealized expectations and cultural stereotypes, some may be very sensitive or may have a low threshold for certain behaviors and will exaggerate symptoms, whereas other parents may underreport deviant or troublesome child behaviors.

Modest agreement is the norm for different informants’ ratings of a given child's functioning (Achenbach et al., 1987, Achenbach, 1991a), raising concerns about the relative validity of any single source of information and creating measurement obstacles for both research and clinical endeavors. Mental health professionals have distinct opinions about the relative value of different informants for behavioral criteria. The frequency, base rate, and conspicuousness of behaviors may affect the degree of concordance among informants (Kolko & Kazdin, 1993). Considerable literature addresses issues of method effects in cross-informant studies, and there are many explanations for rater disagreement (e.g., Drabick et al., 2008, Gadow et al., 2004). In general, concordance has been found to be higher when informants have similar relationships with the children being rated than when raters represent different roles (Achenbach et al., 1987, Greenbaum et al., 1994). There is also reason to believe that children's presentation of behavior problems varies across different settings such as the home or school (Kazdin and Kagan, 1994, Kolko and Kazdin, 1993). Additionally, cross-informant agreement may vary considerably depending on item content: Teachers may be more sensitive to disruptive behavior and parents more to depression or anxiety (Abikoff, Courtney, Pelham, & Koplewicz, 1993). On the other hand, teachers are more likely to agree with other teachers about depressive symptoms or overall levels of depressive and aggressive behaviors (Epkins, 1995, McDermott, 1994), suggesting that there may be more cross-situational continuity in these behaviors. Teacher reports also may be more specific for these symptoms versus disruptive or attentional problems, in as much as teachers appear less likely to over report internalizing versus disruptive problem behaviors (Abikoff et al., 1993). Another view is that agreement should be greater about externalizing behaviors, whereas internalizing behaviors might be more difficult to observe and less disruptive to family or classroom functioning and therefore less likely to attract the attention of adult informants (Achenbach et al., 1987, Kolko and Kazdin, 1993).

Clinicians and researchers generally perceive children and youth self-report as the least useful source of behavior ratings pertaining to hyperactivity, inattention, and oppositional behaviors; whereas both youths and caregivers are preferred to teachers as sources of information about internalizing problems (Youngstrom, Loeber, & Stouthamer-Loeber, 2000).

Several studies suggest evidence for the presence of externalizing and/or internalizing symptoms can be obtained in multiple active situations, and a number of behavioral symptoms can be observed during physical education classes, team games and during standardised play procedures (Kashani et al., 1997, Mol Lous et al., 2002). However, there are only a few instruments that use physical educators as main source of information about children's development and the majority of them are focusing on movement and motor coordination problems like the Bruininks-Oseretsky Test of Motor Proficiency (BOT-2; Bruininks & Bruininks, 2005), the Test of Motor Development (TGMD; Ulrich, 2000), or the Movement Assessment Battery for Children (MABC-II; Henderson & Sugden, 2007), which assess gross and fine motor skills, problems in concentration, balance and levels of motor skill development as part of psychological test batteries, for making decisions about educational placement, developing and evaluating intervention programs.

The DSM-IV criteria for Disruptive behavior disorders (DBDs) include several items related to motor characteristics (American Psychiatric Association, 2000). During physical activities, children with ADHD exhibit age-inappropriate features of hyperactivity, excessive impulsivity, problems in lateralization, and are often left-handed (Corrigan, 2003, Reid and Norvilitis, 2000). Children with conduct problems at educational settings deviate from school and social principles, rules and regulations; display delinquent behavior, difficulties in social relationships, aggressiveness, combustible disobedience, anger, lack in empathy or concern for others, misperception of the intent of others in ambiguous social situations, lack in guilt or remorse, and low self-esteem (Dodge, 1993).

The importance of examining the interaction of peer relationships and other social relationships in the physical activity context, and the value of using the physical activity setting to promote quality peer relationships have been established (Smith, 2003). Motor related behaviors observed in children with developmental disorders include physical aggression, self-injury, property destruction, stereotyped behaviors, and tantrums which are highly disruptive to classroom, community, and home environments and without intervention, they are more likely to increase than improve (Horner, Carr, Strain, Todd, & Reed, 2002). During physical activities, children with ASD, indicate stereotyped and repetitive motor mannerisms, impairments of facial expression, postures, and gestures, and are often characterized as clumsy and as having problems in motor coordination (Berkeley et al., 2001, Piek and Dyck, 2004).

Physical education teachers spend a lot of time with the children and have the flexibility to work with them and observe their behaviors in several ways (e.g., structured lessons or free play situations) and in several different settings (inside or outside the classroom, at the playground or at the school-yard). However, there is a lack of literature concerning the investigation of the added value of the information provided by physical educators on children's emotional, behavioral and social problems in school settings.

The study aimed to investigate the convergent validity of the Motor Behavior Checklist (MBC; Efstratopoulou et al., 2012) using data from a typical elementary school-aged sample. A multitrait–multimethod (MTMM) research design (Campbell & Fiske, 1959) was used to examine the validity of specific problem scales of the MBC. Correlations among scores on related problem scales were examined. Using data from three different sources of information, (i.e., physical educators, teachers and parents), and ratings on four behavioral assessment instruments (MBC, Efstratopoulou et al., 2012; TRF, Achenbach, 1991b; CBCL, Achenbach, 1991a; ADHD Rating Scale-IV, DuPaul et al., 1998) the agreement among different raters on specific problem scales was also investigated. The following hypotheses were tested:

(i) There are significant correlations among corresponding scales of the instruments used. More specifically, convergent relationships are expected for the Rules Breaking problem scale of MBC and the Delinquent Behavior scales of TRF and CBCL. In addition, convergent relationships are also expected on Attention problem scales, Social Problem scales, Externalizing, Internalizing, and Total scores.

(ii) The multitrait correlations, discussed above, are significantly higher when measured by raters in school settings (e.g. teachers; physical educators) than when measured across settings (e.g. teachers; parents).

Section snippets

Participants

The data derived from 35 typical Greek elementary schools widely spread across the country selected so that the sample distribution would be representative of the urban and rural population. The schools were located in urban areas (63.3%) and in rural areas and islands (36, 7%). The data analysed were collected from a randomly selected sample (N = 841) of elementary school-aged children. The overall sample consisted of 421 (50, 1%) girls and 420 (49, 9%) boys, ranging from 6 to 11 years (M = 8.4

Results

The convergent validity of the Motor Behavior Checklist problem scales (MBC, Efstratopoulou et al., 2012) was examined through the correlation coefficients along the validity diagonals. A multitrait–multimethod-correlation matrix (Campbell & Fiske, 1959) of all the inter-correlations between the six subscales and the three measurements (physical educators, teachers and parents) was generated, and the results are in Table 2. Convergent validities are the bold values.

Examination of the validity

Discussion

The purpose of this study was to investigate the convergent validity of the Motor Behavior Checklist (MBC; Efstratopoulou et al., 2012) using data from a typical elementary school-aged sample. The MTMM model (Campbell & Fiske, 1959) was used in order to examine the agreement among raters on children's problematic behavior. Based on the MTMM design we created tables of correlations to examine the validity diagonals between similar contracts (subscales) rated by different raters and we tested

References (55)

  • American Psychiatric Association

    Diagnostic and statistical manual of mental disorders

    (2000)
  • J.P. Allen et al.

    The two faces of adolescents’ success with peers: Adolescent popularity, social adaptation, and deviant behavior

    Child Development

    (2005)
  • S. Aos et al.

    Benefits and costs of prevention and early intervention programs for youth

    (2004)
  • S.L. Berkeley et al.

    Locomotor and object control skills of children diagnosed with autism

    Adapted Physical Activity Quarterly

    (2001)
  • R. Bruininks et al.

    Bruininks–Oseretsky test of motor proficiency – 2nd edition manual

    (2005)
  • D.T. Campbell et al.

    Convergent and discriminant validation by the matrix

    Psychological Bulletin

    (1959)
  • W. Chen et al.

    Diagnostic accuracy of the Children Behavior Checklist scalesfor attention-deficit hyperactivity disorder: A receiver-operating characteristic analysis

    Journal of Consulting and Clinical Psychology

    (1994)
  • B. Corrigan

    Attention deficit hyperactivity disorder in sport: A review

    International Journal of Sports Medicine

    (2003)
  • K. Dodge

    Social-cognitive mechanisms in the development of conduct disorder and depression

    Annual Review of Psychology

    (1993)
  • A. De Los Reyes et al.

    Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study

    Psychological Bulletin

    (2005)
  • D.A.G. Drabick et al.

    Co-occurring ODD and GAD symptom groups: Source-specific syndromes and cross-informant comorbidity

    Journal of Clinical Child and Adolescent Psychology

    (2008)
  • D. Drotar et al.

    Methodological issues in using the child behavior checklist and its related instruments in clinical child psychology research

    Journal of Clinical Child Psychology. Special Issue: Methodological Issues in Clinical Child Psychology Research

    (1995)
  • G.J. DuPaul et al.

    ADHD rating scale, norms, and clinical interpretation

    (1998)
  • C.J. Egye et al.

    Teacher Self-Efficacy, Burnout, Experience and Decision to Refer a Disruptive Student

    School Psychology International

    (2006)
  • M. Eid

    A multitrait–multimethod model with minimal assumptions

    Psychometrika

    (2000)
  • M. Eid et al.

    Structural equation models for multitrait–multimethod data

  • Cited by (16)

    • Parent-teacher agreement on children's externalizing behaviors: Results from a community sample of Portuguese elementary-school children

      2020, Children and Youth Services Review
      Citation Excerpt :

      Nevertheless, one should also consider that the evaluation of an entire class might have a negative influence on the evaluation process given that, due to the bigger workload of teachers, and the number of children they had to evaluate, their ratings may have become more undifferentiated and globalizing and, therefore, a less true reflection of the children’s behavior. Some studies demonstrated the importance of analyzing these factors in order to understand cross-informant disagreements (e.g. Efstratopoulou et al., 2012; Laidra et al., 2006; Winsler & Wallace, 2002). Finally, our results suggested that, even when we extend the comparison between parents and teachers to a more specific level, comparing their ratings of children scoring higher on behavioral problems and children presenting lower levels of problem behaviors, discrepancies were also verified, with higher agreement between parent and teacher reports regarding children perceived as less problematic.

    • Methods matter: A multi-trait multi-method analysis of student behavior

      2018, Journal of School Psychology
      Citation Excerpt :

      Despite longstanding calls for increased attention to reliability and validity evidence for behavioral assessment methods (e.g., Cone, 1977; Silva, 1993), surprisingly few MTMM studies have been conducted in the area of behavioral assessment in educational settings. Indeed, our search of such studies yielded only three published manuscripts (Efstratopoulou et al., 2012; Roberts et al., 1981; Spilt et al., 2011), none of which utilized multi-level modeling to account for the nested structure of the data. Consequently, this investigation represents a substantial contribution to the research literature regarding the amount of trait-level variance actually captured by various behavior assessment methods.

    View all citing articles on Scopus
    View full text