Agreement among physical educators, teachers and parents on children's behaviors: A multitrait–multimethod design approach
Highlights
► Convergent validity of the MBC and agreement among raters on children's behavior was examined. ► A MTMM matrix was applied to data from students rated by Physical educators, Teachers and parents. ► Validity of the MBC scales was supported by significant correlations with TRF, CBCL and ADHD RS-IV. ► MBC can provide valid ratings on children's behavior in school settings.
Introduction
Detection efforts for students with emotional, behavioral and developmental disorders are particularly critical during the early educational years, when these children are most amenable to change in behavioral, social, and academic arenas and before experience negative outcomes within and beyond the school setting (Landrum et al., 2003, Lane, 2003, Volkmar et al., 2004).
Given the costs associated with these disorders to students themselves, their families, and society as a whole, it is not surprising that reducing the incidence of emotional, behavioral and developmental disorders through systematic screening and comprehensive intervention efforts is a growing area of interest in educational research (Kauffman and Landrum, 2009, Lane, 2007, Nelson et al., 2003).
Despite the usefulness of rating instruments for screening children's deviant behaviors, the relatively modest agreements among rating sources it is a problem concerning the validity of the information and the importance of context or setting effects on children's behavior. Information on children's behavior can be gathered by a number of informants who each have their own point of view. Parents can observe their child in a wide range of situations; nonetheless, information from the parents is not always reliable. The accuracy of parents as raters may vary depending on such factors as education, the amount of stress associated with the child's behaviors, and hidden agenda's that parents may have when rating a child (De Los Reyes & Kazdin, 2005). Some parents tends to follow a pattern of idealized expectations and cultural stereotypes, some may be very sensitive or may have a low threshold for certain behaviors and will exaggerate symptoms, whereas other parents may underreport deviant or troublesome child behaviors.
Modest agreement is the norm for different informants’ ratings of a given child's functioning (Achenbach et al., 1987, Achenbach, 1991a), raising concerns about the relative validity of any single source of information and creating measurement obstacles for both research and clinical endeavors. Mental health professionals have distinct opinions about the relative value of different informants for behavioral criteria. The frequency, base rate, and conspicuousness of behaviors may affect the degree of concordance among informants (Kolko & Kazdin, 1993). Considerable literature addresses issues of method effects in cross-informant studies, and there are many explanations for rater disagreement (e.g., Drabick et al., 2008, Gadow et al., 2004). In general, concordance has been found to be higher when informants have similar relationships with the children being rated than when raters represent different roles (Achenbach et al., 1987, Greenbaum et al., 1994). There is also reason to believe that children's presentation of behavior problems varies across different settings such as the home or school (Kazdin and Kagan, 1994, Kolko and Kazdin, 1993). Additionally, cross-informant agreement may vary considerably depending on item content: Teachers may be more sensitive to disruptive behavior and parents more to depression or anxiety (Abikoff, Courtney, Pelham, & Koplewicz, 1993). On the other hand, teachers are more likely to agree with other teachers about depressive symptoms or overall levels of depressive and aggressive behaviors (Epkins, 1995, McDermott, 1994), suggesting that there may be more cross-situational continuity in these behaviors. Teacher reports also may be more specific for these symptoms versus disruptive or attentional problems, in as much as teachers appear less likely to over report internalizing versus disruptive problem behaviors (Abikoff et al., 1993). Another view is that agreement should be greater about externalizing behaviors, whereas internalizing behaviors might be more difficult to observe and less disruptive to family or classroom functioning and therefore less likely to attract the attention of adult informants (Achenbach et al., 1987, Kolko and Kazdin, 1993).
Clinicians and researchers generally perceive children and youth self-report as the least useful source of behavior ratings pertaining to hyperactivity, inattention, and oppositional behaviors; whereas both youths and caregivers are preferred to teachers as sources of information about internalizing problems (Youngstrom, Loeber, & Stouthamer-Loeber, 2000).
Several studies suggest evidence for the presence of externalizing and/or internalizing symptoms can be obtained in multiple active situations, and a number of behavioral symptoms can be observed during physical education classes, team games and during standardised play procedures (Kashani et al., 1997, Mol Lous et al., 2002). However, there are only a few instruments that use physical educators as main source of information about children's development and the majority of them are focusing on movement and motor coordination problems like the Bruininks-Oseretsky Test of Motor Proficiency (BOT-2; Bruininks & Bruininks, 2005), the Test of Motor Development (TGMD; Ulrich, 2000), or the Movement Assessment Battery for Children (MABC-II; Henderson & Sugden, 2007), which assess gross and fine motor skills, problems in concentration, balance and levels of motor skill development as part of psychological test batteries, for making decisions about educational placement, developing and evaluating intervention programs.
The DSM-IV criteria for Disruptive behavior disorders (DBDs) include several items related to motor characteristics (American Psychiatric Association, 2000). During physical activities, children with ADHD exhibit age-inappropriate features of hyperactivity, excessive impulsivity, problems in lateralization, and are often left-handed (Corrigan, 2003, Reid and Norvilitis, 2000). Children with conduct problems at educational settings deviate from school and social principles, rules and regulations; display delinquent behavior, difficulties in social relationships, aggressiveness, combustible disobedience, anger, lack in empathy or concern for others, misperception of the intent of others in ambiguous social situations, lack in guilt or remorse, and low self-esteem (Dodge, 1993).
The importance of examining the interaction of peer relationships and other social relationships in the physical activity context, and the value of using the physical activity setting to promote quality peer relationships have been established (Smith, 2003). Motor related behaviors observed in children with developmental disorders include physical aggression, self-injury, property destruction, stereotyped behaviors, and tantrums which are highly disruptive to classroom, community, and home environments and without intervention, they are more likely to increase than improve (Horner, Carr, Strain, Todd, & Reed, 2002). During physical activities, children with ASD, indicate stereotyped and repetitive motor mannerisms, impairments of facial expression, postures, and gestures, and are often characterized as clumsy and as having problems in motor coordination (Berkeley et al., 2001, Piek and Dyck, 2004).
Physical education teachers spend a lot of time with the children and have the flexibility to work with them and observe their behaviors in several ways (e.g., structured lessons or free play situations) and in several different settings (inside or outside the classroom, at the playground or at the school-yard). However, there is a lack of literature concerning the investigation of the added value of the information provided by physical educators on children's emotional, behavioral and social problems in school settings.
The study aimed to investigate the convergent validity of the Motor Behavior Checklist (MBC; Efstratopoulou et al., 2012) using data from a typical elementary school-aged sample. A multitrait–multimethod (MTMM) research design (Campbell & Fiske, 1959) was used to examine the validity of specific problem scales of the MBC. Correlations among scores on related problem scales were examined. Using data from three different sources of information, (i.e., physical educators, teachers and parents), and ratings on four behavioral assessment instruments (MBC, Efstratopoulou et al., 2012; TRF, Achenbach, 1991b; CBCL, Achenbach, 1991a; ADHD Rating Scale-IV, DuPaul et al., 1998) the agreement among different raters on specific problem scales was also investigated. The following hypotheses were tested:
(i) There are significant correlations among corresponding scales of the instruments used. More specifically, convergent relationships are expected for the Rules Breaking problem scale of MBC and the Delinquent Behavior scales of TRF and CBCL. In addition, convergent relationships are also expected on Attention problem scales, Social Problem scales, Externalizing, Internalizing, and Total scores.
(ii) The multitrait correlations, discussed above, are significantly higher when measured by raters in school settings (e.g. teachers; physical educators) than when measured across settings (e.g. teachers; parents).
Section snippets
Participants
The data derived from 35 typical Greek elementary schools widely spread across the country selected so that the sample distribution would be representative of the urban and rural population. The schools were located in urban areas (63.3%) and in rural areas and islands (36, 7%). The data analysed were collected from a randomly selected sample (N = 841) of elementary school-aged children. The overall sample consisted of 421 (50, 1%) girls and 420 (49, 9%) boys, ranging from 6 to 11 years (M = 8.4
Results
The convergent validity of the Motor Behavior Checklist problem scales (MBC, Efstratopoulou et al., 2012) was examined through the correlation coefficients along the validity diagonals. A multitrait–multimethod-correlation matrix (Campbell & Fiske, 1959) of all the inter-correlations between the six subscales and the three measurements (physical educators, teachers and parents) was generated, and the results are in Table 2. Convergent validities are the bold values.
Examination of the validity
Discussion
The purpose of this study was to investigate the convergent validity of the Motor Behavior Checklist (MBC; Efstratopoulou et al., 2012) using data from a typical elementary school-aged sample. The MTMM model (Campbell & Fiske, 1959) was used in order to examine the agreement among raters on children's problematic behavior. Based on the MTMM design we created tables of correlations to examine the validity diagonals between similar contracts (subscales) rated by different raters and we tested
References (55)
- et al.
Differentiating children with Attention-Deficit/Hyperactivity Disorder, Conduct Disorder, Learning Disabilities and Autistic Spectrum Disorders by means of their motor behavior characteristics
Research in Developmental Disabilities
(2012) - et al.
Dysthymic disorder in clinically referred preschool children
Journal of the American Academy of Child and Adolescents Psychiatry
(1997) - et al.
Symptom profiles in children with ADHD: Effects of comorbidity and gender
Journal of the American Academy of Child and Adolescent Psychiatry
(2001) - et al.
Sensory-motor deficits in children with developmental coordination disorder, attention deficit hyperactivity disorder and autistic disorder
Human Movement Sciences
(2004) - et al.
Evidence for anomalous lateralization across domains in ADHD children as well as adults identified with the Wender Utah rating scale
Journal of Psychiatric Research
(2000) Peer relationships in physical activity contexts: A road less traveled in youth sport and exercise psychology research
Journal of Sport & Exercise Psychology
(2003)- et al.
Teachers’ ratings of disruptive behaviors: The influence of halo effects
Journal of Abnormal Child Psychology
(1993) Manual for the Child Behavior Checklist/4–18 and 1991 Profile
(1991)Manual for the Teacher's Report Form and 1991 profile
(1991)- et al.
Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity
Psychological Bulletin
(1987)
Diagnostic and statistical manual of mental disorders
The two faces of adolescents’ success with peers: Adolescent popularity, social adaptation, and deviant behavior
Child Development
Benefits and costs of prevention and early intervention programs for youth
Locomotor and object control skills of children diagnosed with autism
Adapted Physical Activity Quarterly
Bruininks–Oseretsky test of motor proficiency – 2nd edition manual
Convergent and discriminant validation by the matrix
Psychological Bulletin
Diagnostic accuracy of the Children Behavior Checklist scalesfor attention-deficit hyperactivity disorder: A receiver-operating characteristic analysis
Journal of Consulting and Clinical Psychology
Attention deficit hyperactivity disorder in sport: A review
International Journal of Sports Medicine
Social-cognitive mechanisms in the development of conduct disorder and depression
Annual Review of Psychology
Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study
Psychological Bulletin
Co-occurring ODD and GAD symptom groups: Source-specific syndromes and cross-informant comorbidity
Journal of Clinical Child and Adolescent Psychology
Methodological issues in using the child behavior checklist and its related instruments in clinical child psychology research
Journal of Clinical Child Psychology. Special Issue: Methodological Issues in Clinical Child Psychology Research
ADHD rating scale, norms, and clinical interpretation
Teacher Self-Efficacy, Burnout, Experience and Decision to Refer a Disruptive Student
School Psychology International
A multitrait–multimethod model with minimal assumptions
Psychometrika
Structural equation models for multitrait–multimethod data
Cited by (16)
Assessing children at risk in the United Arab Emirates: Validation of the Arabic version of the Motor Behaviour Checklist (MBC) for use in primary school settings
2023, Research in Developmental DisabilitiesParent-teacher agreement on children's externalizing behaviors: Results from a community sample of Portuguese elementary-school children
2020, Children and Youth Services ReviewCitation Excerpt :Nevertheless, one should also consider that the evaluation of an entire class might have a negative influence on the evaluation process given that, due to the bigger workload of teachers, and the number of children they had to evaluate, their ratings may have become more undifferentiated and globalizing and, therefore, a less true reflection of the children’s behavior. Some studies demonstrated the importance of analyzing these factors in order to understand cross-informant disagreements (e.g. Efstratopoulou et al., 2012; Laidra et al., 2006; Winsler & Wallace, 2002). Finally, our results suggested that, even when we extend the comparison between parents and teachers to a more specific level, comparing their ratings of children scoring higher on behavioral problems and children presenting lower levels of problem behaviors, discrepancies were also verified, with higher agreement between parent and teacher reports regarding children perceived as less problematic.
Methods matter: A multi-trait multi-method analysis of student behavior
2018, Journal of School PsychologyCitation Excerpt :Despite longstanding calls for increased attention to reliability and validity evidence for behavioral assessment methods (e.g., Cone, 1977; Silva, 1993), surprisingly few MTMM studies have been conducted in the area of behavioral assessment in educational settings. Indeed, our search of such studies yielded only three published manuscripts (Efstratopoulou et al., 2012; Roberts et al., 1981; Spilt et al., 2011), none of which utilized multi-level modeling to account for the nested structure of the data. Consequently, this investigation represents a substantial contribution to the research literature regarding the amount of trait-level variance actually captured by various behavior assessment methods.
Investigating the impact of physical activity in enhancing the well-being of children affected by emotional, behavioral, and developmental disorders
2023, Combining Exercise and Psychotherapy to Treat Mental HealthInclusive Strategies and Screening of Autism Spectrum Disorder (ASD) in the Gulf Cooperation Council (GCC) Countries: A Regional Overview
2022, Rethinking Inclusion and Transformation in Special Education