Implications of nested designs in school-based mental health services research
Introduction
There has been a growing interest in both increasing the amount and types of school-based mental health services as well as testing these services to assure they obtain the same efficacy as in highly controlled settings. The report on mental health from the Surgeon General (US Department of Health and Human Services, 1999) suggests that locating mental health services in the schools, may be the best place to offer mental health services for children, and there is evidence to suggest that school-based services increases utilization (Catron & Weiss, 1994). Nationally, there are numerous attempts to increase the amount and types of mental health services in schools with state policy makers and school boards demanding more and better mental health services for all students (Adelman & Taylor, 2000).
This demand for increased services for children who have emotional disturbance (ED) is driven by two factors. First, this group of children has been and continues to be severely under-identified and under-served. It is estimated that less than one-third of the nation's children who are emotionally impaired to the point of needing mental health intervention receive any service (Friedman, Kutash, & Duchnowski, 1996). While precise epidemiological data are not available, there is a consensus in the field that approximately 20% of all children have a diagnosable ED at any point in time and that 5% of all children experience ED at a severe and persistent level (Friedman, Kutash, et al., 1996). Second, the number of children who are identified as needing mental health service grows each year. For example, between 1996 and 2001, the number of school age children in the United States remained relatively stable, while the number of children identified by the school system as emotional disturbed increased by approximately 5%/yr (US Department of Education (1996), US Department of Education (2001)).
Furthermore, there is a significant literature base documenting the poor outcomes of children identified as having an emotional disturbance. They have the lowest graduation rates, the lowest grades, and the highest incidence of contact with the juvenile justice system compared to peers who either have other types of disabilities or no disability (e.g., see Wagner, Kutash, Duchnowski, & Epstein, 2005). As they transition into adulthood they are at continued risk for poor psycho-social outcomes (Greenbaum et al., 1996; Wagner, 1995).
These conditions present a formidable challenge for the public school system. There is evidence from some community studies that approximately 70% of children who receive any mental health service at all receive it in their school (Burns et al., 1995; Leaf et al., 1996). Consequently, school administrators are eager to find effective interventions that are evidence based to improve the outcomes for these children and the quality of their lives. The research community is actively trying to meet this challenge through building the research base in school-based mental health services (Atkins et al., 2006; Weiss, Harris, Catron, & Han, 2003).
However, testing interventions in schools has historically been a challenge for educators (Cook, 2002). A frequently encountered challenge involves the use of individuals as the statistical unit of analysis rather than groups, therefore possibly violating the assumption of “independence of error.” This statistical effect of nesting is due to students being located in classrooms, those classrooms being located in schools, and those schools being located in districts. The logic is that the effect of being in one particular classroom or particular school may be as powerful or even more powerful than the intervention under investigation and may cause students’ scores in that classroom/school to be related, violating the independence assumption of typical statistical analyses. This has caused some researchers to suggest that random assignment should occur at the school level rather than the individual level. Consequently, the power of a study is greatly affected with more schools needing to be recruited than individual subjects per school—a costly implication (Moerbeek, van Breukelen, & Berger, 2000).
The purpose of the current investigation is to empirically document the effects of nested designs in studies exploring psychological and educational constructs with students with ED. The investigation examined the effects of nesting through an analysis of scores for 314 students with ED served in special education programs in 24 schools located in 9 states. Within the method section, the data sets used in the analyses are described along with brief descriptions of the instruments used to measure the constructs of interest. Also in the method section, the rationale for interpreting the results of the intraclass correlations is presented. The results section provides the amount of variance between and within factors and the intraclass correlations (i.e., the design effect) for the measures of psychopathology, impairment, and academic and school related functioning. The results are discussed in terms of implications for alternative research designs and statistical analyses.
Section snippets
Participants
The data on 314 students from 24 schools were supplied from two studies of school reform efforts in special education; the School and Community Study (Kutash et al., 2000) and the Urban School and Community Study (Kutash & Duchnowski, 2004). Both of these studies recruited public schools in which there were special education classrooms educating students with ED within general education buildings. There were a total of 24 schools recruited through the two studies, 10 and 14 schools,
Instruments
The three types of data collection techniques used in both of these investigations were (a) information collected from a review of school records and from school personnel, (b) direct administration of assessments to the student, and (c) an interview of each student's parent. The data collection procedures and the instrument employed are common to the children's mental health services research field.
Design and procedures
All schools in the study reported only the date of birth, gender, race, attendance, and cost of school meals for each of their students formally identified as having an emotional or behavioral disability and served in special education settings. Those students who met study age and attendance requirements constituted the pool of potential participants for the study. A key staff member in each school distributed a letter describing the study to the parents/caregivers of each of these potential
Results
The three distinct areas or constructs measured on the students included: (a) level of psychopathology, as reflected in the total score of the CBCL score as well as the scores for internalizing and externalizing problem behaviors; (b) level of impairment, measured by either the CIS or the CAFAS; and (c) academic and school related functioning as measured by IQ level, standardized measures of math and reading, and percent of time a student spends in the special education environment per week.
Discussion
As more and more researchers investigate the efficacy of school-based mental health services, the debate about determining the correct unit of analysis prevails as an important consideration in determining the design of intervention studies. As the debate continues, perhaps more discussions should center on the theory driving the intervention, how this theory affects the design of the experimental unit, the unit of analysis, as well as how theory addresses the effects of nesting or
Limitations
The implications of the results of this study are limited by several factors. The participants in these studies were essentially convenience samples. Neither the schools nor the students were randomly selected. The number of measures used in the studies was also limited. There are many potential measures in the field for both emotional/behavioral functioning and academic achievement. Only a few of these measures were used in the current study. In addition, a single, specific intervention, such
Lessons learned
In spite of these limitations, some useful information has been produced for the field. Children's mental health services researchers who plan to investigate interventions aimed at improving the emotional functioning of children in schools do need to consider the potential effects of nesting. If the variable of interest is limited to psycho-social symptoms as measured by the CBCL, the effects of nesting are limited and the sampling can include students in classrooms in different schools.
References (38)
Analyzing grouped data with hierarchical linear modeling
Children and Youth Services Review
(2005)- et al.
Mental health service use in the community and schools: Results from the four-community MECA study
Journal of the American Academy of Child and Adolescent Psychiatry
(1996) Assessment and taxonomy of child and adolescent psychopathology
(1985)Manual for the child behavior checklist/4–18 and 1991 profile
(1991)- et al.
Problems and competencies reported by parents of Australian and American children
Journal of Child Psychology and Psychiatry
(1990) - et al.
Shaping the future of mental health in schools
Psychology in the Schools
(2000) - et al.
School-based mental health services for children living in high poverty urban communities
Mental Health Services Research
(2006) - et al.
Global measures of impairment for epidemiologic and clinical use with children and adolescents
International Journal of Methods in Psychiatric Research
(1996) - et al.
The Columbia Impairment Scale (CIS): Pilot findings on a measure of global impairment for children and adolescents
International Journal of Methods in Psychiatric Research
(1993) - et al.
Data watch
Health Affairs
(1995)
The Vanderbilt school-based counseling program: An interagency, primary-care model of mental health services
Journal of Emotional and Behavioral Disorders
Sampling techniques
Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them
Educational Evaluation and Policy Analysis
Modern methods in the sampling of human populations
American Journal of Public Health
Prevalence of serious emotional disturbances in children and adolescents
The population of concern
National adolescent and child treatment study (NACTS): Outcomes for children with serious emotional and behavioral disturbance
Journal of Emotional and Behavioral Disorders
The child and adolescent functional assessment scale
CAFAS manual for training coordinators, clinical administrators, and data managers
Cited by (12)
When Teachers have Autonomy to create SEL Initiatives: Conceptualizations and Iterations
2023, International Journal of Emotional EducationEffects of the SPARK Teen Mentoring Program for High School Students
2022, Journal of Child and Family StudiesHuman Service Administrators’ Attitudes Toward Immigration: Shaped by Community and Organizational Context?
2016, Human Service Organizations Management, Leadership and GovernanceThe Relationship Between Family Education and Support Services and Parent and Child Outcomes Over Time
2013, Journal of Emotional and Behavioral Disorders