Implications of nested designs in school-based mental health services research

Abstract

Evaluating school-based mental health services for children and youth with emotional disturbance (ED) has been a challenge for researchers. One particular challenge is the study design of using the student as the statistical unit of analysis, which in certain cases may lead to a violation of the “independence of error” assumption. However, the alternative to this nested design, including fewer students and more schools, can be costly and administratively complex. This study examines data from two national studies including 314 students with ED and served in special education programs and their caregivers from 24 schools in the US to identify the extent to which nesting or design effects occur in this population. The results show that variables focusing on psychopathology are less affected by nesting but school-related variables such as academic functioning are more affected. Design effects varied by grade level, suggesting that grade should be considered when designing such evaluations.

Introduction

There has been a growing interest in both increasing the amount and types of school-based mental health services as well as testing these services to assure they obtain the same efficacy as in highly controlled settings. The report on mental health from the Surgeon General (US Department of Health and Human Services, 1999) suggests that locating mental health services in the schools, may be the best place to offer mental health services for children, and there is evidence to suggest that school-based services increases utilization (Catron & Weiss, 1994). Nationally, there are numerous attempts to increase the amount and types of mental health services in schools with state policy makers and school boards demanding more and better mental health services for all students (Adelman & Taylor, 2000).

This demand for increased services for children who have emotional disturbance (ED) is driven by two factors. First, this group of children has been and continues to be severely under-identified and under-served. It is estimated that less than one-third of the nation's children who are emotionally impaired to the point of needing mental health intervention receive any service (Friedman, Kutash, & Duchnowski, 1996). While precise epidemiological data are not available, there is a consensus in the field that approximately 20% of all children have a diagnosable ED at any point in time and that 5% of all children experience ED at a severe and persistent level (Friedman, Kutash, et al., 1996). Second, the number of children who are identified as needing mental health service grows each year. For example, between 1996 and 2001, the number of school age children in the United States remained relatively stable, while the number of children identified by the school system as emotional disturbed increased by approximately 5%/yr (US Department of Education (1996), US Department of Education (2001)).

Furthermore, there is a significant literature base documenting the poor outcomes of children identified as having an emotional disturbance. They have the lowest graduation rates, the lowest grades, and the highest incidence of contact with the juvenile justice system compared to peers who either have other types of disabilities or no disability (e.g., see Wagner, Kutash, Duchnowski, & Epstein, 2005). As they transition into adulthood they are at continued risk for poor psycho-social outcomes (Greenbaum et al., 1996; Wagner, 1995).

These conditions present a formidable challenge for the public school system. There is evidence from some community studies that approximately 70% of children who receive any mental health service at all receive it in their school (Burns et al., 1995; Leaf et al., 1996). Consequently, school administrators are eager to find effective interventions that are evidence based to improve the outcomes for these children and the quality of their lives. The research community is actively trying to meet this challenge through building the research base in school-based mental health services (Atkins et al., 2006; Weiss, Harris, Catron, & Han, 2003).

However, testing interventions in schools has historically been a challenge for educators (Cook, 2002). A frequently encountered challenge involves the use of individuals as the statistical unit of analysis rather than groups, therefore possibly violating the assumption of “independence of error.” This statistical effect of nesting is due to students being located in classrooms, those classrooms being located in schools, and those schools being located in districts. The logic is that the effect of being in one particular classroom or particular school may be as powerful or even more powerful than the intervention under investigation and may cause students’ scores in that classroom/school to be related, violating the independence assumption of typical statistical analyses. This has caused some researchers to suggest that random assignment should occur at the school level rather than the individual level. Consequently, the power of a study is greatly affected with more schools needing to be recruited than individual subjects per school—a costly implication (Moerbeek, van Breukelen, & Berger, 2000).

The purpose of the current investigation is to empirically document the effects of nested designs in studies exploring psychological and educational constructs with students with ED. The investigation examined the effects of nesting through an analysis of scores for 314 students with ED served in special education programs in 24 schools located in 9 states. Within the method section, the data sets used in the analyses are described along with brief descriptions of the instruments used to measure the constructs of interest. Also in the method section, the rationale for interpreting the results of the intraclass correlations is presented. The results section provides the amount of variance between and within factors and the intraclass correlations (i.e., the design effect) for the measures of psychopathology, impairment, and academic and school related functioning. The results are discussed in terms of implications for alternative research designs and statistical analyses.