Abstract
Schools have become the largest provider of mental health services to children, thus increasing the need to identify children at risk before problems worsen. The Social, Academic, and Emotional Behavioral Risk Screener-Student Rating Scale (SAEBRS-SRS) student version is a universal screener that assesses pre-symptomology of internalizing and externalizing behaviors. Prior research on the SAEBRS-SRS has supported a psychometrically defensible factor structure; however, additional research is necessary to support applied used in schools. In this study, 1102 middle grades students completed the SAEBRS-SRS and the Strengths and Difficulties Questionnaire. Data were analyzed to evaluate the diagnostic accuracy of the screener via receiver operating characteristic curve analyses. Multi-group confirmatory factor analysis was conducted to identify measurement invariance across gender. Results indicated that (1) the SAEBRS-SRS has adequate diagnostic accuracy statistics, particularly for the SAEBRS Total score, and (2) the SAEBRS-SRS is invariant across gender at the metric and scalar levels of measurement invariance. Implications and recommendations for future research are further discussed.
Similar content being viewed by others
References
Atkins, M. S., Hoagwood, K. E., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health in schools. Administration and Policy in Mental Health and Mental Health Services Research, 37(1–2), 40–47.
Barrett, S., Eber, L., & Weist, M. D. (2013). Advancing education effectiveness: An interconnected systems framework for positive behavioral interventions and supports (PBIS) and school mental health. Center for positive behavioral interventions and supports. Eugene, OR: University of Oregon Press.
Borsboom, D. (2006). When does measurement invariance matter? Medical care, 44(11), S176−S181.
Bradley, R., Doolittle, J., & Bartolotta, R. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance. Journal of Behavioral Education, 17(1), 4–23.
Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education and Treatment of Children, 37, 611–634.
Burns, M. K., Riley-Tillman, T. C., & Van Der Heyden, A. M. (2012). RTI applications, volume 1: Academic and behavioral interventions (Vol. 1). New York City: Guilford Press.
Cappella, E., Frazier, S. L., Atkins, M. S., Schoenwald, S. K., & Glisson, C. (2008). Enhancing schools’ capacity to support children in poverty: An ecological model of school-based mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 35(5), 395–409.
Carran, D. T., & Scott, K. G. (1992). Risk assessment in preschool children: Research implications for the early detection of educational handicaps. Topics in Early Childhood Special Education, 12, 196–211. doi:10.1177/027112149201200205.
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464–504. doi:10.1080/10705510701301834.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. doi:10.1207/S15328007SEM0902_5.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. doi:10.1037/h0026256.
Cook, C. R., Volpe, R. J., & Livanis, A. (2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35(4), 197–205.
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal or Applied Psychology, 78, 98–104.
De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A., Burgers, D. E., et al. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900. doi:10.1037/a0038498.
Dever, B. V., Kamphaus, R. W., Dowdy, E., Raines, T. C., & DiStefano, C. (2013). Surveillance of middle and high school mental health risk by student self-report screener. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 14(4), 384–390. doi:10.5811/westjem.2013.2.15349.
Dowdy, E., Doane, K., Eklund, K., & Dever, B. V. (2011). A comparison of teacher nomination and screening to identify behavioral and emotional risk within a sample of underrepresented students. Journal of Emotional and Behavioral Disorders. doi:10.1177/1063426611417627.
Dowdy, E., & Kim, E. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6, 1–10.
Drummond, T. (1994). The student risk screening scale (SRSS). Grants Pass, OR: Josephine County Mental Health Program.
Flanagan, K. S., Bierman, K. L., & Kam, C. M. (2003). Identifying at-risk children at school entry: The usefulness of multibehavioral problem profiles. Journal of Clinical Child and Adolescent Psychology, 32(3), 396–407.
Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45(2), 117–135.
Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x.
Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Child and Adolescent Psychiatry, 40(11), 1137–1145.
Goodman, A., & Goodman, R. (2009). Strengths and Difficulties Questionnaire as a dimensional measure of child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 400–403.
Goodman, R., & Scott, S. (1999). Comparing the Strengths and Difficulties Questionnaire and the child behavior checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.
Hemphill, J. F. (2003). Interpreting the magnitude of correlation coefficients. American Psychologist, 58, 78–79.
Hintze, J. M., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34, 372–386.
Jenkins, L. N., Demaray, M. K., Wren, N. S., Secord, S. M., Lyell, K. M., Magers, A. M., et al. (2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18, 241–254. doi:10.1007/s40688-014-0026-6.
Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6, 89–97.
Kamphaus, R. W., & Reynolds, C. R. (2007). BASC-2 behavioral and emotional screening system manual. Bloomington: Pearson.
Kilgus, S. P., Chafouleas, S. M., & Riley-Tillman, T. C. (2013). Development and initial validation of the social and academic behavior risk screener for elementary grades. School Psychology Quarterly, 28, 210–226. doi:10.1037/spq0000024.
Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. doi:10.1016/j.jsp.2016.07.001
Kilgus, S. P., Riley-Tillman, T. C., Chafouleas, S. M., Christ, T. J., & Welsh, M. E. (2014). Direct behavior rating as a school-based behavior universal screener: Replication across sites. Journal of School Psychology, 52, 63–82.
Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. doi:10.1037/spq0000087.
Kilgus, S. P., Sims, W., von der Embse, N. P., & Taylor, C. (2016). Technical adequacy of the social, academic, and emotional behavior risk screener in an elementary sample. Assessment for Effective Intervention, 42(1), 46–59. doi:10.1177/153450841562326.
Kilgus, S. P., & von der Embse, N. P. (2015). Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Minneapolis, MN: Theodore J. Christ & Colleagues.
Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.
Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J., Weisenbach, J. L., et al. (2009). A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 17, 93–105. doi:10.1177/1063426608326203.
Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have Tier 2 and Tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182.
Levitt, J. M., Saka, N., Hunter Romanelli, L., & Hoagwood, K. (2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163–191. doi:10.1016/j.jsp.2006.11.005.
McIntosh, K., Ty, S. V., & Miller, L. D. (2014). Effects of school-wide positive behavioral interventions and supports on internalizing problems current evidence and future directions. Journal of Positive Behavior Interventions, 16(4), 209–218. doi:10.1177/1098300713491980.
Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the national comorbidity survey replication-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980–989. doi:10.1016/j.jaac.2010.05.017.
Merikangas, K. R., Nakamura, E. F., & Kessler, R. C. (2009). Epidemiology of mental disorders in children and adolescents. Dialogues in Clinical Neuroscience, 11(1), 7–20.
Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8, 283–298.
Miller, F. G., Cohen, D., Chafouleas, S. M., Riley- Tillman, T. C., Welsh, M. E., & Fabiano, G. A. (2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196.
Mills, C., Stephan, S. H., Moore, E., Weist, M. D., Daly, B. P., & Edwards, M. (2006). The President’s New Freedom Commission: Capitalizing on opportunities to advance school-based mental health services. Clinical Child and Family Psychology Review, 9, 149–161.
Muthén, L. K. (2012). Model fit index WRMR. Retrieved November 23, 2012, from http://www.statmodel.com/discussion/messages/9/5096.html?1321986275.
Muthén, L. K., & Muthén, B. O. (1998–2013). Mplus (Version 7.1) [Computer software]. Los Angeles, CA: Muthén & Muthén.
Pendergast, L., von der Embse, N. P., Kilgus, S., & Eklund, K. (2017). Measurement equivalence/invariance in school psychology research: A conceptual overview and example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82. doi:10.1016/j.jsp.2016.11.002.
Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., et al. (2013). Mental health surveillance among children—United States, 2005–2011. MMWR Surveillance Summary, 62(2), 1–35.
Petscher, Y., Kim, Y., & Foorman, B. R. (2011). The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention, 36, 158–166.
Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. doi:10.1007/s10979-005-6832-7.
Serbin, L. A., Temcheff, C. E., Cooperman, J. M., Stack, D. M., Ledingham, J., & Schwartzman, A. E. (2011). Predicting family poverty and other disadvantaged conditions for child rearing from childhood aggression and social withdrawal: A 30-year longitudinal study. International Journal of Behavioral Development, 35, 97–106. doi:10.1177/0165025410372008.
Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45(2), 193–223.
Smith, S. R. (2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Psychoeducational Assessment, 25, 139–149. doi:10.1177/0734282906296233.
Streiner, D. L., & Cairney, J. (2007). What’s under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry/La Revue Canadienne de Psychiatrie, 52, 121–128.
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Pearson Education.
Taylor, C., Allen, A., Kilgus, S., von der Embse, N., & Garbacz, S. A. (under review). Development and validation of a parent version of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Assessment for Effective Intervention.
von der Embse, N. P. & Kilgus, S. P. (2015). Unpublished technical manual of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale. Minneapolis, MN: Fastbridge Learning.
von der Embse, N.P., Iaccarino, S., Mankin, A., Kilgus, S. P., & Magen, E. (in press). Development and Validation of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale. Assessment for Effective Intervention, 1534508416679410.
von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. R. (2016). Evaluating the applied use of a mental health screener: Structural validity of the Social, Academic, and Emotional Behavior Risk Screener. Psychological Assessment. doi:10.1037/pas0000253.
Yu, C-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Doctoral Dissertation. https://www.statmodel.com/download/Yudissertation.pdf.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Nathaniel P. von der Embse and Stephen P. Kilgus have a financial conflict of interest due to receipt of royalties on net sales for one of the measures discussed.
Ethical Approval
The American Psychological Association ethical standards were completely adhered to throughout the development of this manuscript. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All consent and assent procedures were approved by both the University Institutional Research Review Board and the local school district superintendent.
Rights and permissions
About this article
Cite this article
von der Embse, N.P., Kilgus, S.P., Iaccarino, S. et al. Screening for Student Mental Health Risk: Diagnostic Accuracy, Measurement Invariance, and Predictive Validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS). School Mental Health 9, 273–283 (2017). https://doi.org/10.1007/s12310-017-9214-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12310-017-9214-7