Skip to main content
Log in

Screening for Student Mental Health Risk: Diagnostic Accuracy, Measurement Invariance, and Predictive Validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS)

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

Schools have become the largest provider of mental health services to children, thus increasing the need to identify children at risk before problems worsen. The Social, Academic, and Emotional Behavioral Risk Screener-Student Rating Scale (SAEBRS-SRS) student version is a universal screener that assesses pre-symptomology of internalizing and externalizing behaviors. Prior research on the SAEBRS-SRS has supported a psychometrically defensible factor structure; however, additional research is necessary to support applied used in schools. In this study, 1102 middle grades students completed the SAEBRS-SRS and the Strengths and Difficulties Questionnaire. Data were analyzed to evaluate the diagnostic accuracy of the screener via receiver operating characteristic curve analyses. Multi-group confirmatory factor analysis was conducted to identify measurement invariance across gender. Results indicated that (1) the SAEBRS-SRS has adequate diagnostic accuracy statistics, particularly for the SAEBRS Total score, and (2) the SAEBRS-SRS is invariant across gender at the metric and scalar levels of measurement invariance. Implications and recommendations for future research are further discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Atkins, M. S., Hoagwood, K. E., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health in schools. Administration and Policy in Mental Health and Mental Health Services Research, 37(1–2), 40–47.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barrett, S., Eber, L., & Weist, M. D. (2013). Advancing education effectiveness: An interconnected systems framework for positive behavioral interventions and supports (PBIS) and school mental health. Center for positive behavioral interventions and supports. Eugene, OR: University of Oregon Press.

  • Borsboom, D. (2006). When does measurement invariance matter? Medical care, 44(11), S176−S181.

    Article  PubMed  Google Scholar 

  • Bradley, R., Doolittle, J., & Bartolotta, R. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance. Journal of Behavioral Education, 17(1), 4–23.

    Article  Google Scholar 

  • Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education and Treatment of Children, 37, 611–634.

    Article  Google Scholar 

  • Burns, M. K., Riley-Tillman, T. C., & Van Der Heyden, A. M. (2012). RTI applications, volume 1: Academic and behavioral interventions (Vol. 1). New York City: Guilford Press.

    Google Scholar 

  • Cappella, E., Frazier, S. L., Atkins, M. S., Schoenwald, S. K., & Glisson, C. (2008). Enhancing schools’ capacity to support children in poverty: An ecological model of school-based mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 35(5), 395–409.

    Article  PubMed  PubMed Central  Google Scholar 

  • Carran, D. T., & Scott, K. G. (1992). Risk assessment in preschool children: Research implications for the early detection of educational handicaps. Topics in Early Childhood Special Education, 12, 196–211. doi:10.1177/027112149201200205.

    Article  Google Scholar 

  • Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464–504. doi:10.1080/10705510701301834.

    Article  Google Scholar 

  • Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. doi:10.1207/S15328007SEM0902_5.

    Article  Google Scholar 

  • Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. doi:10.1037/h0026256.

    Article  PubMed  Google Scholar 

  • Cook, C. R., Volpe, R. J., & Livanis, A. (2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35(4), 197–205.

    Article  Google Scholar 

  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal or Applied Psychology, 78, 98–104.

    Article  Google Scholar 

  • De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A., Burgers, D. E., et al. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900. doi:10.1037/a0038498.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dever, B. V., Kamphaus, R. W., Dowdy, E., Raines, T. C., & DiStefano, C. (2013). Surveillance of middle and high school mental health risk by student self-report screener. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 14(4), 384–390. doi:10.5811/westjem.2013.2.15349.

    Article  Google Scholar 

  • Dowdy, E., Doane, K., Eklund, K., & Dever, B. V. (2011). A comparison of teacher nomination and screening to identify behavioral and emotional risk within a sample of underrepresented students. Journal of Emotional and Behavioral Disorders. doi:10.1177/1063426611417627.

    Google Scholar 

  • Dowdy, E., & Kim, E. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6, 1–10.

    Google Scholar 

  • Drummond, T. (1994). The student risk screening scale (SRSS). Grants Pass, OR: Josephine County Mental Health Program.

    Google Scholar 

  • Flanagan, K. S., Bierman, K. L., & Kam, C. M. (2003). Identifying at-risk children at school entry: The usefulness of multibehavioral problem profiles. Journal of Clinical Child and Adolescent Psychology, 32(3), 396–407.

    Article  PubMed  PubMed Central  Google Scholar 

  • Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45(2), 117–135.

    Article  Google Scholar 

  • Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x.

    Article  PubMed  Google Scholar 

  • Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Child and Adolescent Psychiatry, 40(11), 1137–1145.

    Article  Google Scholar 

  • Goodman, A., & Goodman, R. (2009). Strengths and Difficulties Questionnaire as a dimensional measure of child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 400–403.

    Article  PubMed  Google Scholar 

  • Goodman, R., & Scott, S. (1999). Comparing the Strengths and Difficulties Questionnaire and the child behavior checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.

    Article  PubMed  Google Scholar 

  • Hemphill, J. F. (2003). Interpreting the magnitude of correlation coefficients. American Psychologist, 58, 78–79.

    Article  PubMed  Google Scholar 

  • Hintze, J. M., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34, 372–386.

    Google Scholar 

  • Jenkins, L. N., Demaray, M. K., Wren, N. S., Secord, S. M., Lyell, K. M., Magers, A. M., et al. (2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18, 241–254. doi:10.1007/s40688-014-0026-6.

    Article  Google Scholar 

  • Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6, 89–97.

    Google Scholar 

  • Kamphaus, R. W., & Reynolds, C. R. (2007). BASC-2 behavioral and emotional screening system manual. Bloomington: Pearson.

    Google Scholar 

  • Kilgus, S. P., Chafouleas, S. M., & Riley-Tillman, T. C. (2013). Development and initial validation of the social and academic behavior risk screener for elementary grades. School Psychology Quarterly, 28, 210–226. doi:10.1037/spq0000024.

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. doi:10.1016/j.jsp.2016.07.001

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Riley-Tillman, T. C., Chafouleas, S. M., Christ, T. J., & Welsh, M. E. (2014). Direct behavior rating as a school-based behavior universal screener: Replication across sites. Journal of School Psychology, 52, 63–82.

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. doi:10.1037/spq0000087.

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Sims, W., von der Embse, N. P., & Taylor, C. (2016). Technical adequacy of the social, academic, and emotional behavior risk screener in an elementary sample. Assessment for Effective Intervention, 42(1), 46–59. doi:10.1177/153450841562326.

    Article  Google Scholar 

  • Kilgus, S. P., & von der Embse, N. P. (2015). Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Minneapolis, MN: Theodore J. Christ & Colleagues.

    Google Scholar 

  • Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

    Google Scholar 

  • Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J., Weisenbach, J. L., et al. (2009). A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 17, 93–105. doi:10.1177/1063426608326203.

    Article  Google Scholar 

  • Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have Tier 2 and Tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182.

    Article  Google Scholar 

  • Levitt, J. M., Saka, N., Hunter Romanelli, L., & Hoagwood, K. (2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163–191. doi:10.1016/j.jsp.2006.11.005.

    Article  Google Scholar 

  • McIntosh, K., Ty, S. V., & Miller, L. D. (2014). Effects of school-wide positive behavioral interventions and supports on internalizing problems current evidence and future directions. Journal of Positive Behavior Interventions, 16(4), 209–218. doi:10.1177/1098300713491980.

    Article  Google Scholar 

  • Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the national comorbidity survey replication-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980–989. doi:10.1016/j.jaac.2010.05.017.

    Article  PubMed  PubMed Central  Google Scholar 

  • Merikangas, K. R., Nakamura, E. F., & Kessler, R. C. (2009). Epidemiology of mental disorders in children and adolescents. Dialogues in Clinical Neuroscience, 11(1), 7–20.

    PubMed  PubMed Central  Google Scholar 

  • Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8, 283–298.

    Article  PubMed  Google Scholar 

  • Miller, F. G., Cohen, D., Chafouleas, S. M., Riley- Tillman, T. C., Welsh, M. E., & Fabiano, G. A. (2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196.

    Article  PubMed  Google Scholar 

  • Mills, C., Stephan, S. H., Moore, E., Weist, M. D., Daly, B. P., & Edwards, M. (2006). The President’s New Freedom Commission: Capitalizing on opportunities to advance school-based mental health services. Clinical Child and Family Psychology Review, 9, 149–161.

    Article  PubMed  Google Scholar 

  • Muthén, L. K. (2012). Model fit index WRMR. Retrieved November 23, 2012, from http://www.statmodel.com/discussion/messages/9/5096.html?1321986275.

  • Muthén, L. K., & Muthén, B. O. (1998–2013). Mplus (Version 7.1) [Computer software]. Los Angeles, CA: Muthén & Muthén.

  • Pendergast, L., von der Embse, N. P., Kilgus, S., & Eklund, K. (2017). Measurement equivalence/invariance in school psychology research: A conceptual overview and example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82. doi:10.1016/j.jsp.2016.11.002.

    Article  PubMed  Google Scholar 

  • Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., et al. (2013). Mental health surveillance among children—United States, 2005–2011. MMWR Surveillance Summary, 62(2), 1–35.

    Google Scholar 

  • Petscher, Y., Kim, Y., & Foorman, B. R. (2011). The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention, 36, 158–166.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. doi:10.1007/s10979-005-6832-7.

    Article  PubMed  Google Scholar 

  • Serbin, L. A., Temcheff, C. E., Cooperman, J. M., Stack, D. M., Ledingham, J., & Schwartzman, A. E. (2011). Predicting family poverty and other disadvantaged conditions for child rearing from childhood aggression and social withdrawal: A 30-year longitudinal study. International Journal of Behavioral Development, 35, 97–106. doi:10.1177/0165025410372008.

    Article  Google Scholar 

  • Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45(2), 193–223.

    Article  Google Scholar 

  • Smith, S. R. (2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Psychoeducational Assessment, 25, 139–149. doi:10.1177/0734282906296233.

    Article  Google Scholar 

  • Streiner, D. L., & Cairney, J. (2007). What’s under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry/La Revue Canadienne de Psychiatrie, 52, 121–128.

    Article  Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Pearson Education.

    Google Scholar 

  • Taylor, C., Allen, A., Kilgus, S., von der Embse, N., & Garbacz, S. A. (under review). Development and validation of a parent version of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Assessment for Effective Intervention.

  • von der Embse, N. P. & Kilgus, S. P. (2015). Unpublished technical manual of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale. Minneapolis, MN: Fastbridge Learning.

  • von der Embse, N.P., Iaccarino, S., Mankin, A., Kilgus, S. P., & Magen, E. (in press). Development and Validation of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale. Assessment for Effective Intervention, 1534508416679410.

  • von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. R. (2016). Evaluating the applied use of a mental health screener: Structural validity of the Social, Academic, and Emotional Behavior Risk Screener. Psychological Assessment. doi:10.1037/pas0000253.

    Google Scholar 

  • Yu, C-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Doctoral Dissertation. https://www.statmodel.com/download/Yudissertation.pdf.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nathaniel P. von der Embse.

Ethics declarations

Conflict of interest

Nathaniel P. von der Embse and Stephen P. Kilgus have a financial conflict of interest due to receipt of royalties on net sales for one of the measures discussed.

Ethical Approval

The American Psychological Association ethical standards were completely adhered to throughout the development of this manuscript. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All consent and assent procedures were approved by both the University Institutional Research Review Board and the local school district superintendent.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

von der Embse, N.P., Kilgus, S.P., Iaccarino, S. et al. Screening for Student Mental Health Risk: Diagnostic Accuracy, Measurement Invariance, and Predictive Validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS). School Mental Health 9, 273–283 (2017). https://doi.org/10.1007/s12310-017-9214-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-017-9214-7

Keywords

Navigation