Top

Journal of Psychopathology and Behavioral Assessment

Gepubliceerd in:

04-02-2021

Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes

Auteurs: Nathaniel von der Embse, Eunsook Kim, Andrew Jenkins, Alexis Sanchez, Stephen P. Kilgus, Katie Eklund

Gepubliceerd in: Journal of Psychopathology and Behavioral Assessment | Uitgave 3/2021

Abstract

The success of a comprehensive and integrated school mental health system is dependent on data to inform decision-making. Universal screening is one such method utilized by schools to inform treatment across universal, targeted, and individual levels of service. However, the use of multiple informants (teacher and student) remains elusive due to a lack of practical and research based-aggregation guidance. The purpose of the current study is to identify the optimal numbers of rater agreement profiles, determine demographic predictors of profiles membership, and if profiles differentially predict distal academic outcomes. Nearly 24,000 teacher-student dyads were included in this study. Results suggested four profiles emerged with significant differences in academic outcomes. Considerations for research and practice are presented.

vorige artikel Levels of Personality Functioning Questionnaire (LoPF-Q) 12–18 Turkish Version: Reliability, Validity, Factor Structure and Relationship with Comorbid Psychopathology in a Turkish Adolescent Sample

volgende artikel Teacher Nominations of Preschool Children at Risk for Mental Health Problems: how False Is a False Positive Nomination and What Make Teachers Concerned?

We also conducted Little’s MCAR tests: χ²(317) = 746.88, p < .001 for student and teacher SAEBRS scores; χ²(317) = 700.51, p < .001 with gender and race covariates additionally; and χ²(507) = 3645.74, p < .001 with reading and math outcomes additionally. However, given large sample size in this study, we majorly interpreted the relationship between missingness and other variables with correlation coefficients for the missing mechanism.

As reported earlier, the missingness of gender and race were not associated with any variables in the model and thus possibly happened completely at random. When we compared the results of a model with special education status and grade which were complete (N = 24,094) to those of a model with all four predictors (N = 16,710), general findings in terms of special education status and grade remained consistent although their coefficients changed by controlling for the other predictors. Hence, we concluded that listwise deletion less likely made any notable differences in the findings. In addition, we did not consider multiple imputation which generally assumes multivariate normality that is not relevant for binary variables, gender and race.

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioural and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.

Adelman, H. S., & Taylor, L. (2007). Systemic change for school improvement. Journal of Educational and Psychological Consultation, 17(1), 55–77. https://doi.org/10.1080/10474410709336590.CrossRef

Auerbach, E. R., Chafouleas, S. M., Briesch, A. M., & Long, S. J. (2020). Exploring the alignment of behavior screening policies and practices in US public school districts. Journal of School Health, 90, 264–270. https://doi.org/10.1111/josh.12872.CrossRef

Barrett, S., Eber, L., & Weist, M. (Eds.) (2013). Advancing educational effectiveness: Interconnecting school mental health and school-wide positive behavior support. Retrieved from http://www.pbis.org/common/cms/files/Current%20Topics/Final-Monograph.pdf.

Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. https://doi.org/10.1037/a0032475.

Berlin, K. S., Williams, N. A., & Parra, G. R. (2014). An introduction to latent variable mixture modeling (part 1): Overview and cross-sectional latent class and latent profile analyses. Journal of Pediatric Psychology, 39(2), 174–187. https://doi.org/10.1093/jpepsy/jst084.CrossRefPubMed

Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.

Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K–12 schools. Education and Treatment of Children, 37(4), 611–634. https://doi.org/10.1353/etc.2014.0039.CrossRef

Christ, T. J., Aranas, Y., Johnson, L., Kember, J., Kilgus, S., Kiss, A., et al. (2014). Formative Assessment System for Teachers Technical Manual. Minneapolis, MN: FastBridge Learning.

Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. John Wiley & Sons, Inc.

Cunningham, J. M., & Suldo, S. M. (2014). Accuracy of teachers in identifying elementary school students who report at-risk levels of anxiety and depression. School Mental Health, 6, 237–250. https://doi.org/10.1007/s12310-014-9125-9.CrossRef

Curhan, A. L., Rabinowitz, J. A., Pas, E. T., & Bradshaw, C. P. (2020). Informant discrepancies in internalizing and externalizing symptoms in an at-risk sample: The role of parenting and school engagement. Journal of Youth and Adolescence, 49, 311–322. https://doi.org/10.1007/s10964-019-01107-x.CrossRefPubMed

De Los Reyes, A. (2013). Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Development and Psychopathology, 25(3), 669–682.CrossRef

De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16(3), 330–334. https://doi.org/10.1037/1040-3590.16.3.330.CrossRefPubMed

De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A. G., & Burgers, D.,& Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498.CrossRefPubMedPubMedCentral

DiStefano, C. (2012). Cluster analysis and latent class clustering techniques. In B. Laursen, T. D. Little, & N. A. Card (Eds.), Handbook of developmental research methods (pp. 645–666). Guilford Publications: New York.

Domitrovich, C. E., Bradshaw, C. P., Greenberg, M. T., Embry, D., Poduska, J. M., & Ialongo, N. S. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47(1), 71–88. https://doi.org/10.1002/pits.20452.CrossRefPubMedPubMedCentral

Dowdy, E., Dever, B. V., Raines, T. C., & Moffa, K. (2016). A preliminary investigation into the added value of multiple gates and informants in universal screening for behavioral and emotional risk. Journal of Applied School Psychology, 32(2), 178–198. https://doi.org/10.1080/15377903.2016.1165327.CrossRef

Dowdy, E., & Kim, R. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.

Dowdy, E., Ritchey, K., & Kamphaus, R. W. (2010). School-based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. https://doi.org/10.1007/s12310-010-9036-3.CrossRefPubMedPubMedCentral

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.CrossRef

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Special Issue: Raising Healthy Children, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x.CrossRef

Eklund, K., & Dowdy, E. (2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40–49. https://doi.org/10.1007/s12310-013-9109-1.CrossRef

Eklund, K., Kilgus, S., von der Embse, N., Beardmore, M., & Tanner, N. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29(5), 486–499. https://doi.org/10.1037/pas0000355.CrossRefPubMed

Elliott, S. N., & Gresham, F. (2008). Social Skills Improvement System (SSiS)—Performance Screening Guide. San Antonio, TX: PsychCorp Pearson Education.

Evans, S. W., Stephan, S. H., & Sugai, G. (2014). Advancing research in school mental health: Introduction of a special issue on key issues in research. School Mental Health, 6, 63–67. https://doi.org/10.1007/s12310-014-9126-8.CrossRefPubMedPubMedCentral

Franklin, C. G. S., Kim, J. S., Ryan, T. N., Kelly, M. S., & Montgomery, K. L. (2012). Teacher involvement in school mental health interventions: A systematic review. Children and Youth Services Review, 34(5), 973–982. https://doi.org/10.1016/j.childyouth.2012.01.027.CrossRef

Goolsby, C., Jacobs, L., Hunt, R. C., Goralnick, E., Singletary, E. M., Levy, M. J., Goodloe, J. M., Epstein, J. L., Strauss-Riggs, K., Seitz, S. R., Krohmer, J. R., Nemeth, I., Rowe, D. W., Bradley, R. N., Gestring, M. L., & Kirsch, T. D. (2018). Stop the bleed education consortium. Journal of Trauma and Acute Care Surgery, 84(1), 205–210. https://doi.org/10.1097/TA.0000000000001732.CrossRef

Herman, K. C., Cohen, D., Reinke, W. M., Ostrander, R., Burrell, L., McFarlane, E. K., & Duggan, A. K. (2018). Using latent profile and transition analyses to understand patterns of informant ratings of child depressive symptoms. Journal of School Psychology, 69, 84–99. https://doi.org/10.1016/j.jsp.2018.05.004.CrossRefPubMed

Holzinger, K. J., & Swineford, F. (1937). The bi-factor model. Psychometrika, 2, 41–54. https://doi.org/10.1007/BF02287965.

Iaccarino, S., von der Embse, N. P., & Kilgus, S. P. (2018). Interpretation and use of the social, academic, and emotional behavior risk screener: A latent transition approach. Journal of Psychoeducational Assessment, 37, 486–503. https://doi.org/10.1177/0734282918766650.CrossRef

Kahlberg, J. R., Lane, K. L., Driscoll, S., & Wehby, J. (2011). Systematic screening for emotional and behavioral disorders at the high school level: A formidable and necessary task. Remedial and Special Education, 32(6), 506–520. https://doi.org/10.1177/0741932510362508.CrossRef

Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6(4), 89–97.

Kilgus, S. P., Bowman, N. A., Christ, T. J., & Taylor, C. N. (2017). Predicting academics via behavior within an elementary sample: An evaluation of the social, academic, and emotional behavior risk screener (SAEBRS). Psychology in the Schools, 54, 246–260.CrossRef

Kilgus, S. P., Chafouleas, S. M., Riley-Tillman, T. C., & von der Embse, N. P. (2014). Social, academic, and emotional behavior risk screener (SAEBRS). Minneapolis: Theodore J. Christ & Colleagues.

Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C., & Sims, W. A. (2016). Psychometric defensibility of the social, academic, and emotional behavior risk screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. https://doi.org/10.1016/j.jsp.2016.07.001.CrossRefPubMed

Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. https://doi.org/10.1037/spq0000087.CrossRefPubMed

Kilgus, S. P., & von der Embse, N. P. (2014). Unpublished technical manual of the social, academic, and emotional behavior risk screener.

Kilgus, S. P., Bonifay, W., von der Embse, N. P., & Taylor, C. (2018). Evidence for the interpretation of Social, Academic, and Emotional, Behavior Risk Screener (SAEBRS) scores: An argument-based approach to screener validation. Journal of School Psychology, 68, 129–141. https://doi.org/10.1016/j.jsp.2018.03.002.CrossRefPubMed

Kim, E. S., Joo, S. -H., Lee, P., Wang, Y., & Stark, S. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870–887. https://doi.org/10.1080/10705511.2016.1196108.

Kim, E. S., & Wang, Y. (2019). Investigating sources of heterogeneity with three-step multilevel factor mixture modeling: Beyond testing measurement invariance in cross-national studies. Structural Equation Modeling: A Multidisciplinary Journal, 26(2), 165–181. https://doi.org/10.1080/10705511.2018.1521284.CrossRef

King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the validity of the behavioral and emotional screening system in a rural elementary school: Validity of the BESS. Journal of Psychoeducational Assessment, 30(6), 527–538. https://doi.org/10.1177/0734282912440673.CrossRef

Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(9), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566.CrossRef

Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have tier 2 and tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182. https://doi.org/10.1080/1045988X.2014.895573.CrossRef

Laursen, B., & Hoff, E. (2006). Person-centered and variable-centered approaches to longitudinal data. Merrill-Palmer Quarterly, 52(3), 377–389.CrossRef

Lubke, G., & Neale, M. C. (2008). Distinguishing between latent classes and continuous factors with categorical outcomes: Class invariance of parameters of factor mixture models. Multivariate Behavioral Research, 43(4), 592–620. https://doi.org/10.1080/00273170802490673.CrossRefPubMedPubMedCentral

Lyon, A. R., Maras, M. A., Pate, C. M., Igusa, T., & Stoep, A. V. (2016). Modeling the impact of school-based universal depression screening on additional service capacity needs: A system dynamics approach. Administration and Policy in Mental Health and Mental Health Services Research, 43, 168–188. https://doi.org/10.1007/s10488-015-0628-y.CrossRefPubMed

Makol, B. A., Youngstrom, E. A., Racz, S. J., Qasmieh, N., Glenn, L. E., & De Los Reyes, A. (2020). Integrating multiple informants’ reports: How conceptual and measurement models may address long-standing problems in clinical decision-making. Clinical Psychological Science.

Masyn, K. E. (2013). 25 latent class analysis and finite mixture modeling. The Oxford handbook of quantitative methods, 551

Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling, 16, 191–225.CrossRef

Meyer, J. P., & Morin, A. J. S. (2016). A person-centered approach to commitment research: Theory, research, and methodology. Journal of Organizational Behavior, 37, 584–612. https://doi.org/10.1002/job.2085.CrossRef

Pastor, D. A., Barron, K. E., Miller, B. J., & Davis, S. L. (2007). A latent profile analysis of college students’ achievement goal orientation. Contemporary Educational Psychology, 32, 8–47.CrossRef

Pendergast, L., von der Embse, N. P., Kilgus, S. P., & Eklund, K. (2017). Measurement equivalence in school psychology research: A primer and illustrated example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82.CrossRef

Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., Hedden, S. L., Crosby, A. E., Visser, S. N., Schieve, L. A., Parks, S. E., Hall, J. E., Brody, D., Simile, C. M., Thompson, W. W., Baio, J., Avenevoli, S., Kogan, M. D., & Huang, L. N. (2013). Mental health surveillance among children--United States, 2005-2011. Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report, 62(2), 1–35.

Romer, N., von der Embse, N., Eklund, K., Kilgus, S., Perales, K., Splett, J. W., Sudlo, S., Wheeler, D. (2020). Best Practices in Social, Emotional, and Behavioral Screening: An Implementation Guide. Version 2.0. From www.smhcollaborative.org/universalscreening

Shelton, A. J., Wang, C. D., & Zhu, W. (2017). Perceived social support and mental health: Cultural orientations as moderators. Journal of College Counseling, 20(3), 194–207.CrossRef

Skiba, R. J., Michael, R. S., Nardo, A. C., & Peterson, R. L. (2002). The color of discipline: Sources of racial and gender disproportionality in school punishment. The Urban Review, 34(4), 317–342.CrossRef

Sklad, M., Diekstra, R., Ritter, M., Ben, J., & Gravesteijn, C. (2012). Effectiveness of school-based universal social, emotional, behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment? Psychology in the Schools, 49(9), 892–909. https://doi.org/10.1002/pits.21641.CrossRef

Substance Abuse and Mental Health Services Administration. (2014). National Survey on Drug Use and Health: Summary of Methodological Studies, 1971–2014 [Internet]. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2014 Nov. Available from: https://www.ncbi.nlm.nih.gov/books/NBK519735/

Taylor, C., Allen, A., Kilgus, S. P., von der Embse, N. P., & Garbacz, S. A. (2018). Development and validation of a parent version of the social, academic, and emotional behavior risk screener (SAEBRS) in an elementary sample. Behavioral Disorders., 44, 205–214. https://doi.org/10.1177/0198742918798561.CrossRef

Tein, J., Coxe, S., & Cham, H. (2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling, 20(4), 640–657. https://doi.org/10.1080/10705511.2013.824781.CrossRefPubMedPubMedCentral

Vermunt, J. K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469. https://doi.org/10.1093/pan/mpq025.CrossRef

von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017). Development and factor structure of the Social, Academic, and Emotional Behavior Risk Screener Student Rating Scale (SAEBRS-SRS). Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.CrossRef

von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017a). Development and validation of the social, academic, and emotional behavior risk screener-student rating scale. Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.CrossRef

von der Embse, N. P., Kilgus, S. P., Iaccarino, S., & Levi-Nielsen, S. (2017b). Screening for student mental health risk: Diagnostic accuracy, measurement invariance, and predictive validity of the social, academic, and emotional behavior risk screener-student rating scale (SAEBRS-SRS). School Mental Health, 9(3), 273–283. https://doi.org/10.1007/s12310-017-9214-7.CrossRef

von der Embse, N. P., Kilgus, S. P., Eklund, K., Ake, E., & Levi-Neilsen, S. (2018). Training teachers to facilitate early identification of mental and behavioral health risks. School Psychology Review, 47(4), 372–384. https://doi.org/10.17105/SPR-2017-0094.V47-4.CrossRef

von der Embse, N., Kim, E. S., Kilgus, S., Dedrick, R., & Sanchez, A. (2019). Multi-informant universal screening: Evaluation of rater, item, and construct variance using a trifactor model. Journal of School Psychology, 77, 52–66. https://doi.org/10.1016/j.jsp.2019.09.005.CrossRefPubMed

von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. (2016). Evaluating the applied use of a mental health screener: Structural validity of the social, academic, and emotional behavior risk screener (SAEBRS). Psychological Assessment, 28(10), 1265–1275. https://doi.org/10.1037/pas0000253.CrossRefPubMed

Youngstrom, E. A., Halverson, T. F., Youngstrom, J. K., Lindhiem, O., & Findling, R. L. (2018). Evidence-based assessment from simple clinical judgments to statistical learning: Evaluating a range of options using pediatric bipolar disorder as a diagnostic challenge. Clinical Psychological Science, 6(2), 243–265. https://doi.org/10.1177/2167702617741845.CrossRefPubMed

Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038–1050. https://doi.org/10.1037//0022-006X.68.6.1038.CrossRefPubMed

Titel: Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes
Auteurs: Nathaniel von der Embse
Eunsook Kim
Andrew Jenkins
Alexis Sanchez
Stephen P. Kilgus
Katie Eklund
Publicatiedatum: 04-02-2021
Uitgeverij: Springer US
Gepubliceerd in: Journal of Psychopathology and Behavioral Assessment / Uitgave 3/2021
Print ISSN: 0882-2689
Elektronisch ISSN: 1573-3505
DOI: https://doi.org/10.1007/s10862-021-09869-0

Bohn Stafleu van Loghum

Deel dit onderdeel of sectie (kopieer de link)

Abstract

Log in om toegang te krijgen

Andere artikelen Uitgave 3/2021

Which Emotion Regulation Strategies are Most Associated with Trait Emotion Dysregulation? A Transdiagnostic Examination

Comorbidity Between Depression and Anxiety in Adolescents: Bridge Symptoms and Relevance of Risk and Protective Factors

Psychometric Properties of the Bullying Participant Behaviors Questionnaire (BPBQ) among Chinese Middle School Students

Factor Structure of the Psychopathy Checklist: Screening Version (PCL:SV): A Systematic Review Using Narrative Synthesis

Exploring Latent Profiles of Psychopathology in a Sample of Lonely People Seeking Treatment

Negative Halo Effects in Parent Ratings of ADHD, ODD, and CD