Top

Journal of Psychopathology and Behavioral Assessment

Gepubliceerd in:

01-02-2023

Multi-informant Assessment of Internalizing Concerns: Rater Concordance and Implications for Decision-Making

Auteurs: Nathaniel von der Embse, Eunsook Kim, Dorie Ross, Stephen Kilgus, Thomas Koza

Gepubliceerd in: Journal of Psychopathology and Behavioral Assessment | Uitgave 1/2023

Abstract

Rising rates of mental health challenges among youths have become a significant concern following the COVID 19 pandemic. Although strong evidence supports the implementation of universal screening as a preventative approach to address unmet mental health concerns, the research is less clear surrounding the use of such data in decision-making processes when significant discrepancies between informants (e.g., students and teachers) exist. The purpose of the study was twofold. First, the study aimed to determine the degree of rater concordance between teachers and students on students’ internalizing concerns. The second objective was to determine whether concordance on internalizing behaviors differs across ages/grades and if this differentially impacts distal (i.e., academic) outcomes. Results indicated that teachers and students demonstrated limited agreement on ratings of internalizing behaviors. However, when students and teachers agreed, higher and more positive emotional behaviors were linked to higher reading/math performance. Furthermore, patterns of informant dis/agreement and relationships between internalizing concerns and academic outcomes were similar across grade levels. Implications and areas for future research are discussed.

vorige artikel School-Age Child Routines: Adaptation and Validation Studies of the Portuguese Version of the Child Routines Questionnaire

volgende artikel Development and Validation of an Arabic Version of the World Health Organization Well-Being Index (WHO-5)

http://public.kenan-flagler.unc.edu/faculty/edwardsj/downloads.htm.

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213.CrossRefPubMed

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705.CrossRef

Anderson, J. K., Ford, T., Soneson, E., Coon, J. T., Humphrey, A., Rogers, M., Moore, D., Jones, P. B., Clarke, E., & Howarth, E. (2019). A systematic review of effectiveness and cost-effectiveness of school-based identification of children and young people at risk of, or currently experiencing mental health difficulties. Psychological Medicine, 49(1), 9–19. https://doi.org/10.1017/S0033291718002490.CrossRefPubMed

Auerbach, E. R., Chafouleas, S. M., & Briesch, A. M. (2019). State-Level Guidance on School-Based screening for Social, Emotional, and behavioral risk: a Follow-Up study. School Mental Health, 11(1), 141–147. https://doi.org/10.1007/s12310-018-9278-z.CrossRef

Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological methods, 18(4), 475–493. https://doi.org/10.1037/a0032475.CrossRefPubMedPubMedCentral

Baumgartner, N., Häberling, I., Emery, S., Strumberger, M., Nalani, K., Erb, S., Bachmann, S., Wöckel, L., Müller-Knapp, U., Rhiner, B., Contin-Waldvogel, B., Schmeck, K., Walitza, S., & Berger, G. (2020). When parents and children disagree: informant discrepancies in reports of depressive symptoms in clinical interviews. Journal of Affective Disorders, 272, 223–230. https://doi.org/10.1016/j.jad.2020.04.008.CrossRefPubMed

Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening Practices in K-12 schools. Education and Treatment of Children, 37(4), 611–634. https://doi.org/10.1353/etc.2014.0039.CrossRef

Christ, T. J., Aranas, Y., Johnson, L., Kember, J., Kilgus, S., Kiss, A., McCarthy Trentman, A., Monaghen, B., Nelson, G., Nelson, P., Newell, K., Van Norman, E., White, M., & Windram, H. (2014). Formative Assessment System for Teachers Technical Manual. Minneapolis, MN: FastBridge Learning.

Collins, T. A., Dart, E. H., & Arora, P. G. (2019). Addressing the internalizing behavior of students in schools: Applications of the MTSS model. School Mental Health, 11(2), 191–193.CrossRef

De Los Reyes, A. (2013). Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Development and Psychopathology, 25(3), 669–682. https://doi.org/10.1017/S0954579413000096.CrossRef

De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A. G., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900.CrossRefPubMedCentral

De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16(3), 330–334. https://doi.org/10.1037/1040-3590.16.3.330.CrossRef

DiPerna, J.C. (2006) Academic enablers and student achievement: Implications for assessment and intervention services in the schools. Psychology in the Schools 43(1) 7–17 https://doi.org/10.1002/pits.20125CrossRef

Dever, B. V., Dowdy, E., & DiStefano, C. (2018). Examining the Stability, Accuracy, and predictive validity of behavioral–emotional screening Scores across Time to inform repeated screening procedures. School Psychology Review, 47(4), 360–371. https://doi.org/10.17105/SPR-2017-0092.V47-4.CrossRef

Dicke, A. L., Lüdtke, O., Trautwein, U., Nagy, G., & Nagy, N. (2012). Judging students’ achievement goal orientations: are teacher ratings accurate? Learning and Individual Differences, 22(6), 844–849. https://doi.org/10.1016/j.lindif.2012.04.004.CrossRef

Edwards, J. R., & Cable, D. M. (2009). The value of value congruence. Journal of Applied Psychology, 94(3), 654–677. https://doi.org/10.1037/a0014891.CrossRefPubMed

Eklund, K., Kilgus, S. P., von der Embse, N., Beardmore, M., & Tanner, N. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: a multi-level approach. Psychological Assessment, 29, 486–499. doi:https://doi.org/10.1037/pas0000355.CrossRefPubMed

Evans, S. W., Simonsen, B., & Dolan, G. (2013). School-level practices. In L. Eber, S. Barrett, & M. Weist (Eds.), Advancing education effectiveness: interconnecting school mental health and school-wide positive behavioral interventions and supports (pp. 86–107). Eugene, OR: University of Oregon, College of Education.

Herman, K. C., Cohen, D., Reinke, W. M., Ostrander, R., Burrell, L., McFarlane, E., & Duggan, A. K. (2018). Using latent profile and transition analyses to understand patterns of informant ratings of child depressive symptoms. Journal of School Psychology, 69, 84–99. https://doi.org/10.1016/j.jsp.2018.05.004.CrossRefPubMed

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118.CrossRef

Jenkins, A., Thoman, S., Wang, Y., von der Embse, N., Kilgus, S., & Reynolds, F. (2021). Identifying type and evaluating stability of behavioral risk. Psychology in the Schools, 58(11), 2225–2237. https://doi.org/10.1002/pits.22587.CrossRef

Kalberg, J. R., Lane, K. L., Driscoll, S., & Wehby, J. (2011). Systematic screening for emotional and behavioral Disorders at the High School Level: a formidable and necessary Task. Remedial and Special Education, 32(6), 506–520. https://doi.org/10.1177/0741932510362508.CrossRef

Kilgus, S. P., Bowman, N. A., Christ, T. J., & Taylor, C. N. (2017). Predicting academics via behavior within an elementary sample: an evaluation of the Social, Academic, and emotional behavior risk screener (SAEBRS). Psychology in the Schools, 54(3), 246–260. https://doi.org/10.1002/pits.21995.CrossRef

Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and emotional behavior risk screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. https://doi.org/10.1016/j.jsp.2016.07.001.CrossRefPubMed

Kilgus, S. P., Sims, W. A., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS). School Psychology Quarterly, 30(3), 335–352. https://doi.org/10.1037/spq0000087.CrossRefPubMed

Kilgus, S. P., Sims, W. A., von der Embse, N. P., & Taylor, C. N. (2016). Technical adequacy of the Social, Academic, and Emotional Behavior Risk Screener in an Elementary Sample. Assessment for Effective Intervention, 42(1), 46–59. https://doi.org/10.1177/15345084156232693.CrossRef

Kilgus, S. P., & von der Embse, N. P. (2014). Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Unpublished technical manual.

Kim, E. & von der Embse, N. & von der, Combined approach to multi-informant data using latent factors and latent classes: Trifactor mixture model, https://doi.org/10.1177/0013164420973722 (2020).

Leadbeater, B. J., & Homel, J. (2015). Irritable and defiant sub-dimensions of ODD: their stability and prediction of internalizing symptoms and conduct problems from adolescence to young adulthood. Journal of Abnormal Child Psychology, 43(3), 407–421. https://doi.org/10.1007/s10802-014-9908-3.CrossRefPubMedPubMedCentral

Lyneham, H. J., Street, A. K., Abbott, M. J., & Rapee, R. M. (2008). Psychometric properties of the school anxiety scale—teacher report (SAS-TR). Journal of Anxiety Disorders, 22(2), 292–300. https://doi.org/10.1016/j.janxdis.2007.02.001.CrossRefPubMed

Lyon, A. R., Whitaker, K., French, W. P., Richardson, L. P., Wasse, J. K., & McCauley, E. (2016). Collaborative care in schools: enhancing integration and impact in youth mental health. Advances in School Mental Health Promotion, 9(3–4), 148–168. https://doi.org/10.1080/1754730X.2016.1215928.CrossRefPubMedPubMedCentral

Makol, B. A., Youngstrom, E. A., Racz, S. J., Qasmieh, N., Glenn, L. E., & De Los Reyes, A. (2020). Integrating multiple Informants’ reports: how conceptual and measurement models may address long-standing problems in clinical decision-making. Clinical Psychological Science, 8(6), 953–970. https://doi.org/10.1177/2167702620924439.CrossRef

McDonald, R. P. (1999). Test theory: a unified treatment. Mahwah, NJ: Erlbaum.

Merikangas, K. R., He, J., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., Benjet, C., Georgiades, K., & Swendsen, J. (2010). Lifetime prevalence of Mental Disorders in US adolescents: results from the National Comorbidity Survey replication–adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 49(10), 980–989. https://doi.org/10.1016/j.jaac.2010.05.017.CrossRef

Miller, L. D., Martinez, Y. J., Shumka, E., & Baker, H. (2014). Multiple informant agreement of child, parent, and teacher ratings of child anxiety within community samples. Canadian Journal Of Psychiatry, 59(1), 34–39. https://doi.org/10.1177/070674371405900107.CrossRefPubMedPubMedCentral

Mojtabai, R., Olfson, M., & Han, B. (2016). National Trends in the prevalence and treatment of Depression in Adolescents and Young adults. Pediatrics, 138(6), e20161878. https://doi.org/10.1542/peds.2016-1878.CrossRefPubMedPubMedCentral

Navarro, M. C., Orri, M., Nagin, D., Tremblay, R. E., Oncioiu, S. I., Ahun, M. N., Melchior, M., van der Waerden, J., Galéra, C., & Côté, S. M. (2020). Adolescent internalizing symptoms: the importance of multi-informant assessments in childhood. Journal of Affective Disorders, 266, 702–709. https://doi.org/10.1016/j.jad.2020.01.106.CrossRefPubMed

Perfect, M. M., Turley, M. R., Carlson, J. S., Yohanna, J., & Saint Gilles, M. P. (2016). School-Related outcomes of traumatic event exposure and traumatic stress symptoms in students: a systematic review of Research from 1990 to 2015. School Mental Health, 8(1), 7–43. https://doi.org/10.1007/s12310-016-9175-2.CrossRef

Radhakrishnan, L., Leeb, R. T., Bitsko, R. H., Carey, K., Gates, A., Holland, K. M., Hartnett, K. P., Kite-Powell, A., DeVies, J., Smith, A. R., van Santen, K. L., Crossen, S., Sheppard, M., Wotiz, S., Lane, R. I., Njai, R., Johnson, A. G., Winn, A., & Kirking,. Anderson, K. N. (2022). Pediatric Emergency Department Visits Associated with Mental Health Conditions Before and During the COVID-19 Pandemic — United States, January 2019–January 2022. MMWR Morb Mortal Wkly Rep 2022, 71(8), 319–324. https://doi.org/10.15585/mmwr.mm7108e2

Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor Models and Rotations: exploring the extent to which Multidimensional Data Yield Univocal Scale Scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477.CrossRefPubMedPubMedCentral

Reise, S.P. (2012) The Rediscovery of Bifactor Measurement Models. Multivariate Behavioral Research, 47(5) 667–696 1 https://doi.org/10.1080/00273171.2012.715555

Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013). Multidimensionality and Structural Coefficient Bias in Structural equation modeling: a bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26. https://doi.org/10.1177/0013164412449831.CrossRef

Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045.CrossRefPubMed

von der Romer, N., Eklund, K., Kilgus, S., Perales, K., Splett, J. W., Sudlo, S., & Wheeler, D. (2020). Best Practices in Social, Emotional, and Behavioral Screening: An Implementation Guide. Version 2.0. Retrieved from smhcollaborative.org/universalscreening

Splett, J., Smith-Millman, M., Raborn, A., Brann, K., Flaspohler, P., & Maras, M. (2018) Student teacher and classroom predictors of between-teacher variance of students’ teacher-rated behavior. School Psychology Quarterly 33(3) 460–468 https://doi.org/10.1037/spq0000241CrossRefPubMed

Suldo, S. M., & Shaffer, E. J. (2008). Looking beyond psychopathology: The dual-factor model of Mental Health in Youth. School Psychology Review, 37(1), 52–68. https://doi.org/10.1080/02796015.2008.12087908.CrossRef

Suldo, S. M., & Shaunessy-Dedrick, E. (2013). The psychosocial functioning of high school students in academically rigorous programs. Psychology in the Schools, 50, 823–843. https://doi.org/10.1002/pits.21708.CrossRef

Ten Berge, J. M., & Sočan, G. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613–625. https://doi.org/10.1007/BF02289858.CrossRef

Twenge, J. M., Cooper, A. B., Joiner, T. E., Duffy, M. E., & Binau, S. G. (2019). Age, period, and cohort trends in mood disorder indicators and suicide-related outcomes in a nationally representative dataset, 2005–2017. Journal of Abnormal Psychology, 128(3), 185–199. https://doi.org/10.1037/abn0000410.CrossRefPubMed

van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2012). Agreement of informants on emotional and behavioral problems from childhood to adulthood. Psychological Assessment, 24(2), 293–300. https://doi.org/10.1037/a0025500.CrossRefPubMed

von der Embse, N., Eklund, K., & Kilgus, S. (2022). Conducting behavioral and social-emotional assessments in MTSS: screen to Intervene. Routledge. https://doi.org/10.4324/9780429355790.

von der Embse, N.P., Kilgus, S.P., Iaccarino, S., & Levi-Nielsen, S. (2017). Screening for student mental health risk: Diagnostic accuracy, measurement, invariance, and predictive validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS). School Mental Health, 9,273–283. https://doi.org/10.1007/s12310-017-9214-7.

von der Embse, N., Kim, E., Jenkins, A., Sanchez, A., Kilgus, S. P., & Eklund, K. (2021). Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes. Journal of Psychopathology and Behavioral Assessment, 43(3), 632–645. https://doi.org/10.1007/s10862-021-09869-0.CrossRef

von der Embse, N., Kim, E. S., Kilgus, S., Dedrick, R., & Sanchez, A. (2019). Multi-informant universal screening: evaluation of rater, item, and construct variance using a trifactor model. Journal of School Psychology, 77, 52–66. https://doi.org/10.1016/j.jsp.2019.09.005.CrossRefPubMed

von der Embse, N. P., Pendergast, L. L., Kilgus, S. P., & Eklund, K. R. (2016). Evaluating the applied use of a mental health screener: structural validity of the Social, Academic, and emotional behavior risk screener. Psychological Assessment, 28(10), 1265–1275. https://doi.org/10.1037/pas0000253.CrossRefPubMed

Weir, K. (2020). September). Safeguarding student mental health.Monitor on Psychology, 51(6). http://www.apa.org/monitor/2020/09/safeguarding-mental-health

Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038–1050. https://doi.org/10.1037/0022-006X.68.6.1038.CrossRefPubMed

Zee, M., & Moritz Rudasill, K. (2021). Catching sight of children with internalizing symptoms in upper elementary classrooms. Journal of School Psychology, 87, 1–17. https://doi.org/10.1016/j.jsp.2021.05.002.CrossRefPubMed

Titel: Multi-informant Assessment of Internalizing Concerns: Rater Concordance and Implications for Decision-Making
Auteurs: Nathaniel von der Embse
Eunsook Kim
Dorie Ross
Stephen Kilgus
Thomas Koza
Publicatiedatum: 01-02-2023
Uitgeverij: Springer US
Gepubliceerd in: Journal of Psychopathology and Behavioral Assessment / Uitgave 1/2023
Print ISSN: 0882-2689
Elektronisch ISSN: 1573-3505
DOI: https://doi.org/10.1007/s10862-023-10026-y

Bohn Stafleu van Loghum

Deel dit onderdeel of sectie (kopieer de link)

Abstract

Log in om toegang te krijgen

Andere artikelen Uitgave 1/2023

Deficits in Parent Knowledge of Behavior Management Skills is Strongly Associated With CD Symptoms but not ODD Symptoms

Narcissus Going Public: Pathological Narcissism and Reactions to Public vs. Private Exposure in Ego-relevant Events

Repetitive Negative Thinking Explains the Relationship Between Perceived Attentional Control and Generalized Anxiety Symptoms

Is the Youth Self-Report Total Score a Reliable Measure of both a General Factor of Psychopathology and Achenbach’s Eight Syndromes? A cross-cultural Study

The Development and Validation of the Firesetting Questionnaire

The Dynamic Interplay of Anxious and Depressive Symptoms in a Sample of Undergraduate Students