Skip to main content
Top
Gepubliceerd in: Journal of Child and Family Studies 4/2017

09-12-2016 | Original Paper

In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior

Auteurs: Valerie B. Shapiro, Sarah Accomazzo, Jennifer L. Robitaille

Gepubliceerd in: Journal of Child and Family Studies | Uitgave 4/2017

Log in om toegang te krijgen
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail

Abstract

There is a lack of psychometrically sound tools for measuring youth outcomes in out-of-school time (OST) settings. Consequently, behavior ratings completed by OST staff are being scored as though the raters were teachers, even though cross-informant correlations are notoriously low (meta-analysis r = .27). Across 26 schools, 227 students were assessed by both teachers and OST staff using the Devereux Student Strengths Assessment (DESSA) to measure Social Emotional Competence. These 4th and 5th grade students were 50% male; 53% 5th graders; and 51% Latino, 20% Mixed/Other, 11% Black, 11% Asian, and 7% White. In the full sample, OST staff rated children’s behavior more harshly than teachers (p < .001; d = .32), although the scores were associated (r = .31, p < .001). Among the ratings completed within the same week, teacher and staff distributions were not statistically different. Teacher and staff ratings had a “medium” correlation (r = .42; p = .01) and a classification consistency (88%) that exceeded chance by a “moderate” amount (κ = .43). Few, if any, studies have previously compared the ratings of the same children by teachers and OST providers. Cross-informant inter-rater reliability between teachers and OST staff was higher than expected on the DESSA.
Literatuur
go back to reference Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. CrossRefPubMed Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. CrossRefPubMed
go back to reference Atlas, J. A. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 178–180). Lincoln, NE: Buros Center for Testing. Atlas, J. A. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 178–180). Lincoln, NE: Buros Center for Testing.
go back to reference Chain J., Shapiro, V.B., LeBuffe, P.A., & Bryson, A.M. (2012, August). Indigenous cultural differences in social emotional competence and academic achievement. Poster session presented at the meeting of the Society for the Psychological Study of Ethnic Minority Issues of the American Psychological Association, Orlando, FL. Chain J., Shapiro, V.B., LeBuffe, P.A., & Bryson, A.M. (2012, August). Indigenous cultural differences in social emotional competence and academic achievement. Poster session presented at the meeting of the Society for the Psychological Study of Ethnic Minority Issues of the American Psychological Association, Orlando, FL.
go back to reference Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. CrossRef Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. CrossRef
go back to reference Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Erlbaum. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Erlbaum.
go back to reference Cowen, E. L., Lorion, R. P., & Wilson, A. B. (1976). Knowing, liking, and judged problem severity in relation to referral and outcome measures in a school-based intervention program. Journal of Consulting and Clinical Psychology, 44, 317–328. CrossRefPubMed Cowen, E. L., Lorion, R. P., & Wilson, A. B. (1976). Knowing, liking, and judged problem severity in relation to referral and outcome measures in a school-based intervention program. Journal of Consulting and Clinical Psychology, 44, 317–328. CrossRefPubMed
go back to reference Crane, J., Mincic, M. S., & Winsler, A. (2011). Parent-teacher agreement and reliability on the Devereux early childhood assessment (DECA) in English and Spanish for ethnically diverse children living in poverty. Early Education & Development, 22(3), 520–547. CrossRef Crane, J., Mincic, M. S., & Winsler, A. (2011). Parent-teacher agreement and reliability on the Devereux early childhood assessment (DECA) in English and Spanish for ethnically diverse children living in poverty. Early Education & Development, 22(3), 520–547. CrossRef
go back to reference Denham, S. A., Ji, P., & Hamre, B. (2010). Compendium of preschool through elementary social-emotional learning and associated assessment measures. Chicago, IL: Collaborative for Academic, Social, and Emotional Learning. Denham, S. A., Ji, P., & Hamre, B. (2010). Compendium of preschool through elementary social-emotional learning and associated assessment measures. Chicago, IL: Collaborative for Academic, Social, and Emotional Learning.
go back to reference Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll, W. Phofl, J. Yoon (Eds.), Handbook of youth prevention science (pp. 70–95). New York, NY: Routledge. Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll, W. Phofl, J. Yoon (Eds.), Handbook of youth prevention science (pp. 70–95). New York, NY: Routledge.
go back to reference Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology, 43(6), 543–549. CrossRefPubMed Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology, 43(6), 543–549. CrossRefPubMed
go back to reference Friedman, K. A., Leone, P. E., & Friedman, P. (1999). Strengths-based assessment of children with SED: Consistency of reporting by teachers and parents. Journal of Child and Family Studies, 8(2), 169–180. CrossRef Friedman, K. A., Leone, P. E., & Friedman, P. (1999). Strengths-based assessment of children with SED: Consistency of reporting by teachers and parents. Journal of Child and Family Studies, 8(2), 169–180. CrossRef
go back to reference Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 30–40. CrossRef Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 30–40. CrossRef
go back to reference Gullotta, T. P. (2015). After-school programming and SEL. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice. New York, NY: Guilford. Gullotta, T. P. (2015). After-school programming and SEL. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice. New York, NY: Guilford.
go back to reference Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. The American psychologist, 58(1), 78. CrossRefPubMed Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. The American psychologist, 58(1), 78. CrossRefPubMed
go back to reference Huebner, E. S., Gilman, R., & Furlong, M. J. (2009). A conceptual model for research in positive psychology in children and youth. Handbook of positive psychology in schools. In M. J. Furlong, R. Gilman, E. S. Huebner (Eds.), Handbook of positive psychology in schools (pp. 3–8). New York, NY: Routledge. Huebner, E. S., Gilman, R., & Furlong, M. J. (2009). A conceptual model for research in positive psychology in children and youth. Handbook of positive psychology in schools. In M. J. Furlong, R. Gilman, E. S. Huebner (Eds.), Handbook of positive psychology in schools (pp. 3–8). New York, NY: Routledge.
go back to reference Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: The validity of multi-informant reports. Developmental Medicine & Child Neurology, 56, 453–459. doi: 10.​1111/​dmcn.​12360. CrossRef Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: The validity of multi-informant reports. Developmental Medicine & Child Neurology, 56, 453–459. doi: 10.​1111/​dmcn.​12360. CrossRef
go back to reference LeBuffe, P. A., Shapiro, V. B., & Naglieri, J. A. (2014). The Devereux Student Strengths Assessment (DESSA): Assessment, technical manual, and user’s guide. Charlotte, NC: Apperson, Inc.(Original work published 2009). LeBuffe, P. A., Shapiro, V. B., & Naglieri, J. A. (2014). The Devereux Student Strengths Assessment (DESSA): Assessment, technical manual, and user’s guide. Charlotte, NC: Apperson, Inc.(Original work published 2009).
go back to reference De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), 483. CrossRefPubMed De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), 483. CrossRefPubMed
go back to reference Malcomb, K. K. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.),  The eighteenth mental measurements yearbook (pp. 180–182). Lincoln, NE: Buros Center for Testing. Malcomb, K. K. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.),  The eighteenth mental measurements yearbook (pp. 180–182). Lincoln, NE: Buros Center for Testing.
go back to reference McKown, C. (2015). Challenges and opportunities in the direct assessment of children’s social and emotional comprehension. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp.320–335). New York, NY: Guilford. McKown, C. (2015). Challenges and opportunities in the direct assessment of children’s social and emotional comprehension. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp.320–335). New York, NY: Guilford.
go back to reference Merrell, K. W., & Gueldner, B. A. (2010). Social and emotional learning in the classroom: promoting mental health and academic success. New York, NY: Guilford Press. Merrell, K. W., & Gueldner, B. A. (2010). Social and emotional learning in the classroom: promoting mental health and academic success. New York, NY: Guilford Press.
go back to reference Naglieri, J. A., LeBuffe, P. A., & Shapiro, V. B. (2013). Assessment of social-emotional competencies related to resilience. In S. Goldstein, R. Brooks (Eds.). Handbook of resilience in children (pp. 261–272). New York, NY: Kluwer/Academic Press. CrossRef Naglieri, J. A., LeBuffe, P. A., & Shapiro, V. B. (2013). Assessment of social-emotional competencies related to resilience. In S. Goldstein, R. Brooks (Eds.). Handbook of resilience in children (pp. 261–272). New York, NY: Kluwer/Academic Press. CrossRef
go back to reference Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent validity of the Devereux student strengths assessment. School Psychology Quarterly, 24(1), 48–59. CrossRef Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent validity of the Devereux student strengths assessment. School Psychology Quarterly, 24(1), 48–59. CrossRef
go back to reference Rosas, S., Chaiken, L., & Case, J. (2008). Cross-setting consistent and setting-specific strength behaviors in preschoolers: Influence on reported concerns. In C. Newman, C. Liberton, K. Kutash, R. Friedman (Eds.). A system of care for children’s mental health: Expanding the research base, 20 (pp. 201–204). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health. Rosas, S., Chaiken, L., & Case, J. (2008). Cross-setting consistent and setting-specific strength behaviors in preschoolers: Influence on reported concerns. In C. Newman, C. Liberton, K. Kutash, R. Friedman (Eds.). A system of care for children’s mental health: Expanding the research base, 20 (pp. 201–204). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health.
go back to reference Shapiro, V. B., Accomazzo, S., Claassen, J., & Robitaille, J. L. (2015). The choices, challenges, and lessons learned from a multi-method social-emotional/character assessment in an out of school time setting. Journal of Youth Development: Bridging Research and Practice, 10(3), 32–45. Shapiro, V. B., Accomazzo, S., Claassen, J., & Robitaille, J. L. (2015). The choices, challenges, and lessons learned from a multi-method social-emotional/character assessment in an out of school time setting. Journal of Youth Development: Bridging Research and Practice, 10(3), 32–45.
go back to reference Simmons, C. A., Shapiro, V. B., Accomazzo, S., & Manthey, T. J. (2016). Strengths-based social work: A meta-theory to guide social work research and practice. In N. Coady, P. Lehmann (Eds.), Theoretical perspectives for direct social work practice. 3rd edn. (pp. 131–154). New York, NY: Springer Publishing Company. Simmons, C. A., Shapiro, V. B., Accomazzo, S., & Manthey, T. J. (2016). Strengths-based social work: A meta-theory to guide social work research and practice. In N. Coady, P. Lehmann (Eds.), Theoretical perspectives for direct social work practice. 3rd edn. (pp. 131–154). New York, NY: Springer Publishing Company.
go back to reference Smith, G. T., Shapiro, V. B., Sperry, R. W., & LeBuffe, P. A. (2014). A strengths-based approach to supervised visitation in child welfare. Child Care in Practice, 20(1), 98–119. CrossRef Smith, G. T., Shapiro, V. B., Sperry, R. W., & LeBuffe, P. A. (2014). A strengths-based approach to supervised visitation in child welfare. Child Care in Practice, 20(1), 98–119. CrossRef
go back to reference StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: StataCorp LP. StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: StataCorp LP.
go back to reference Tsang, K. L. V., Wong, P. Y. H., & Lo, S. K. (2012). Assessing psychosocial well-being of adolescents: A systematic review of measurement instruments. Child: Care, Health and Development, 38(35), 629–646. Tsang, K. L. V., Wong, P. Y. H., & Lo, S. K. (2012). Assessing psychosocial well-being of adolescents: A systematic review of measurement instruments. Child: Care, Health and Development, 38(35), 629–646.
go back to reference Wei, C., Hoff, A., Villabø, M. A., Peterman, J., Kendall, P. C., & Piacentini, J., et al. (2014). Assessing anxiety in youth with the multidimensional anxiety scale for children. Journal of Clinical Child & Adolescent Psychology, 43(4), 566–578. CrossRef Wei, C., Hoff, A., Villabø, M. A., Peterman, J., Kendall, P. C., & Piacentini, J., et al. (2014). Assessing anxiety in youth with the multidimensional anxiety scale for children. Journal of Clinical Child & Adolescent Psychology, 43(4), 566–578. CrossRef
go back to reference Wolcott, C. S., & Williford, A. P. (2015). Teacher and TA ratings of preschoolers’ externalizing behavior agreement and associations with observed classroom behavior. Topics in Early Childhood Special Education, 34(4), 211–222. CrossRef Wolcott, C. S., & Williford, A. P. (2015). Teacher and TA ratings of preschoolers’ externalizing behavior agreement and associations with observed classroom behavior. Topics in Early Childhood Special Education, 34(4), 211–222. CrossRef
Metagegevens
Titel
In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior
Auteurs
Valerie B. Shapiro
Sarah Accomazzo
Jennifer L. Robitaille
Publicatiedatum
09-12-2016
Uitgeverij
Springer US
Gepubliceerd in
Journal of Child and Family Studies / Uitgave 4/2017
Print ISSN: 1062-1024
Elektronisch ISSN: 1573-2843
DOI
https://doi.org/10.1007/s10826-016-0632-1

Andere artikelen Uitgave 4/2017

Journal of Child and Family Studies 4/2017 Naar de uitgave