Top

Gepubliceerd in:

09-12-2016 | Original Paper

In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior

Auteurs: Valerie B. Shapiro, Sarah Accomazzo, Jennifer L. Robitaille

Gepubliceerd in: Journal of Child and Family Studies | Uitgave 4/2017

Abstract

There is a lack of psychometrically sound tools for measuring youth outcomes in out-of-school time (OST) settings. Consequently, behavior ratings completed by OST staff are being scored as though the raters were teachers, even though cross-informant correlations are notoriously low (meta-analysis r = .27). Across 26 schools, 227 students were assessed by both teachers and OST staff using the Devereux Student Strengths Assessment (DESSA) to measure Social Emotional Competence. These 4th and 5th grade students were 50% male; 53% 5th graders; and 51% Latino, 20% Mixed/Other, 11% Black, 11% Asian, and 7% White. In the full sample, OST staff rated children’s behavior more harshly than teachers (p < .001; d = .32), although the scores were associated (r = .31, p < .001). Among the ratings completed within the same week, teacher and staff distributions were not statistically different. Teacher and staff ratings had a “medium” correlation (r = .42; p = .01) and a classification consistency (88%) that exceeded chance by a “moderate” amount (κ = .43). Few, if any, studies have previously compared the ratings of the same children by teachers and OST providers. Cross-informant inter-rater reliability between teachers and OST staff was higher than expected on the DESSA.

vorige artikel The Observation of Child Behavior During Parent-Child Interaction: The Psychometric Properties of the Crowell Procedure

volgende artikel Parentification of Adult Siblings of Individuals with Autism Spectrum Disorder

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232.CrossRefPubMed

Afterschool Alliance. (2014). America After 3PM. http://www.afterschoolalliance.org/documents/AA3PM-2014/AA3PM_National_Report.pdf.

Atlas, J. A. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 178–180). Lincoln, NE: Buros Center for Testing.

Chain J., Shapiro, V.B., LeBuffe, P.A., & Bryson, A.M. (2012, August). Indigenous cultural differences in social emotional competence and academic achievement. Poster session presented at the meeting of the Society for the Psychological Study of Ethnic Minority Issues of the American Psychological Association, Orlando, FL.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.CrossRef

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Erlbaum.

Cowen, E. L., Lorion, R. P., & Wilson, A. B. (1976). Knowing, liking, and judged problem severity in relation to referral and outcome measures in a school-based intervention program. Journal of Consulting and Clinical Psychology, 44, 317–328.CrossRefPubMed

Crane, J., Mincic, M. S., & Winsler, A. (2011). Parent-teacher agreement and reliability on the Devereux early childhood assessment (DECA) in English and Spanish for ethnically diverse children living in poverty. Early Education & Development, 22(3), 520–547.CrossRef

Denham, S. A., Ji, P., & Hamre, B. (2010). Compendium of preschool through elementary social-emotional learning and associated assessment measures. Chicago, IL: Collaborative for Academic, Social, and Emotional Learning.

Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll, W. Phofl, J. Yoon (Eds.), Handbook of youth prevention science (pp. 70–95). New York, NY: Routledge.

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology, 43(6), 543–549.CrossRefPubMed

Friedman, K. A., Leone, P. E., & Friedman, P. (1999). Strengths-based assessment of children with SED: Consistency of reporting by teachers and parents. Journal of Child and Family Studies, 8(2), 169–180.CrossRef

Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 30–40.CrossRef

Gullotta, T. P. (2015). After-school programming and SEL. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice. New York, NY: Guilford.

Haggerty, K., Elgin, J., & Woolley, A. (2011). Social-emotional learning and school climate assessment measures for middle school youth. Retrieved from the Raikes Foundation website: http://www.raikesfoundation.org/

Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. The American psychologist, 58(1), 78.CrossRefPubMed

Huebner, E. S., Gilman, R., & Furlong, M. J. (2009). A conceptual model for research in positive psychology in children and youth. Handbook of positive psychology in schools. In M. J. Furlong, R. Gilman, E. S. Huebner (Eds.), Handbook of positive psychology in schools (pp. 3–8). New York, NY: Routledge.

Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: The validity of multi-informant reports. Developmental Medicine & Child Neurology, 56, 453–459. doi:10.1111/dmcn.12360.CrossRef

LeBuffe, P. A., Shapiro, V. B., & Naglieri, J. A. (2014). The Devereux Student Strengths Assessment (DESSA): Assessment, technical manual, and user’s guide. Charlotte, NC: Apperson, Inc.(Original work published 2009).

De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), 483.CrossRefPubMed

Malcomb, K. K. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 180–182). Lincoln, NE: Buros Center for Testing.

McKown, C. (2015). Challenges and opportunities in the direct assessment of children’s social and emotional comprehension. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp.320–335). New York, NY: Guilford.

Merrell, K. W., & Gueldner, B. A. (2010). Social and emotional learning in the classroom: promoting mental health and academic success. New York, NY: Guilford Press.

Naglieri, J. A., LeBuffe, P. A., & Shapiro, V. B. (2013). Assessment of social-emotional competencies related to resilience. In S. Goldstein, R. Brooks (Eds.). Handbook of resilience in children (pp. 261–272). New York, NY: Kluwer/Academic Press.CrossRef

Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent validity of the Devereux student strengths assessment. School Psychology Quarterly, 24(1), 48–59.CrossRef

Rosas, S., Chaiken, L., & Case, J. (2008). Cross-setting consistent and setting-specific strength behaviors in preschoolers: Influence on reported concerns. In C. Newman, C. Liberton, K. Kutash, R. Friedman (Eds.). A system of care for children’s mental health: Expanding the research base, 20 (pp. 201–204). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health.

Shapiro, V. B., Accomazzo, S., Claassen, J., & Robitaille, J. L. (2015). The choices, challenges, and lessons learned from a multi-method social-emotional/character assessment in an out of school time setting. Journal of Youth Development: Bridging Research and Practice, 10(3), 32–45.

Shapiro, V.B., Kim, B.K.E., Robitaille, J.L., & LeBuffe, P.A. (2016). Protective factor screening for prevention practice: Sensitivity and specificity of the DESSA-Mini. School Psychology Quarterly. Advance online publication. http://dx.doi.org/10.1037/spq0000181.

Shapiro, V. B., & LeBuffe, P. A. (2006). Using protective factors in practice. Annals of the New York Academy of Sciences, 1094, 350–353. doi:10.1196/annals.1376.048.CrossRefPubMed

Simmons, C. A., Shapiro, V. B., Accomazzo, S., & Manthey, T. J. (2016). Strengths-based social work: A meta-theory to guide social work research and practice. In N. Coady, P. Lehmann (Eds.), Theoretical perspectives for direct social work practice. 3rd edn. (pp. 131–154). New York, NY: Springer Publishing Company.

Smith, G. T., Shapiro, V. B., Sperry, R. W., & LeBuffe, P. A. (2014). A strengths-based approach to supervised visitation in child welfare. Child Care in Practice, 20(1), 98–119.CrossRef

StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: StataCorp LP.

Tsang, K. L. V., Wong, P. Y. H., & Lo, S. K. (2012). Assessing psychosocial well-being of adolescents: A systematic review of measurement instruments. Child: Care, Health and Development, 38(35), 629–646.

Wei, C., Hoff, A., Villabø, M. A., Peterman, J., Kendall, P. C., & Piacentini, J., et al. (2014). Assessing anxiety in youth with the multidimensional anxiety scale for children. Journal of Clinical Child & Adolescent Psychology, 43(4), 566–578.CrossRef

Wolcott, C. S., & Williford, A. P. (2015). Teacher and TA ratings of preschoolers’ externalizing behavior agreement and associations with observed classroom behavior. Topics in Early Childhood Special Education, 34(4), 211–222.CrossRef

Titel: In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior
Auteurs: Valerie B. Shapiro
Sarah Accomazzo
Jennifer L. Robitaille
Publicatiedatum: 09-12-2016
Uitgeverij: Springer US
Gepubliceerd in: Journal of Child and Family Studies / Uitgave 4/2017
Print ISSN: 1062-1024
Elektronisch ISSN: 1573-2843
DOI: https://doi.org/10.1007/s10826-016-0632-1

Andere artikelen Uitgave 4/2017

School Aged Children’s Experiences 7 and 13 Months Following a Sibling’s Death

Original Paper

Shyness, Unsociability, and Socio-Emotional Functioning at Preschool: The Protective Role of Peer Acceptance

Original Paper

Moral Disengagement and the Propensity to Endorse Physical Punishment Practices

Original Paper

An Investigation of Factors that Influence Acceptability of Parent Training

Original Paper

Authoritative Parenting of Chinese Mothers of Children with and without Intellectual Disability

Original Paper

Barriers to Adherence to Child Assessment Recommendations

Original Paper