Skip to main content

Advertisement

SpringerLink
Log in

Is Implementation Fidelity Important? A Systematic Review on School-Based Mental Health Programs

  • Systematic Review
  • Published:
Contemporary School Psychology Aims and scope Submit manuscript

Abstract

Although implementation fidelity (IF) is an important factor in interpreting the outcomes of intervention programs, so far there is little knowledge about how it actually relates to them. The purpose of this systematic review was to identify which component of IF (adherence, quality of the intervention, exposure to the intervention and receptiveness) is the most important one for attaining the expected results in school-based mental health programs. A search in four electronic databases (APA PsycNET, PUBMED, EBSCO, and ISI-WEB Science) yielded 31 articles published between 2006 and 2016 that met all the established inclusion criteria. To determine the associations present, the proportion was calculated between the number of times that the components of IF were significantly linked to outcomes and the total number of times that the association was evaluated. It was observed that the various components of IF are linked to outcomes 40% of the time and that the strongest association is established with students’ exposure and receptiveness to the intervention. Lastly, findings and their implications for future research are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

References marked with an asterisk indicate studies included in the systematic review

  • *Abry, T., Hulleman, C.S., & Rimm-Kaufman, S.E. (2015). Using indices of fidelity to intervention core components to identify program active ingredients. American Journal of Evaluation, 36(3), 320–338. https://doi.org/10.1177/1098214014557009.

    Article  Google Scholar 

  • *Askell-williams, H., Dix, K.L., Lawson, M.J., & Slee, P.T. (2013). Quality of implementation of a school mental health initiative and changes over time in students’ social and emotional competencies. School Effectiveness and School Improvement, 24(3), 357–381. https://doi.org/10.1080/09243453.2012.692697.

    Article  Google Scholar 

  • *Barlow, A., Humphrey, N., Lendrum, A., Wigelsworth, M., & Squires, G. (2015). Evaluation of the implementation and impact of an integrated prevention model on the academic progress of students with disabilities. Research in Developmental Disabilities, 36, 505–525. https://doi.org/10.1016/j.ridd.2014.10.029.

    Article  Google Scholar 

  • *Berry, V., Axford, N., Blower, S., Taylor, R.S., Edwards, R.T., Tobin, K., … Bywater, T. (2015). The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: A cluster-randomised controlled trial. School Mental Health, 8(2), 1–19. https://doi.org/10.1007/s12310-015-9160-1.

    Article  Google Scholar 

  • Bhattacharyya, O., Reeves, S., & Zwarenstein, M. (2009). What is implementation research?: Rationale, concepts, and practices. Research on Social Work Practice, 19(5), 491–502. https://doi.org/10.1177/1049731509335528

    Article  Google Scholar 

  • Bruhn, A. L., Hirsch, S. E., & Lloyd, J. W. (2015). Treatment Integrity in School-Wide Programs: A Review of the Literature (1993-2012). Journal of Primary Prevention, 36(5), 335–349. https://doi.org/10.1007/s10935-015-0400-9

    Article  PubMed  Google Scholar 

  • *Burke, R.V., Oats, R.G., Ringle, J.L., Fichtner, L.O., & DelGaudio, M.B. (2011). Implementation of a classroom management program with urban elementary schools in low-income Neighborhoods: Does program Fidelity affect student behavior and academic outcomes? Journal of Education for Students Placed at Risk, 16(3), 201–218. https://doi.org/10.1080/10824669.2011.585944.

    Article  Google Scholar 

  • *Clarke, A.M., Bunting, B., & Barry, M.M. (2014). Evaluating the implementation of a school-based emotional well-being programme: A cluster randomized controlled trial of Zippy’s friends for children in disadvantaged primary schools. Health Education Research, 29(5), 786–98. https://doi.org/10.1093/her/cyu047.

    Article  PubMed  Google Scholar 

  • Codding, R. S., & Lane, K. L. (2015). A spotlight on treatment intensity: An important and often overlooked component of intervention inquiry. Journal of Behavioral Education, 24(1), 1–10. https://doi.org/10.1007/s10864-014-9210-z

    Article  Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/S0272-7358(97)00043-3

    Article  PubMed  Google Scholar 

  • DiGennaro Reed, F. D., & Codding, R. S. (2014). Advancements in procedural Fidelity assessment and intervention: Introduction to the special issue. Journal of Behavioral Education, 23(1), 1–18. https://doi.org/10.1007/s10864-013-9191-3

    Article  Google Scholar 

  • *Dix, K.L., Slee, P.T., Lawson, M.J., & Keeves, J.P. (2012). Implementation quality of whole-school mental health promotion and students’ academic performance. Child and Adolescent Mental Health, 17(1), 45–51. https://doi.org/10.1111/j.1475-3588.2011.00608.x.

    Article  Google Scholar 

  • *Domitrovich, C.E., Gest, S.D., Jones, D., Gill, S., & DeRousie, R.M.S. (2010). Implementation quality: Lessons learned in the context of the head start REDI trial. Early Childhood Research Quarterly, 25(3), 284–298. https://doi.org/10.1016/j.ecresq.2010.04.001.

    Article  PubMed  PubMed Central  Google Scholar 

  • DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta analytic review. American Journal of Community Psychology, 30(2), 157–198. https://doi.org/10.1023/A:1014628810714

    Article  PubMed  Google Scholar 

  • Dupaul, G. J. (2009). Assessing integrity of intervention Implementation : Critical factors and future directions. School Mental Health, 1(3), 154–157. https://doi.org/10.1007/s12310-009-9016-7

    Article  Google Scholar 

  • Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. https://doi.org/10.1007/s11121-015-0606-3

    Article  PubMed  Google Scholar 

  • Durlak, J. A. (2016). Programme implementation in social and emotional learning: Basic issues and research findings. Cambridge Journal of Education, 46(3), 333–345. https://doi.org/10.1080/0305764X.2016.1142504

    Article  Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464-008-9165-0

    Article  PubMed  Google Scholar 

  • Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x

    Article  PubMed  Google Scholar 

  • *Evans, S.W., Schultz, B.K., & Demars, C.E. (2014). High school-based treatment for adolescents with attention-deficit / hyperactivity Disorder: Results from a pilot study examining outcomes and dosage. School Psychology Review, 43(2), 185–202.

  • *Ferrer-Wreder, L., Cadely, H.S.E., Domitrovich, C.E., Small, M.L., Caldwell, L.L., & Cleveland, M.J. (2010). Is more better? Outcome and dose of a universal drug prevention effectiveness trial. Journal of Primary Prevention, 31(5–6), 349–363. https://doi.org/10.1007/s10935-010-0226-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • *Flannery, K.B., Fenning, P., Kato, M.M., & McIntosh, K. (2014). Effects of school-wide positive Behavioral interventions and supports and Fidelity of implementation on problem behavior in high schools. School Psychology Quarterly, 29(2), 111–124. https://doi.org/10.1037/spq0000039.

    Article  PubMed  Google Scholar 

  • Forman, S.G., Shapiro, E.S., Codding, R.S., Gonzales, J.E., Reddy, L.A., Rosenfield, S.A., ... & Stoiber, K.C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77–100. https://doi.org/10.1037/spq0000019.

    Article  PubMed  Google Scholar 

  • Gresham, F. M. (2009). Evolution of the treatment integrity concept: Current status and future directions. School Psychology Review, 38(4), 533–540.

    Google Scholar 

  • Griffith, A. K., Duppong Hurley, K., & Hagaman, J. L. (2009). Treatment Integrity of Literacy Interventions for Students With Emotional and/or Behavioral Disorders: A Review of Literature. Remedial and Special Education, 30(4), 245–255. https://doi.org/10.1177/0741932508321013

    Article  Google Scholar 

  • *Haataja, A., Voeten, M., Boulton, A.J., Ahtola, A., Poskiparta, E., & Salmivalli, C. (2014). The KiVa antibullying curriculum and outcome: Does fidelity matter? Journal of School Psychology, 52(5), 479–493. https://doi.org/10.1016/j.jsp.2014.07.001.

    Article  PubMed  Google Scholar 

  • Hagermoser-Sanetti, L., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.

    Google Scholar 

  • Hagermoser-Sanetti, L., Gritter, K., & Dobey, L. (2011). Treatment integrity of interventions with children in the journal of positive behavior interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29–46. https://doi.org/10.1177/1098300711405853

    Article  Google Scholar 

  • *Hamre, B.K., Pianta, R.C., Mashburn, A.J., & Downer, J.T. (2012). Promoting young children’s social competence through the preschool PATHS curriculum and MyTeachingPartner professional development resources. Early Education & Development, 23(6), 809–832. https://doi.org/10.1080/10409289.2011.607360.

    Article  Google Scholar 

  • *Hirschstein, M.K., Edstrom, L.V.S., Frey, K.S., Snell, J.L., & MacKenzie, E.P. (2007). Walking the talk in bullying prevention: Teacher implementation variables related to initial impact of the steps to respect program. School Psychology Review, 36(1), 3–21.

  • *Humphrey, N., Lendrum, A., Barlow, A., Wigelsworth, M., & Squires, G. (2013). Achievement for all: Improving psychosocial outcomes for students with special educational needs and disabilities. Research in Developmental Disabilities, 34(4), 1210–1225. https://doi.org/10.1016/j.ridd.2012.12.008.

    Article  PubMed  Google Scholar 

  • *Kiviruusu, O., Bjorklund, K., Koskinen, H., Liski, A., Lindblom, J., Kuoppamaki, H., … Santalahti, P. (2016). Short-term effects of the “together at school” intervention program on children’s socio-emotional skills: A cluster randomized controlled trial. BMC Psychology, 4(27).

  • Kutash, K., Duchnowski, A. J., & Lynn, N. (2006). School-based mental health: An empirical guide for decision-makers. Tampa: University of South Florida, The Louis De La Parte Florida Mental Health Institute, Department of Child and Family Studies, Research and Training Center for Children's Mental Health.

    Google Scholar 

  • Lewis, C.C., Fischer, S., Weiner, B.J., Stanick, C., Kim, M., & Martinez, R.G. (2015). Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science, 10. https://doi.org/10.1186/s13012-015-0342-x.

  • *Low, S., Cook, C.R., Smolkowski, K., & Buntain-Ricklefs, J. (2014a). Promoting social-emotional competence: An evaluation of the elementary version of second step®. Journal of School Psychology. https://doi.org/10.1016/j.jsp.2015.09.002.

    Article  PubMed  Google Scholar 

  • *Low, S., Van Ryzin, M.J., Brown, E.C., Smith, B.H., & Haggerty, K.P. (2014b). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15(2), 165–176. https://doi.org/10.1007/s11121-012-0359-1.

    Article  Google Scholar 

  • *Low, S., Smolkowski, K., & Cook, C. (2016). What constitutes high-quality implementation of SEL programs? A latent class analysis of second step® implementation. Prevention Science, 17(8), 981–991. https://doi.org/10.1007/s11121-016-0670-3.

    Article  PubMed  Google Scholar 

  • Macklem, G. (2014). Preventive mental health at school. Evidence-based services for students. New York: Springer.

    Book  Google Scholar 

  • McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005. Journal of Applied Behavior Analysis, 40(4), 659–72. https://doi.org/10.1901/jaba.2007.659

  • McGinty, A. S., Breit-Smith, A., Fan, X., Justice, L. M., & Kaderavek, J. N. (2011). Does intensity matter? Preschoolers’ print knowledge development within a classroom-based intervention. Early Childhood Research Quarterly, 26(3), 255–267. https://doi.org/10.1016/j.ecresq.2011.02.002

    Article  Google Scholar 

  • *Obsuth, I., Cope, A., Sutherland, A., Pilbeam, L., Murray, A.L., & Eisner, M. (2016). London education and inclusion project (LEIP): Exploring negative and null effects of a cluster-randomised school-intervention to reduce school exclusion-findings from protocol-based subgroup analyses. PLoS One, 11(4), 1–21. https://doi.org/10.1371/journal.pone.0152423.

    Article  PubMed  PubMed Central  Google Scholar 

  • *Pas, E.T., & Bradshaw, C.P. (2012). Examining the association between implementation and outcomes: State-wide scale-up of school-wide positive behavior intervention and supports. Journal of Behavioral Health Services and Research, 39(4), 417–433. https://doi.org/10.1007/s11414-012-9290-2.

    Article  Google Scholar 

  • Perepletchikova, F. (2011). On the Topic of Treatment Integrity. Clinical Psychology: Science and Practice, 18(2), 148–153. https://doi.org/10.1111/j.1468-2850.2011.01246.x

    Article  Google Scholar 

  • Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75(6), 829–841. https://doi.org/10.1037/0022-006X.75.6.829

    Article  PubMed  Google Scholar 

  • *Pettigrew, J., Graham, J.W., Miller-Day, M., Hecht, M.L., Krieger, J.L., & Shin, Y.J. (2015). Adherence and delivery: Implementation quality and program outcomes for the seventh-grade keepin’it REAL program. Prevention Science, 16(1), 90–99. https://doi.org/10.1007/s11121-014-0459-1.

    Article  Google Scholar 

  • Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34. https://doi.org/10.1007/s10488-008-0197-4

    Article  PubMed  Google Scholar 

  • *Reyes, M.R., Brackett, M. A, Rivers, S. E., Elbertson, N. A, & Salovey, P. (2012). The interaction effects of program training, dosage, and implementation quality on targeted student outcomes for the RULER approach to social and emotional learning. School Psychology Review, 41(1), 82–99.

  • *Rimm-Kaufman, S.E., Larsen, R.A.A., Baroody, A.E., Curby, T.W., Ko, M., Thomas, J.B., … DeCoster, J. (2014). Efficacy of the responsive classroom approach: Results from a 3-year, longitudinal randomized controlled trial. American Educational Research Journal, 51(3), 567–603. https://doi.org/10.3102/0002831214523821.

    Article  Google Scholar 

  • *Rosenblatt, J.L., & Elias, M.J. (2008). Dosage effects of a preventive social-emotional learning intervention on achievement loss associated with middle school transition. Journal of Primary Prevention, 29(6), 535–555. https://doi.org/10.1007/s10935-008-0153-9.

    Article  PubMed  Google Scholar 

  • Sarno, J., Lyon, A. R., Brandt, N. E., Warner, C. M., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation science in school mental health: Key constructs in a developing research agenda. School Mental Health, 6(2), 99–111. https://doi.org/10.1007/s12310-013-9115-3

    Article  Google Scholar 

  • Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25(1), 146–156. https://doi.org/10.1037/a0029715.A

    Article  PubMed  Google Scholar 

  • Schulte, A. C., Easton, J. E., & Parker, J. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38(4), 541–546.

    Google Scholar 

  • Schultes, M. T., Jöstl, G., Finsterwald, M., Schober, B., & Spiel, C. (2015). Measuring intervention fidelity from different perspectives with multiple methods: The reflect program as an example. Studies in Educational Evaluation, 47, 102–112. https://doi.org/10.1016/j.stueduc.2015.10.001

    Article  Google Scholar 

  • Sklad, M., Diekstra, R., De Ritter, M., Ben, J., & Gravesteijn, C. (2012). Effectiveness of school-based universal social, emotional, and behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment? Psychology in the Schools, 49(9), 892–909. https://doi.org/10.1002/pits.21641

    Article  Google Scholar 

  • Smith, J. D., Schneider, B. H., Smith, P. K., & Ananiadou, K. (2004). The effectiveness of whole-school antibullying programs: A synthesis of evaluation research. School Psychology Review, 33(4), 547–560.

    Google Scholar 

  • *Smokowski, P.R., Guo, S., Wu, Q., Evans, C.B., Cotter, K.L., & Bacallao, M. (2016). Evaluating dosage effects for the positive action program: How implementation impacts internalizing symptoms, aggression, school hassles, and self-esteem. American Journal of Orthopsychiatry, 86(3), 310. https://doi.org/10.1037/ort0000167.

    Article  PubMed  Google Scholar 

  • *Sørlie, M.-A., & Ogden, T. (2007). Immediate impacts of PALS: A school-wide multi-level programme targeting behaviour problems in elementary school. Scandinavian Journal of Educational Research, 51(5), 471–492. https://doi.org/10.1080/00313830701576581.

    Article  Google Scholar 

  • *Sørlie, M. A., & Ogden, T. (2015). School-wide positive behavior support–Norway: Impacts on problem behavior and classroom climate. International Journal of School & Educational Psychology, 3(3), 202–217. https://doi.org/10.1080/21683603.2015.1060912.

    Article  Google Scholar 

  • Suldo, S. M., Gormley, M. J., DuPaul, G. J., & Anderson-Butcher, D. (2014). The impact of school mental health on student and school-level academic outcomes: Current status of the research and future directions. School Mental Health, 6(2), 84–98. https://doi.org/10.1007/s12310-013-9116-2

    Article  Google Scholar 

  • Sutherland, K. S., McLeod, B. D., Conroy, M. a., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/Behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35, 129–149. https://doi.org/10.1177/1053815113515025

    Article  Google Scholar 

  • Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16(4), 537–567. https://doi.org/10.1177/002204268601600405

    Article  Google Scholar 

  • Weare, K., & Nind, M. (2011). Mental health promotion and problem prevention in schools: What does the evidence say? Health Promotion International, 26(SUPL. 1). https://doi.org/10.1093/heapro/dar075.

    Article  PubMed  Google Scholar 

  • Weist, M., & Lever, N. (2014). Further advancing the field of school mental health. In M. Weist, N. Lever, C. P. Bradshaw, & J. S. Owens (Eds.), Handbook of school Mental Health. Research, training, practice and Police (2nd ed., pp. 1–16). New York: Springer.

    Chapter  Google Scholar 

  • Wheeler, J. J., Mayton, M. R., Ton, J., & Reese, J. E. (2014). Evaluating treatment integrity across interventions aimed at social and emotional skill development in learners with emotional and behaviour disorders. Journal of Research in Special Educational Needs, 14(3), 164–169. https://doi.org/10.1111/j.1471-3802.2011.01229.x

    Article  Google Scholar 

  • *Wigelsworth, M., Humphrey, N., & Lendrum, A. (2013a). Evaluation of a school-wide preventive intervention for adolescents: The secondary social and emotional aspects of learning (SEAL) programme. School Mental Health, 5(2), 96–109. https://doi.org/10.1007/s12310-012-9085-x.

    Article  Google Scholar 

  • *Wigelsworth, M., Lendrum, A., & Humphrey, N. (2013b). Assessing differential effects of implementation quality and risk status in a whole-school social and emotional learning programme: Secondary SEAL. Mental Health and Prevention, 1(1), 11–18. https://doi.org/10.1016/j.mhp.2013.06.001.

    Article  Google Scholar 

  • Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136–149. https://doi.org/10.1037/0022-006X.71.1.136

    Article  PubMed  Google Scholar 

  • *Zhai, F.H., Raver, C.C., & Jones, S.M. (2015). Social and emotional learning services and child outcomes in third grade: Evidence from a cohort of head start participants. Children and Youth Services Review, 56, 42–51.

Download references

Funding

This study was funded by National Fund for Scientific and Technological Development [Grant number 1171634, 2017]; Postgraduate studies funded by CONICYT-PCHA / PhD National / 2015-21150612.

Author information

Authors and Affiliations

  1. School of Psychology, Faculty of Social Sciences, University Academy of Christian Humanism, Condell 343, Providencia, Santiago, Chile

    Rodrigo Rojas-Andrade

  2. Department of Psychology, Faculty of Social Sciences, University of Chile, Capitan Ignacio Carrera Pinto 1045, Ñuñoa, Santiago, Chile

    Rodrigo Rojas-Andrade & Loreto Leiva Bahamondes

Authors
  1. Rodrigo Rojas-Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Loreto Leiva Bahamondes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rodrigo Rojas-Andrade.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.

Conflict of Interest

Author A has received research grants from Postgraduate studies funded by CONICYT-PCHA / PhD National / 2015-21,150,612. Author B has received research grants from National Fund for Scientific and Technological Development [Grant number 1171634, 2017].

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rojas-Andrade, R., Bahamondes, L.L. Is Implementation Fidelity Important? A Systematic Review on School-Based Mental Health Programs. Contemp School Psychol 23, 339–350 (2019). https://doi.org/10.1007/s40688-018-0175-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40688-018-0175-0

Keywords

Search

Navigation