Abstract
Programs delivered in the “real world” often look substantially different from what was originally intended by program developers. Depending on which components of a program are being trimmed or altered, such modifications may seriously undermine the effectiveness of a program. In the present study, these issues are explored within a widely used school-based, non-curricular intervention, Positive Behavioral Intervention and Supports. The present study takes advantage of a uniquely large dataset to gain a better understanding of the “real-world” implementation quality of PBIS and to take a first step toward identifying the components of PBIS that “matter most” for student outcomes. Data from 27,689 students and 166 public primary and secondary schools across seven states included school and student demographics, indices of PBIS implementation quality, and reports of problem behaviors for any student who received an office discipline referral during the 2007–2008 school year. Results of the present study identify three key components of PBIS that many schools are failing to implement properly, three program components that were most related to lower rates of problem behavior (i.e., three “active ingredients” of PBIS), and several school characteristics that help to account for differences across schools in the quality of PBIS implementation. Overall, findings highlight the importance of assessing implementation quality in “real-world” settings, and the need to continue improving understanding of how and why programs work. Findings are discussed in terms of their implications for policy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Barber, B. K., & Olsen, J. A. (2004). Assessing the transitions to middle and high school. Journal of Adolescent Research, 19, 3–30. doi:10.1177/0743558403258113.
Becker, K. D., & Domitrovich, C. E. (2011). The conceptualization, integration, and support of evidence-based interventions in the schools. School Psychology Review, 40, 582–589.
Bethea, K. R. (2012). A cross-case analysis of the implementation and impact of smaller learning communities in selected S.C. public middle schools. Retrieved from ProQuest Information & Learning. (AAI3454674)
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 1114–1119. doi:10.1097/CHI.0b013e3181825af8.
Botvin, G. J., Eng, A., & Williams, C. L. (1980). Preventing the onset of cigarette smoking through life skills training. Preventive Medicine, 9, 135–143. doi:10.1016/0091-7435(80)90064-x.
Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide Positive Behavioral Interventions and Supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115. doi:10.1007/s11121-008-0114-9.
Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133–148. doi:10.1177/1098300709334798.
Brown, L. D., Feinberg, M. E., & Greenberg, M. T. (2010). Determinants of community coalition ability to support evidence-based programs. Prevention Science, 11, 287–297. doi:10.1007/s11121-010-0173-6.
Cane, J., O’Connor, D., & Michie, S. (2012). Validation of the theoretical domains framework for use in behaviour change and implementation research. Implementation Science, 7. doi: 10.1186/1748-5908-7-37.
Center for the Study and Prevention of Violence (n.d.). Blueprints for violence prevention. Retrieved from http://www.colorado.edu/cspv/blueprints/index.html
Clonan, S. M., McDougal, J. L., Clark, K., & Davison, S. (2007). Use of office discipline referrals in school-wide decision making: A practical example. Psychology in the Schools, 44, 19–27. doi:10.1002/pits.20202.
Dariotis, J. K., Bumbarger, B. K., Duncan, L. G., & Greenberg, M. T. (2008). How do implementation efforts relate to program adherence? Examining the role of organizational, implementer, and program factors. Journal of Community Psychology, 36, 744–760. doi:10.1002/jcop.20255.
Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. doi:10.1207/s1532768xjepc1102_04.
Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28. doi:10.1080/1754730x.2008.9715730.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.
Elias, M. J., Zins, J. E., Graczyk, P. A., & Weissburg, R. P. (2003). Implementation, sustainability, and scaling up of social-emotional and academic innovations in public schools. School Psychology Review, 32, 303–319.
Fagan, A. A., Hanson, K., Briney, J. S., & Hawkins, J. D. (2012). Sustaining the utilization and high quality implementation of tested and effective prevention programs using the communities that care prevention system. American Journal of Community Psychology, 49, 365–377. doi:10.1007/s10464-011-9463-9.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida.
Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39, 3–35. doi:10.1177/002242780203900101.
Greenberg, M. T. (2010). School-based prevention: Current status and future challenges. Effective Education, 2, 27–52. doi:10.1080/19415531003616862.
Harms, A. L. S. (2011). A three-tier model of integrated behavior and learning supports: Linking system-wide implementation to student outcomes. Retrieved from ProQuest Information & Learning. (AAI3433110)
Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The School-Wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12. doi:10.1177/10983007040060010201.
Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., et al. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144. doi:10.1177/1098300709332067.
Irvin, L. K., Tobin, T. J., Sprague, J. R., Sugai, G., & Vincent, C. G. (2004). Validity of office discipline referral measures as indices of school-wide behavioral status and effects of school-wide behavioral interventions. Journal of Positive Behavior Interventions, 6, 131–147. doi:10.1177/10983007040060030201.
Lipsey, M. W. (2009). The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview. Victims & Offenders, 4, 124–147. doi:10.1080/15564880802612573.
Michie, S., Fixsen, D., Grimshaw, J., & Eccles, M. (2009). Specifying and reporting complex behaviour change interventions: The need for a scientific method. Implementation Science, 4. doi: 10.1186/1748-5908-4-40
Michie, S., van Stralen, M. M., & West, R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6. doi:10.1186/1748-5908-6-42.
Olweus, D. (1991). Bully/victim problems among schoolchildren: Basic facts and effects of a school based intervention program. In K. H. Rubin & D. J. Pepler (Eds.), The development and treatment of childhood aggression (pp. 411–448). Hillsdale: Erlbaum.
OSEP Positive Behavioral Interventions and Supports. (2012). What is school-wide PBIS? Retrieved from http://www.pbis.org/school/default.aspx
Pas, E. T., Bradshaw, C. P., & Mitchell, M. M. (2011). Examining the validity of office discipline referrals as an indicator of student behavior problems. Psychology in the Schools, 48, 541–555. doi:10.1002/pits.20577.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Newbury Park: Sage.
Spaulding, S., Horner, R., May, S., & Vincent, C. (2008, November). Evaluation brief: Implementation of school-wide PBS across the United States. Retrieved from http://www.pbis.org/evaluation/evaluation_briefs/nov_08_(2).aspx
Sugai, G., Sprague, J. R., Horner, R. H., & Walker, H. M. (2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8, 94–101. doi:10.1177/106342660000800205.
Sugai, G., Lewis-Palmer, Todd, A. W., & Homer, R. H. (2001). School-wide Evaluation Tool (version 2.0). Eugene, OR: Educational and Community Supports, University of Oregon.
Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based interventions under real-world conditions: Results from a large-scale diffusion project. Prevention Science, 11, 252–262. doi:10.1007/s11121-010-0170-9.
Wehby, J. H., Maggin, D. M., Moore Partin, T. C., & Robertson, R. (2012). The impact of working alliance, social validity, and teacher burnout on implementation fidelity of the Good Behavior Game. School Mental Health, 4, 22–33. doi:10.1007/s12310-011-9067-4.
Author Note
Data for this study were obtained from the OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. This research was supported by the Office of Special Education Programs US Department of Education (H326S980003). Opinions expressed herein are those of the authors and do not necessarily reflect the position of the US Department of Education, and such endorsements should not be inferred.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Molloy, L.E., Moore, J.E., Trail, J. et al. Understanding Real-World Implementation Quality and “Active Ingredients” of PBIS. Prev Sci 14, 593–605 (2013). https://doi.org/10.1007/s11121-012-0343-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-012-0343-9