Skip to main content

Advertisement

SpringerLink
Log in

Adherence and Delivery: Implementation Quality and Program Outcomes for the Seventh-Grade keepin’ it REAL Program

  • Published:
Prevention Science Aims and scope Submit manuscript

An Erratum to this article was published on 02 October 2014

Abstract

Poor implementation quality (IQ) is known to reduce program effects making it important to consider IQ for evaluation and dissemination of prevention programs. However, less is known about the ways specific implementation variables relate to outcomes. In this study, two versions of keepin’ it REAL, a seventh-grade drug prevention intervention, were implemented in 78 classrooms in 25 schools in rural districts in Pennsylvania and Ohio. IQ was measured through observational coding of 276 videos. IQ variables included adherence to the curriculum, teacher engagement (attentiveness, enthusiasm, seriousness, clarity, positivity), student engagement (attention, participation), and a global rating of teacher delivery quality. Factor analysis showed that teacher engagement, student engagement, and delivery quality formed one factor, which was labeled delivery. A second factor was adherence to the curriculum. Self-report student surveys measured substance use, norms (beliefs about prevalence and acceptability of use), and efficacy (beliefs about one’s ability to refuse substance offers) at two waves (pretest, immediate posttest). Mixed model regression analysis which accounted for missing data and controlled for pretest levels examined implementation quality’s effects on individual level outcomes, statistically controlling for cluster level effects. Results show that when implemented well, students show positive outcomes compared to students receiving a poorly implemented program. Delivery significantly influenced substance use and norms, but not efficacy. Adherence marginally significantly predicted use and significantly predicted norms, but not efficacy. Findings underscore the importance of comprehensively measuring and accounting for IQ, particularly delivery, when evaluating prevention interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alberts, J. K., Miller-Rassulo, M., & Hecht, M. L. (1991). A typology of drug resistance strategies. Journal of Applied Communication Research, 19, 129–151. doi:10.1080/00909889109365299.

    Article  Google Scholar 

  • Arthur, M. W., Hawkins, J. D., Pollard, J. A., Catalano, R. F., & Baglioni, A. J. (2002). Measuring risk and protective factors for use, delinquency, and other adolescent problem behaviors the communities that care youth survey. Evaluation Review, 26, 575–601. doi:10.1177/0193841X0202600601.

    PubMed  Google Scholar 

  • Bandura, A., & Wood, R. E. (1989). Effect of perceived controllability and performance standards on self-regulation of complex decision making. Journal of Personality and Social Psychology, 56, 805–814.

    Article  CAS  PubMed  Google Scholar 

  • Berkel, C., Mauricio, A., Schoenfelder, E., & Sandler, I. (2011). Putting the pieces together: an integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106–1112.

    Article  CAS  PubMed  Google Scholar 

  • Choi, H. J., Krieger, J. L., & Hecht, M. L. (2013). Reconceptualizing efficacy in substance use prevention research: refusal response efficacy and drug resistance self-efficacy in adolescent substance use. Health Communication, 28, 40–52. doi:10.1080/10410236.2012.720245.

    Article  PubMed Central  PubMed  Google Scholar 

  • Colby, M., Hecht, M. L., Miller-Day, M., Krieger, J. R., Syvertsen, A. K., Graham, J. W., et al. (2013). Adapting school-based substance use prevention curriculum through cultural grounding: a review and exemplar of adaptation processes for rural schools. American Journal of Community Psychology, 1–2, 190–205. doi:10.1007/s10464-012-9524-8.

    Article  Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/S0272-7358(97)00043-3.

    Article  CAS  PubMed  Google Scholar 

  • Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. doi:10.1207/S1532768XJEPC1102_04.

    Article  Google Scholar 

  • Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention & Intervention in the Community, 17, 5–18. doi:10.1300/J005v17n02_02.

    Article  Google Scholar 

  • Durlak, J., & DuPre, E. (2008). Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.

    Article  PubMed  Google Scholar 

  • Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256. doi:10.1093/her/18.2.237.

    Article  PubMed  Google Scholar 

  • Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313. doi:10.1093/her/cyg134.

    Article  PubMed  Google Scholar 

  • Ellickson, P. L., & Bell, R. M. (1990). Drug prevention in junior high: a multi-site longitudinal test. Science, 247, 1299–1305. doi:10.1126/science.2180065.

    Article  CAS  PubMed  Google Scholar 

  • Graczyk, P. A., Domitrovich, C. E., & Zins, J. E. (2002). Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. Handbook of School Mental Health Advancing Practice and Research, 301–318. doi:10.1007/978-0-387-73313-5_21

  • Graham, J. W. (2012). Missing data: analysis and design. New York: Springer.

    Book  Google Scholar 

  • Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323–343. doi:10.1037/1082-989X.11.4.323.

    Article  PubMed  Google Scholar 

  • Graham, J., Olchowski, A., & Gilreath, T. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206–213. doi:10.1007/s11121-007-0070-9.

    Article  PubMed  Google Scholar 

  • Graham, J. W., Pettigrew, J., Miller-Day, M., Krieger, J. L., Zhou, J., & Hecht, M. L. (2013). Random assignment of schools to groups in the drug resistance strategies rural project: some new methodological twists. Prevention Science. doi:10.1007/s11121-013-0403-9.

    Google Scholar 

  • Hallfors, D., & Godette, D. (2002). Will the “principles of effectiveness” improve prevention practice? Early findings from a diffusion study. Health Education Research, 17, 461–470. doi:10.1093/her/17.4.461.

    Article  CAS  PubMed  Google Scholar 

  • Hansen, W. B., & Graham, J. W. (1991). Preventing alcohol, marijuana, and cigarette use among adolescents: peer pressure resistance training versus establishing conservative norms. Preventive Medicine, 20, 414–430. doi:10.1016/0091-7435(91)90039-7.

    Article  CAS  PubMed  Google Scholar 

  • Hansen, W. B., Graham, J. W., Wolkenstein, B. H., & Rohrbach, L. A. (1991). Program integrity as a moderator of prevention program effectiveness: results for fifth-grade students in the adolescent alcohol prevention trial. Journal of Studies on Alcohol and Drugs, 52, 568–579.

    Article  CAS  Google Scholar 

  • Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77–89. doi:10.1080/19312450709336664.

    Article  Google Scholar 

  • Hecht, M. L., Graham, J. W., & Elek, E. (2006). The drug resistance strategies intervention: program effects on substance use. Health Communication, 20, 267–276. doi:10.1207/s15327027hc2003_6.

    Article  PubMed  Google Scholar 

  • Ho, A. D. & Kane, T. J. (2013). The reliability of classroom observations by school personnel Retrieved from http://metproject.org/downloads/MET_Reliability_of_Classroom_Observations_Research_Paper.pdf

  • Knoche, L. L., Sheridan, S. M., Edwards, C. P., & Osborn, A. Q. (2010). Implementation of a relationships-based school readiness intervention: a multidimensional approach to fidelity measurement for early childhood. Early Childhood Research Quarterly, 25, 299–313. doi:10.1016/j.ecresq.2009.05.003.

    Article  PubMed Central  PubMed  Google Scholar 

  • Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257. doi:10.1177/1090198103260514.

    Article  Google Scholar 

  • Low, S., Ryzin, M. J. V., Brown, E. C., Smith, B. H., & Haggerty, K. P. (2013). Engagement matters: lessons from assessing classroom implementation of steps to respect: a bullying prevention program over a one-year period. Prevention Science. doi:10.1007/s11121-012-0359-1.

    Google Scholar 

  • Marsiglia, F. F., Kulis, S., Yabiku, S. T., Nieri, T. A., & Coleman, E. (2011). When to intervene: elementary school, middle school or both? Effects of keepin’ it REAL on substance use trajectories of Mexican heritage youth. Prevention Science, 12, 48–62. doi:10.1007/s11121-010-0189-y.

    Article  PubMed Central  PubMed  Google Scholar 

  • Miller-Day, M., Pettigrew, J., Hecht, M. L. M., Shin, Y., Graham, J., & Krieger, J. (2013). How prevention curricula are taught under real-world conditions: types of and reasons for teacher curriculum adaptations in 7th grade drug prevention curriculum. Health Education, 113, 324–344. doi:10.1108/09654281311329259.

    Article  PubMed Central  PubMed  Google Scholar 

  • Needle, R. H., Jou, S. C., & Su, S. S. (1989). The impact of changing methods of data collection on the reliability of self-reported drug use of adolescents. The American Journal of Drug and Alcohol Abuse, 15, 275–289. doi:10.3109/00952998908993408.

    Article  CAS  PubMed  Google Scholar 

  • Odom, S. L., Fleming, K., Diamond, K., Lieber, J., Hanson, M., Butera, G., et al. (2010). Examining different forms of implementation and in early childhood curriculum research. Early Childhood Research Quarterly, 25, 314–328. doi:10.1016/j.ecresq.2010.03.001.

    Article  PubMed Central  PubMed  Google Scholar 

  • Pentz, M. A., Trebow, E. A., Hansen, W. B., Mackinnon, D. P., Dwyer, J. H., Johnson, C. A., et al. (1990). Effects of program implementation on adolescent drug use behavior: the Midwestern Prevention Project (MPP). Evaluation Review, 14, 264–289.

    Article  Google Scholar 

  • Pettigrew, J., Miller-Day, M., Krieger, J., & Hecht, M. L. (2011). Alcohol and other drug resistance strategies employed by rural adolescents. Journal of Applied Communication Research, 39, 103–122. doi:10.1080/00909882.2011.556139.

    Article  PubMed Central  PubMed  Google Scholar 

  • Pettigrew, J., Miller-Day, M., Shin, Y., Hecht, M. L., Krieger, J. R., & Graham, J. W. (2013). Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin’ it REAL substance use intervention. American Journal of Community Psychology. doi:10.1007/s10464-012-9539-1.

    Google Scholar 

  • Ringwalt, C., Vincus, A., Hanley, S., Ennett, S., Bowling, J., & Haws, S. (2011). The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2008. Prevention Science, 12, 63–69. doi:10.1007/s11121-010-0184-3.

    Article  PubMed Central  PubMed  Google Scholar 

  • Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Preventive Medicine, 22, 237–260. doi:10.1006/pmed.1993.1020.

    Article  CAS  PubMed  Google Scholar 

  • Rohrbach, L., Gunning, M., Sun, P., & Sussman, S. (2010). The project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prevention Science, 11, 77–88. doi:10.1007/s11121-009-0151-z.

    Article  PubMed Central  PubMed  Google Scholar 

  • Sanchez, V., Steckler, A., Nitirat, P., Hallfors, D., Cho, H., & Brodish, P. (2007). Fidelity of implementation in a treatment effectiveness trial of reconnecting youth. Health Education Research, 22, 95–107.

    Article  PubMed  Google Scholar 

  • Shin, Y., Miller-Day, M., Pettigrew, J., & Hecht, M. L. (2011). Qualitative approach to implementer typologies: how teachers and students interact in implementation of school based prevention intervention. Washington DC: Paper presented at the meeting of the Society for Prevention Research.

    Google Scholar 

  • Tobler, N. S., Roona, M. R., Ochshorn, P., Marshall, D. G., Streke, A. V., & Stackpole, K. M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. The Journal of Primary Prevention, 20, 275–336. doi:10.1023/A:1021314704811.

    Article  Google Scholar 

Download references

Author’s Note

This publication was supported by Grant Number R01DA021670 from the National Institute on Drug Abuse to The Pennsylvania State University (Michael Hecht, Principal Investigator). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. (NIH Manuscript # NIHMS272843). Portions of this paper were presented at the 2013 annual meeting of the Society of Prevention Research.

Author information

Author notes

  1. Jonathan Pettigrew

    Present address: School of Communication Studies, University of Tennessee, 293 Communications Building, Knoxville, TN, 37996, USA

  2. Michelle Miller-Day

    Present address: Department of Communication Studies, Chapman University, Orange, CA, USA

  3. Young Ju Shin

    Present address: Department of Communication Studies, Indiana University–Purdue University Indianapolis, Indianapolis, IN, USA

Authors and Affiliations

  1. Department of Communication Arts and Sciences, Penn State, State College, PA, USA

    Jonathan Pettigrew, Michelle Miller-Day, Michael L. Hecht & Young Ju Shin

  2. Department of Biobehavioral Health, Penn State, State College, PA, USA

    John W. Graham

  3. Prevention Research Center, Penn State, State College, PA, USA

    John W. Graham & Michael L. Hecht

  4. School of Communication, The Ohio State University, Columbus, OH, USA

    Janice L. Krieger

Authors
  1. Jonathan Pettigrew
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John W. Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michelle Miller-Day
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael L. Hecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Janice L. Krieger
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Young Ju Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jonathan Pettigrew.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pettigrew, J., Graham, J.W., Miller-Day, M. et al. Adherence and Delivery: Implementation Quality and Program Outcomes for the Seventh-Grade keepin’ it REAL Program. Prev Sci 16, 90–99 (2015). https://doi.org/10.1007/s11121-014-0459-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-014-0459-1

Keywords

Search

Navigation