Reported are results of an independent effectiveness study of the Project ALERT drug prevention program implemented in eight Pennsylvania middle schools by outside program leaders employed by Cooperative Extension. In this randomized, 2-cohort longitudinal evaluation, 1,649 seventh-grade students completed a pretest and four waves of posttests over the 2-year program and 1-year follow-up. Project ALERT's effectiveness was tested through a 3-level hierarchical linear model. Analyses failed to yield any positive effects for substance use or mediators for use in the adult or teen-assisted delivery of the curriculum. An extensive set of additional analyses detected no differential program effects by student risk level, gender, school, or level of implementation quality. Potential explanations for outcomes relative to Project ALERT's original effectiveness trial are discussed, as well as implications for future research, including the need to conduct independent effectiveness studies of previously validated programs in a variety of contexts.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We accomplished this by assigning a value of zero to the pretest and using codes with a mean of zero for the remaining waves. Note that program effects on the second and third contrasts alone (and not the first) would reflect a mixed picture in which the program produced negative effects at some waves and positive effects at others. Overall beneficial effects of the program will be reflected by the first contrast.
Specifically, with so few schools it would be unwise to estimate random variance components, especially based on an assumed normal error distribution.
Results of these analyses (and all others referred to, but not presented in Tables 1, 2, & 3) appear in an appendix to this paper available from the first author.
Ellickson et al. (2003) controlled for pretest marijuana use because their treatment and control groups differed on this variable, but we had no need to do so. Also, differences between our questionnaire and Ellickson et al.'s meant that we could not control for two variables that we did not measure: age, which was essentially constant for this single-grade sample, and normative beliefs about adult use. Ellickson et al. (2003) assigned entire schools to treatment and control groups while we assigned classrooms within schools, so the nesting in their study was within schools rather than classrooms. Our analyses controlled for overall school differences through dummy variables, as in our primary analyses. We obtained generalized estimating equations with sandwich errors through the HLM program's population average estimates with robust standard errors (Raudenbush & Bryk, 2002).
We obtained this probability by converting Ellickson et al.'s percentage difference to log odds and conducting a z-test of that value based on the standard error of the corresponding coefficient in our analysis (i.e., the interaction of the pre-post contrast with the dummy variable for combined treatment groups).
Our decision to base analyses on within-individual change reduced statistical power in favor of a stronger control for preexisting individual differences. Additional analyses using covariance controls for pretest differences offered greater statistical power but, like the rest of our analyses, did not yield evidence of beneficial program effects.
REFERENCES
Allison, P. D. (1990). Change scores as dependent variables in regression analysis. Sociological Methodology, 20, 93–114.
Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a White middle-class population. Journal of the American Medical Association, 273(14), 1106–1112.
Brown, C. H., Berndt, D., Brinales, J. M., Zong, X., & Bhagwat, D. (2000). Evaluating the evidence of effectiveness for preventive interventions: Using a registry system to influence policy through science. Addictive Behaviors, 25(6), 955–964.
Collins, L. M., Murphy, S. A., & Bierman, K. L. (2004). A conceptual framework for adaptive preventive interventions. Prevention Science, 5, 185–196.
Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221.
CYFAR (Children, Youth, & Families) Annual Report (2001) (Web Page). Retrieved from http://www.reeusda.gov/4h/cyfar/annual_report.htm.
Dusenbury, L., & Falco, M. (1995). Eleven components of effective drug abuse prevention curricula. Journal of School Health, 65, 420–425.
Ellickson, P. L., & Bell, R. M. (1990). Drug prevention in junior high: A multi-site longitudinal test. Science, 247, 1299–1305.
Ellickson, P. L., McCaffrey, D. F., Ghosh-Dastidar, G., & Longshore, D. L. (2003). New inroads in preventing adolescent drug use: Results from a large-scale trial of Project ALERT in middle schools. American Journal of Public Health, 93, 1830–1836.
Ellickson, P. L., Miller, L., Robyn, A., Wildflower, L. A., & Zellman, G. L. (2000). Project ALERT. Los Angeles, CA: The BEST Foundation for a Drug-Free Tomorrow.
Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.
Esbensen, F.-A., Osgood, D. W., Taylor, T. J., Peterson, D., & Freng, A. (2001). How great is G.R.E.A.T.?: Results from a longitudinal quasi-experimental design. Criminology and Public Policy, 1, 87–115.
Greenberg, M. T. (2004). Current and future challenges in school-based prevention: The researcher perspective. Prevention Science, 5, 1–13.
Hallfors, D. (2004). Findings from a randomized controlled effectiveness trial of “Reconnecting Youth.” Paper presented at the 12th Annual Society for Prevention Research Meeting, Quebec City, Quebec, Canada.
Hallfors, D., & Godette, D.(2002). Will the “Principles of Effectiveness” improve prevention practice? Early findings from a diffusion study. Health Education Research, 17, 461–470.
Hallfors, D., Pankratz, M., & Sporer, A. (2001). Drug free schools survey II: Report of results. Chapel Hill, NC: Department of Maternal and Child Health, School of Public Health, University of North Carolina.
Harrington, N. G., Giles, S. M., Hoyle, R. H., Feeney, G. J., & Yungbluth, S. C. (2001). Evaluation of the All Stars character education and problem behavior prevention program: Effects on mediator and outcome variables for middle school students. Health Education and Behavior, 28(5), 533–546.
Johnston, L. D., O’Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2003). Monitoring the Future national survey results on drug use, 1975-2003: Volume I, Secondary school students (NIH Publication No. 04-5507). Bethesda, MD: National Institute on Drug Abuse.
Kramer, L., Laumann, G., & Brunson, L. (2000). Implementation and diffusion of the Rainbows program in rural communities: Implications for school-based prevention programming. Journal of Educational and Psychological Consultation, 11, 37–64.
Mayer, J. P., & Davidson, W. S., II. (2000). Dissemination of innovation as social change. In J. Rappaport & E. Seidman (Eds.), Handbook of community psychology. (pp. 421–438). New York: Kluwer Academic/Plenum Publishers.
Miller, P. A. (1988). To our colleagues in the land-grant universities: A statement on the crisis of youth in America. Unpublished manuscript.
Molgard, V. K. (1997). The extension service as key mechanism for research and services delivery for prevention of mental health disorders in rural areas. American Journal of Community Psychology, 25(4), 515–543.
Osgood, D. W., & Smith, G. (1995). Applying hierarchical linear modeling to extended longitudinal evaluations: The Boys Town follow-up study. Evaluation Review, 19, 3–38.
Pentz, M. A. (2003). Evidence-based prevention: Characteristics, impact, and future direction. Journal of Psychoactive Drugs, 35 (Special Suppl.), 143–152.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models, (2nd Edn). Newbury Park, CA: Sage.
Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.
Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1988). Social learning theory and the health belief model. Health Education Quarterly, 15(2), 175–183.
Society for Prevention Research. (2004). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Falls Church, VA.
St. Pierre, T. L., & Kaltreider, D. L. (2004). Tales of refusal, adoption, and maintenance: Evidence-based substance abuse prevention via school-extension collaborations. American Journal of Evaluation, 25(4), 479–491.
Substance Abuse and Mental Health Services Administration (2003). National Registry of Effective Programs. Retrieved July 26, 2004 from http://www.modelprograms.samhsa.gov/template.cfm?page = nrepbutton.
U.S. Department of Education, Safe and Drug-Free School Program (1998). Notice of final principles of effectiveness. Federal Register, 63(104), 29901–29906.
U.S. Department of Education (2001). Exemplary and promising safe, disciplined and drug free schools program 2001. Retrieved July 26, 2004 from http://www.ed.gov/admins/lead/safety/exemplary01/report_pg3.html.
ACKNOWLEDGMENTS
This research was supported by a grant from the National Institute on Drug Abuse (RO1DA12011). We wish to thank Frances Burden, Gretchen Ruth, and Brent Teasdale for their invaluable assistance with data analysis. We thank Dr. Susan McHale for her important contributions to the study's early conceptualization
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
St. Pierre, T.L., Osgood, D.W., Mincemoyer, C.C. et al. Results of an Independent Evaluation of Project ALERT Delivered in Schools by Cooperative Extension. Prev Sci 6, 305–317 (2005). https://doi.org/10.1007/s11121-005-0015-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-005-0015-0