Abstract
This study tested the effectiveness of a prevention curriculum, All Stars, as implemented in a year-long school-based after school program and provides an independent replication of the effects of All Stars on targeted mediators and problem behaviors using an experimental methodology. Middle school students (N = 447) who registered for the after school program were randomly assigned to the experimental or control condition. The sample included approximately equal proportions of males and females, was 70% African American, and 59% of the students received subsidized meals at school. All Stars was delivered with reasonable integrity to the program design, although with lower quality than reported in earlier efficacy trials. However, actual student exposure to the program was lower than expected due to low levels of attendance in the after school program. Students who ever attended received an average of 16 h of All Stars instruction. Results showed no differences between the treatment and control students at post-test on any of the outcomes or mediators. Further, no positive effects were found for youths receiving higher dosage, higher quality program delivery, or both. Insufficient time to achieve high quality implementation in the after school context and potential deviancy training are suggested as reasons for the failure to replicate positive program effects.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Some research has examined All Stars effects on mediators not specifically targeted by All Stars and on sexual activity outcomes. We do not discuss these findings as they are beyond the scope of this research.
We conducted power analyses for two-tailed independent t-tests, fixing the Type I error rate (alpha) at 0.05. These power analyses estimated the power available to detect a moderate-sized effect (0.30) and indicated that, using the entire sample, the power to detect differences between the treatment and control groups is 0.99. We calculated power for a number of different outcomes, including aggression (measured continuously) and vulnerability to drug use and last-month frequency drug use (measured as binary outcomes). These power analyses also estimated the minimum effect size detectable with a sample of our size and power fixed at 0.80. Analyses including both a dummy variable for school and the pre-test measure as a covariate will allow us to detect effects of about 0.17.
These students were excluded from outcome analysis because of missing posttest data. They either refused to take the posttest (N = 10), had transferred out of Maryland schools (N = 10), or left more than 40% of the survey items blank (N = 11).
Participants who reported some use at pre-test or failed to report at one or both time periods for cigarettes (37, 9%), alcohol (75, 18%), marijuana (30, 7%), inhalants (40, 10%) and other illegal drugs (11, 3%) were excluded from analyses of initiation for each substance.
Each All Stars lesson is designed to be delivered start-to-finish during one session. However, the ASP sites delivered one lesson over 2 days in two 45-min sessions because the program structure provided extra time.
Subjects were clustered within site. Clustering often violates the independence assumption in OLS regression and may introduce negative bias in standard errors, therefore inflating alpha levels in statistical tests. This negative bias increases in severity as the degree of correlation among clustered observations increases. When the proportion of variance in outcomes that is between groups (the intra-class correlation, or ICC) approaches zero, the consequences of clustering are nil. In this study, very little of the variance in the time 1 measures was between schools. The ICCs ranged from 0.000 to 0.025 and for the most part did not reach nominal levels of statistical significance. The largest proportions of variance between schools were found for disruptive classroom behavior and commitment to abstain from drugs. This suggests that clustering is unlikely to have inflated significance tests statistics that assumed a simple random sample. Nevertheless, we follow the advice of Snijders and Bosker (1999) for handling clustered data when the number of clusters is small (<10), and include dummy variables for schools in all outcome analyses to correct for intercept differences across schools.
The grouping for All Stars was haphazard, depending on the number of students present on each day. The All Stars groups were not systematically recorded.
We considered testing for interactions by the use of certified teachers, following McNeal et al. (2004) who showed that positive effects were found only for the teacher condition. However, in our study, such analyses were redundant with the analyses of interactions by quality because only in sites A, B, and C were lessons led by certified teachers.
The 398 discrete activities included 101 All Stars sessions. However, the number of All Stars sessions in which responses to misbehavior were observed (N = 74) is too small to allow for fine-grained analysis of differences across the five sites.
Several combinations of high v. low implementation schools resulted in significant interactions. The strongest effect appeared in analyses comparing the highest implementation site (Site B) to all others, in a pattern that replicated those reported in Hallfors et al. (2007). To maintain consistency with the analyses by program quality reported earlier, we report here the results of the All Stars by quality interaction test in which quality was measured using a dummy variable contrasting sites A, B, and C vs. D and E.
Response is coded 1 = reinforcing, 2 = neutral, 3 = chastising.
References
Allison, P.D. (2002). Missing data. Thousand Oaks, CA: Sage.
Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H., & Hawkins, J.D. (2004). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Annals of the American Academy of Political and Social Sciences, 591, 98–124.
Cho, H., Hallfors, D., & Sanchez, V. (2005). Evaluation of a high school peer group intervention for at-risk youth. Journal of Abnormal Child Psychology, 33, 363–374.
Dishion, T.J., Spracklen, K.M., & Andrews, D.W. (1996). Deviancy training in male adolescent friendships. Behavior Therapy, 27, 373–390.
Dishion, T.J., McCord, J., & Poulin, F. (1999). When interventions harm: Peer groups and problem behavior. American Psychologist, 54, 755–764.
Durlak, J.A., & Weissberg, R.P. (2007). The impact of after-school programs that promote personal and social skills. Chicago: Collaborative for Academic, Social, and Emotional Learning.
Eysenck, S.B., Easting, G., & Pearson, P.R. (1984). Age norms for impulsiveness, venturesomeness and empathy in children. Personality and Individual Differences, 5, 315–321.
Fagan, A.A., Hanson, K., Hawkins, J.D., & Arthur, M.W. (2008). Bridging science to practice: Achieving prevention program implementation fidelity in the Community Youth Development study. American Journal of Community Psychology, 41, 235–249.
Flay, B.R., Biglan, A., Boruch, R.F., Castro, F.G., Gottfredson, D.C., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–176.
Gottfredson, G.D., & Gottfredson, D.C. (1992). What about you?. Ellicott City, MD: Gottfredson Associates.
Gottfredson, G.D., & Gottfredson, D.C. (2001). What schools do to prevent delinquency and promote safe environments. Journal of Educational and Psychological Consultation, 12, 313–344.
Gottfredson, D.C., & Wilson, D.B. (2003). Characteristics of effective school-based substance abuse prevention. Prevention Science, 4, 27–38.
Gottfredson, G.D., Gottfredson, D.C., Czeh, E.R., Cantor, D., Crosse, S.B., & Hantman, I. (2000). A national study of delinquency prevention in school final report. Ellicott City, MD: Gottfredson Associates (ERIC No. ED 459 409) (available on line at http://www.gottfredson.com).
Gottfredson, G.D., Jones, E.M., & Gore, T.W. (2002a). Implementation and evaluation of a cognitive–behavioral intervention to prevent problem behavior in a disorganized school. Prevention Science, 3, 43–56.
Gottfredson, D.C., Wilson, D.B., & Najaka, S.S. (2002b). School-based crime prevention. In L.W. Sherman, D.P. Farrington, B.C. Welsh, & D.L. MacKenzie (Eds.), Evidence-based crime prevention (pp. 56–164). London: Routledge.
Gottfredson, D.C., Weisman, S.A., Soulé, D.A., Womer, S.C., & Lu, S. (2004). Do after school programs reduce delinquency? Prevention Science, 5, 253–266.
Gottfredson, D.C., Cross, A.B., & Soulé, D.A. (2007). Distinguishing characteristics of effective and ineffective after-school programs to prevent delinquency and victimization. Criminology & Public Policy, 6, 289–318.
Gottfredson, D.C., Cross, A.B., Wilson, D.M., Connell, N., & Rorie, M. (2009). A randomized trial of the effects of an enhanced after school program. College Park: MD. Unpublished report available from the authors.
Hallfors, D., Cho, H., Dishion, T., Cuffee, J., & Hartman, S. (2007). The role of teachers in school-based interventions for high risk adolescents: An alternative paradigm for predicting deviancy training. Unpublished manuscript.
Hamilton, L.S., Stecher, B.M., Marsh, J.A., Sloan McCombs, J., Robyn, A., Russell, J., et al. (2007). Standards-based accountability under No Child Left Behind: Experiences of teachers and administrators in three states. Santa Monica, CA: RAND Corporation.
Hansen, W.B. (1996). Pilot test results comparing the All Stars program with seventh grade D.A.R.E.: Program integrity and mediating variable analysis. Substance Use & Misuse, 31, 1359–1377.
Hansen, W.B., & Dusenbury, L. (2004). All Stars Plus: A competence and motivation enhancement approach to prevention. Health Education, 104, 371–381.
Harrington, N.G., Giles, S.M., Hoyle, R.H., Feeney, G.J., & Yungbluth, S.C. (2001). Evaluation of the All Stars character education and problem behavior prevention program: Effects on mediator and outcome variables for middle school students. Health Education & Behavior, 28, 533–546.
Huizinga, D., & Esbensen, F. (1990). Scales and measures of the Denver Youth Survey. Boulder, CO: Institute of Behavioral Science.
Kane, T.J. (2004). The impact of after school programs: Interpreting the results of four recent evaluations. The William T. Grant Foundation Working Paper.
Lauer, P.A., Akiba, M., Wilkerson, S.B., Apthorp, H.S., Snow, D., & Martin-Glenn, M.L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.
Lipsey, M.W., & Wilson, D.B. (1998). Effective intervention for serious juvenile offenders: A synthesis of research. In R. Loeber & D.P. Farrington (Eds.), Serious and violent juvenile offenders: Risk factors and successful intervention (pp. 313–345). Thousand Oaks, CA: Sage.
McNeal, R.B., Hansen, W.B., Harrington, N.G., & Giles, S.M. (2004). How All Stars works: An examination of program effects on mediating variables. Health Education & Behavior, 31, 165–178.
Patterson, G.R., Dishion, T.J., & Yoerger, K. (2000). Adolescent growth in new forms of problem behavior: Macro- and micro-peer dynamics. Prevention Science, 1, 3–13.
Rorie, M., Gottfredson, D.C., Cross, A., Wilson, D., & Connell, N. (2009). Structure and deviancy training in after-school programs. Manuscript submitted for publication.
Scott-Little, C., Hamann, M.S., & Jurs, S.G. (2002). Evaluations of after-school programs: A meta-evaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23, 387–419.
Slater, M.D., Kelly, K.J., Edwards, R.W., Thurman, P.J., Plested, B.A., Keefe, T.J., et al. (2006). Combining in-school and community-based media efforts: Reducing marijuana and alcohol uptake among younger adolescents. Health Education Research, 21, 157–167.
Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
Tanglewood Research. (2008a). All Stars quality assessments. Retrieved July, 2006, from www.allstarsprevention.com/programs/programsEvaluationQA.asp.
Tanglewood Research. (2008b). All Stars student surveys. Retrieved July, 2006, from www.allstarsprevention.com/programs/programsEvaluationOA.asp.
Wilson, D.B., Gottfredson, D.C., & Najaka, S.S. (2001). School-based prevention of problem behaviors: A meta-analysis. Journal of Quantitative Criminology, 17, 247–272.
Zief, S.G., & Lauver, S. (2006). The impacts of after-school programs on student outcomes: A systematic review for the Campbell Collaboration. Retrieved April, 2008, from www.campbellcollaboration.org/doc-pdf/zief_afterschool_review.pdf.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported through grant number R305F050069 from the U.S. Department of Education, Institute of Educational Sciences, to the University of Maryland. We thank Elise Andrews of the Baltimore County Local Management Board and Beahta Davis and the staff of the Baltimore County Dept. of Recreation and Parks for implementing and managing the after school programs. Thanks also to Matthew Brigham, Stephanie DiPietro, Matt Gugino, Lynda Okeke, Freshta Rahimi, and Jaynie Trageser for research assistance and to William Hansen and three anonymous reviewers for helpful comments on an earlier draft.
Rights and permissions
About this article
Cite this article
Gottfredson, D.C., Cross, A., Wilson, D. et al. An Experimental Evaluation of the All Stars Prevention Curriculum in a Community After School Setting. Prev Sci 11, 142–154 (2010). https://doi.org/10.1007/s11121-009-0156-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-009-0156-7