Skip to main content
SpringerLink
Log in

Methodological Challenges Examining Subgroup Differences: Examples from Universal School-based Youth Violence Prevention Trials

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

This article reviews the literature on school-based universal violence prevention programs to illustrate key methodological challenges for investigating subgroup differences in prevention effects. The variety of potential moderating factors examined within this literature is discussed within the context of a social-ecological model. Our review of this literature identified the following methodological issues: the need for a clear a priori theoretical basis for selecting potential moderators, inflated Type I error rates that result from large numbers of comparisons, the absence of explicit tests of moderation, interpretive issues arising from a restricted range on moderator variables, the failure to report effect size estimates, the presence of potential confounding factors, and the importance of examining factors that might operate at multiple ecological levels. These points are illustrated using examples of studies, primarily within youth violence prevention research, that have identified factors within the individual, school, and community that moderate the outcomes of preventive interventions. We conclude with general recommendations for future work. These include the benefits of using the social-ecological model to provide a basis for moving from exploratory to more theory-driven confirmatory models of subgroup differences, the potential merits of qualitative research designed to identify factors that may influence the effectiveness of intervention efforts for specific subgroups of individuals, and the provision of effect size estimates and confidence intervals for effect sizes in prevention reports.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. A table listing the studies included in the review may be obtained from the first author.

References

  • Aber, J. L., Jones, S. M., Brown, J. L., Chaundry, N., & Samples, F. (1998). Resolving conflict creatively: Evaluating the developmental effects of a school-based violence prevention program in neighborhood and classroom context. Development and Psychopathology, 10, 187–214.

    Article  PubMed  CAS  Google Scholar 

  • Aber, J. L., Brown, J. L., & Jones, S. M. (2003). Developmental trajectories toward violence in middle childhood: Course, demographic differences, and response to school-based intervention. Developmental Psychology, 3, 324–348.

    Article  Google Scholar 

  • Asch, S. E. (1955). Opinions and social pressure. Scientific American, 193, 31–35.

    Article  Google Scholar 

  • Bronfenbrenner, U. (1979). Contexts of child rearing: Problems and prospects. American Psychologist, 34, 844–850.

    Article  Google Scholar 

  • Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., et al. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48, 1013–1022.

    Article  PubMed  CAS  Google Scholar 

  • Conduct Problems Prevention Research Group. (1999a). Initial impact of the Fast Track prevention trial for conduct problems: I. The high-risk sample. Journal of Consulting and Clinical Psychology, 67, 631–647.

    Article  Google Scholar 

  • Conduct Problems Prevention Research Group. (1999b). Initial impact of the Fast Track prevention trial for conduct problems II: Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.

    Article  Google Scholar 

  • Conduct Problems Prevention Research Group. (2010). The effects of a multiyear universal social—emotional learning program: The role of student and school characteristics. Journal of Consulting and Clinical Psychology, 78, 156–168.

    Article  Google Scholar 

  • Dahlberg, L. L., & Krug, E. G. (2002). Violence—a global public health problem. In E. Krug, L. L. Dahlberg, J. A. Mercy, A. B. Zwi, & R. Lozano (Eds.), World report on violence and health (pp. 1–56). Geneva, Switzerland: World Health Organization.

    Google Scholar 

  • Davis, C. H., MacKinnon, D. P., Schultz, A., & Sandler, I. (2003). Cumulative risk and population attributable fraction in prevention. Journal of Clinical Child and Adolescent Psychology, 32, 228–235.

    PubMed  Google Scholar 

  • Dishion, T. J., Patterson, G. R., & Griesler, P. C. (1994). Peer adaptation in the development of antisocial behavior: A confluence model. In L. R. Huesmann (Ed.), Aggressive behavior: Current perspectives (pp. 61–95). New York: Plenum Press.

    Google Scholar 

  • Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm: Peer groups and problem behavior. American Psychologist, 54, 755–764.

    Article  PubMed  CAS  Google Scholar 

  • Dishion, T. J., Kavanagh, K., Schneiger, A., Nelson, S., & Kaufman, N. K. (2002). Preventing early adolescent substance use: A family-centered strategy for the public middle school. Prevention Science, 3, 191–201.

    Article  PubMed  Google Scholar 

  • Dodge, K. A., Lansford, J. E., & Dishion, T. J. (2006). The problem of deviant peer influences in intervention programs. In K. A. Dodge, T. J. Dishion, & J. E. Lansford (Eds.), Deviant peer influences in programs for youth: Problems and solutions (pp. 3–13). New York: Guilford.

    Google Scholar 

  • Fagan, J., & Wilkinson, D. L. (1998). Social contexts and functions of adolescent violence. In D. S. Elliott, B. A. Hamburg, & K. R. Williams (Eds.), Violence in American schools: A new perspective (pp. 55–93). New York: Cambridge University Press.

    Google Scholar 

  • Farrell, A. D., & Meyer, A. L. (1997). The effectiveness of a school-based curriculum for reducing violence among urban sixth-grade students. American Journal of Public Health, 87, 979–984.

    Article  PubMed  CAS  Google Scholar 

  • Farrell, A. D., & Vulin-Reynolds, M. (2007). Violent behavior and the science of prevention. In D. J. Flannery, A. T. Vazsonyi, & I. D. Waldman (Eds.), The Cambridge handbook of violent behavior and aggression (pp. 767–786). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Farrell, A. D., Meyer, A. L., Kung, E. M., & Sullivan, T. N. (2001a). Development and evaluation of school-based violence prevention programs. Journal of Clinical Child Psychology, 30, 207–220.

    Article  CAS  Google Scholar 

  • Farrell, A. D., Meyer, A. L., & White, K. S. (2001b). Evaluation of Responding in Peaceful and Positive Ways (RIPP): A school-based prevention program for reducing violence among urban adolescents. Journal of Clinical Child Psychology, 30, 451–463.

    Article  CAS  Google Scholar 

  • Farrell, A. D., Erwin, E. H., Bettencourt, A., Mays, S., Vulin-Reynolds, M., Sullivan, T. N., et al. (2008). Individual factors influencing effective nonviolent behavior and fighting in peer situations: A qualitative study with urban African American adolescents. Journal of Clinical Child and Adolescent Psychology, 37, 397–411.

    PubMed  Google Scholar 

  • Farrell, A. D., Mays, S., Bettencourt, A., Erwin, E. H., Vulin-Reynolds, M., & Allison, K. W. (2010). Environmental influences on fighting versus nonviolent behavior in peer situations: A qualitative study with urban African American adolescents. American Journal of Community Psychology, 46, 19–35.

    Article  PubMed  Google Scholar 

  • Farrell, A. D., Henry, D., Mays, S., & Schoeny, M. (2011). Parents as moderators of the impact of peers and school environment. Child Development, 82, 146–161.

    Article  PubMed  Google Scholar 

  • Flannery, D. J., Vazsonyi, A. T., & Waldman, I. D. (Eds.). (2007). The Cambridge handbook of violent behavior and aggression. New York: Cambridge University Press.

    Google Scholar 

  • Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.

    Article  PubMed  Google Scholar 

  • Foshee, V. A., Bauman, K. E., Ennett, S. T., Suchindran, C., Benefield, T., & Linder, G. F. (2005). Assessing the effects of the dating violence prevention program "Safe Dates" using random coefficient regression modeling. Prevention Science, 6, 245–258.

    Article  PubMed  Google Scholar 

  • Fraley, C., & Raferty, A. E. (1998). How many clusters? Which clustering method? Answers via model-based cluster analysis. The Computer Journal, 41, 478–488.

    Article  Google Scholar 

  • Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., et al. (2005). Social information-processing skills training to promote social competence and prevent aggressive behavior in the third grade. Journal of Consulting and Clinical Psychology, 73, 1045–1055.

    Article  PubMed  Google Scholar 

  • Frey, K. S., Hirchstein, M. K., & Guzzo, B. A. (2000). Second Step: Preventing aggression by promoting social competence. Journal of Emotional and Behavioral Disorders, 8, 102–112.

    Article  Google Scholar 

  • Gottfredson, D. C., Gottfredson, G. D., & Hybl, L. G. (1993). Managing adolescent behavior: A multiyear, multischool study. American Educational Research Journal, 30, 179–215.

    Google Scholar 

  • Grabosky, P. N. (1996). Unintended consequences of crime prevention. In R. Homel (Ed.), Crime prevention studies, vol 5: The politics and practice of situational crime prevention (pp. 25–57). Monsey, NY: Criminal Justice Press.

    Google Scholar 

  • Gregory, A., Henry, D., Schoeny, M. & The Metropolitan Area Child Study Research Group. (2007). School climate and implementation of a preventive intervention. American Journal of Community Psychology, 40, 250–260.

    Article  Google Scholar 

  • Griffin, J. P., Jr., Chen, D., Eubanks, A., Brantley, K. M., & Willis, L. A. (2007). A randomized violence prevention trial with comparison: Responses by gender. Journal of School Violence, 6, 65–81.

    Article  Google Scholar 

  • Hanish, L. D., & Guerra, N. G. (2000). The roles of ethnicity and school context in predicting children's victimization by peers. American Journal of Community Psychology. Special Issue: Minority Issues in Prevention, 28, 201–223.

    Article  CAS  Google Scholar 

  • Hawkins, J. D., Herrenkohl, T., Farrington, D. P., Brewer, D., Catalano, R. F., Harachi, T. W., et al. (1998). A review of predictors of youth violence. In R. Loeber & D. P. Farrington (Eds.), Serious & violent juvenile offenders: Risk factors and successful interventions (pp. 106–146). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Henry, D., & Chan, W. Y. (2010). Cross-sectional and longitudinal effects of sixth grade setting-level norms for nonviolent problem solving on aggression and associated attitudes. Journal of Community Psychology, 38, 1007–1022.

    Article  Google Scholar 

  • Henry, D. B., & Schoeny, M. E. (2007, June). Moderation of universal intervention effects by norms and ethnicity (S-11372: “Violence and poverty: Testing theoretical models for African American youth”). Paper presented at the Bienniel Meeting of the Society for Community Research and Action, Pasadena, CA.

  • Henry, D. B., Farrell, A. D., Schoeny, M. E., Tolan, P. H., & Dymnicki, A. B. (2010). Influence of school-level variables on aggression and associated attitudes of middle school students. Manuscript submitted for publication.

  • Huesmann, L. R., Maxwell, C., Eron, L., Dahlberg, L. L., Guerra, N. G., Tolan, P. H., et al. (1996). Evaluating a cognitive/ecological program for the prevention of aggression in urban children. Preventive Medicine, 12, 120–128.

    CAS  Google Scholar 

  • Jackson, J. (1966). A conceptual and measurement model for norms and roles. Pacific Sociological Review, 9, 35–47.

    Article  Google Scholar 

  • Jaycox, L. H., McCaffrey, D., Eiseman, B., Aronoff, J., Shelley, G. A., & Collins, R. L. (2006). Impact of a school-based dating violence prevention program among Latino teens: Randomized controlled effectiveness trial. Journal of Adolescent Health, 39, 694–704.

    Article  PubMed  Google Scholar 

  • Jo, B. (2002). Estimation of intervention effects with noncompliance: Alternative model specifications. Journal of Educational and Behavioral Statistics, 27, 385–409.

    Article  Google Scholar 

  • Kam, C., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.

    Article  PubMed  Google Scholar 

  • Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.

    Article  Google Scholar 

  • Kellam, S. G., Ling, X., Merisca, R., Brown, C. H., & Ialongo, N. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–185.

    Article  PubMed  CAS  Google Scholar 

  • Kiesler, D. J. (1966). Some myths of psychotherapy research and the search for a paradigm. Psychological Bulletin, 65, 110–136.

    Article  Google Scholar 

  • Leadbeater, B., Hoglund, W., & Woods, T. (2003). Changing contents? The effects of a primary prevention program on classroom levels of peer relational and physical victimization. Journal of Community Psychology, 31, 397–418.

    Article  Google Scholar 

  • Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.

    Article  Google Scholar 

  • Lipsey, M. W., & Derzon, J. H. (1998). Predictors of violent or serious delinquency in adolescence and early adulthood: A synthesis of longitudinal research. In R. Loeber & D. P. Farrington (Eds.), Serious & violent juvenile offenders: Risk factors and successful interventions (pp. 86–105). Thousand Oaks, CA: Sage Publications, Inc.

    Google Scholar 

  • Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767–778.

    Article  Google Scholar 

  • MacKinnon, D. P. (2008). Introduction to statistical mediation analysis. New York: Taylor & Francis Group/Lawrence Erlbaum Associates.

    Google Scholar 

  • MacKinnon, D. P., & Lockwood, C. M. (2003). Advances in statistical methods for substance abuse prevention research. Prevention Science, 4, 155–171.

    Article  PubMed  Google Scholar 

  • Metropolitan Area Child Study Research Group (MACS). (2002). A cognitive-ecological approach to preventing aggression in urban settings: Initial outcomes for high risk children. Journal of Consulting and Clinical Psychology, 70, 179–194.

    Article  Google Scholar 

  • Meyer, A. L., Farrell, A. D., Northup, W., Kung, E., & Plybon, L. (2000a). Promoting nonviolence in early adolescence: Responding in peaceful and positive ways. New York: Kluwer.

    Book  Google Scholar 

  • Meyer, A., Nicholson, R., Danish, S., Fries, E., & Polk, V. (2000b). A model to measure program integrity of peer-led health promotion programs in rural middle schools: Assessing the implementation of the sixth grade goals for health program. Journal of Educational and Psychological Consultation, 11, 223–252.

    Article  Google Scholar 

  • Multisite Violence Prevention Project. (2008). Impact of a universal school-based violence prevention program on social-cognitive outcomes. Prevention Science, 9, 231–244.

    Article  Google Scholar 

  • Multisite Violence Prevention Project. (2009). The ecological effects of universal and selective violence prevention programs for middle school students: A randomized trial. Journal of Consulting and Clinical Psychology, 77, 526–542.

    Article  Google Scholar 

  • Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109–118.

    Article  PubMed  CAS  Google Scholar 

  • Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2004). Classroom Assessment Scoring System [CLASS]. Unpublished measure. University of Virginia.

  • Raudenbush, S. W., & Sampson, R. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Sociological Methodology, 29, 1–41.

    Article  Google Scholar 

  • Reid, J. B., Eddy, J. M., Fetrow, R. A., & Stoolmiller, M. (1999). Description and immediate impacts of a preventive intervention for conduct problems. American Journal of Community Psychology, 27, 483–517.

    Article  PubMed  CAS  Google Scholar 

  • Rosenthal, R. (1979). The 'file drawer problem' and tolerance for null results. Psychological Bulletin, 86, 638–641.

    Article  Google Scholar 

  • Rousseau, D. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1–37.

    Google Scholar 

  • Segawa, E., Ngwe, J. E., Li, Y., & Flay, B. R. (2005). Evaluation of the effects of the Aban Aya Youth Project in reducing violence among African American adolescent males using latent class growth mixture modeling techniques. Evaluation Review, 29, 128–148.

    Article  PubMed  Google Scholar 

  • Shinn, M., & Rapkin, B. D. (2000). Cross-level research without cross-ups in community psychology. In J. Rappaport & E. Seidman (Eds.), Handbook of community psychology (pp. 669–695). Dordrecht, The Netherlands: Kluwer Academic Publishers.

    Chapter  Google Scholar 

  • Smith, B., & Sechrest, L. (1991). Treatment of aptitude × treatment interactions. Journal of Consulting and Clinical Psychology, 59, 233–244.

    Article  PubMed  CAS  Google Scholar 

  • Smokowski, P. R., Fraser, M. W., Day, S. H., Galinsky, M. J., & Bacallao, M. L. (2004). School-based skills training to prevent aggressive behavior and peer rejection in childhood: Evaluating the making choices program. Journal of Primary Prevention, 25, 233–251.

    Article  Google Scholar 

  • Spoth, R., Redmond, C., Shin, C., Lepper, H., Haggerty, K., & Wall, M. (1998). Risk moderation of parent and child outcomes in a preventive intervention: A test and replication. American Journal of Orthopsychialry, 68, 565–579.

    Article  CAS  Google Scholar 

  • Steiger, J. H. (2004). Beyond the F Test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9, 164–182.

    Article  PubMed  Google Scholar 

  • Stoolmiller, M., Eddy, J. M., & Reid, J. B. (2000). Detecting and describing preventive intervention effects in a universal school-based randomized trial targeting delinquent and violent behavior. Journal of Consulting and Clinical Psychology, 68, 296–306.

    Article  PubMed  CAS  Google Scholar 

  • Stuart, E. A., Perry, D. F., Le, H.-N., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science, 9, 288–298.

    Article  PubMed  Google Scholar 

  • Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 24–31.

    Article  Google Scholar 

  • Tolan, P. H., & Brown, C. H. (1998). Evaluation research on violence interventions: Issues and strategies for design. In P. K. Trickett & C. J. Schellenbach (Eds.), Violence against children in the family and the community (pp. 439–464). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Tolan, P. H., Gorman-Smith, D., & Henry, D. (2004). Supporting families in a high-risk setting: proximal effects of the SAFEChildren preventive intervention. Journal of Consulting and Clinical Psychology, 72, 855–869.

    Article  PubMed  Google Scholar 

  • Tseng, V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology. Special Issue: Systems Change, 39, 217–228.

    Article  Google Scholar 

  • US Department of Health and Human Services (USDHHS) Public Health Service (PHS); Office of the Surgeon General. (2001). Youth violence: A report of the Surgeon General. Rockville MD: US Department of Health and Human Services.

    Google Scholar 

  • Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136–149.

    Article  PubMed  Google Scholar 

  • Wright, J. C., & Zimmerman, M. A. (2006). Culturally sensitive interventions to prevent youth violence. In N. G. Guerra & E. P. Smith (Eds.), Preventing youth violence in a multicultural society (pp. 221–269). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Yale, M. E., Scott, K. G., Gross, M., & Gonzalez, A. (2003). Using developmental epidemiology to choose the target population for an intervention program in a high-risk neighborhood. Journal of Clinical Child and Adolescent Psychology, 32, 236–242.

    PubMed  Google Scholar 

  • Yau, L. H. Y., & Little, R. J. (2001). Inference for the complier-average causal effect from longitudinal data subject to noncompliance and missing data, with application to a job training assessment for the unemployed. Journal of the American Statistical Association, 96, 1232–1244.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Virginia Commonwealth University, P.O. Box 842018, Richmond, VA, 23284-2018, USA

    Albert D. Farrell

  2. School of Public Health, University of Illinois at Chicago, Chicago, IL, USA

    David B. Henry

  3. Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA

    Amie Bettencourt

Authors
  1. Albert D. Farrell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David B. Henry
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amie Bettencourt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Albert D. Farrell.

Additional information

This research was supported by cooperative agreements U49/CE000730 (VCU) and U49/CE000732 (UIC) from the Centers for Disease Control and Prevention (CDC). The research and interpretations reported are the sole responsibility of the authors and are not necessarily endorsed by CDC or represent the views, opinions, or policies of the CDC or their staff.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Farrell, A.D., Henry, D.B. & Bettencourt, A. Methodological Challenges Examining Subgroup Differences: Examples from Universal School-based Youth Violence Prevention Trials. Prev Sci 14, 121–133 (2013). https://doi.org/10.1007/s11121-011-0200-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-011-0200-2

Keywords

Search

Navigation