Skip to main content
SpringerLink
Log in

The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Using meta-analysis, we report on an investigation of the evaluator's influence in the treatment setting on criminal recidivism outcomes. Many evaluators and users of evaluation of social interventions worry that mixing of the roles of program developer and program evaluator may bias results reported in intervention studies in a positive direction. We first review the results of prior investigations of this issue across 50 prior meta-analyses, finding 12 that tested the impact of investigator influence in the treatment setting. Eleven of these reported that effect size increased positively, sometimes substantially so, when evaluators were influential or involved in the treatment setting. We followed this with a meta-analysis of 300 randomized field trials in individually focused crime reduction, also finding intervention studies in which evaluators who were greatly influential in the treatment setting report consistently and substantially larger effect sizes than other types of evaluators. We discuss two major views — the ‘cynical’ and ‘high fidelity’ theories — on why this is consistently the case, and conclude with a further agenda for research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alexander, M. A. (1999). Sexual offender treatment efficacy revisited. Sexual Abuse: A Journal of Research and Treatment 11(2), 101–117.

    Google Scholar 

  • Andrews, D. A., Zinger, I., Hoge, R. D., Bonta, J., Gendreau, P. & Cullen, F. T. (1990). Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology 28(3), 369–404.

    Google Scholar 

  • Antonowicz, D. H. & Ross R. R. (1994). Essential components of Successful rehabilitation programs for offenders. Internaltional Journal of Offender and Comparative Criminology 38(2), 97–104.

    Google Scholar 

  • Berecochea, J. & Jaman, D. (1981). Time served in prison and parole outcome: An experimental study. Report 2. Sacramento: California Department of Corrections.

  • Boruch, R. F. (1997). Randomized experiments for policy and planning. Newbury Park, CA: Sage.

    Google Scholar 

  • Boruch, R. F. & Petrosino, A. (2004). Meta-analyses, systematic reviews, and research syntheses Chapter 7. In J. Wholey, H. Hatry & K. Newcomer (Eds.), Handbook of practical program evaluation. 2nd edn. San Francisco: Jossey-Bass.

    Google Scholar 

  • Campbell, D. T. (1969). Reforms as experiments. American Psychologist 24, 409–429.

    Google Scholar 

  • Cleland, C. M., Pearson, F. S. & Lipton, D. S. (1996). A meta-analytic approach to the link between needs-targeted treatment and reductions in criminal offending. Paper presented at the Annual Meeting of the American Society of Criminology, Chicago, Illinois.

  • Cook, T. D. (2005). Why have educational evaluators chosen not to do randomized experiments? Paper presented at Random Evaluation in a Non-Routinized Environment: New Perceptions on R&D in Education and Other Social Sectors, Ecole Polytechnique Federale de Lausanne, Switzerland, April 9th.

  • Cox, S., Davidson, W. & Bynum, T. (1995). A meta-analytic asessment of deliquency-related outcomes of alternative education programs. Crime and Delinquency 41(2), 219–234.

    Google Scholar 

  • Curtis, N. M., Ronan, K. R. & Borduin, C. M. (2004). Multisystemic treatment: A meta-analysis of outcome studies. Journal of Family Psychology 18(3), 411–419.

    Article  Google Scholar 

  • Davidson, W. S., Redner, R., Mitchell, C. M. & Amdur, R. (1990). Alternative treatments for troubled youth. New York: Plenum.

    Google Scholar 

  • Dowden, C. & Andrews, D. A. (1999a). What works for female offenders: A meta-analytic review. Crime and Delinquency 45, 438–452.

    Google Scholar 

  • Dowden, C. & Andrews, D. A. (1999b). What works in young offender treatment: A meta-analysis. Forum on Corrections Research 11(2), 21–24.

    Google Scholar 

  • Dowden, C. & Andrews, D. A. (2000). Effective correctional treatment and violent reoffending: A meta-analysis. Canadian Journal of Criminology 449–467.

  • Dowden, C., Antonowicz, D. & Andrews, D. A. (2003). The effectiveness of relapse prevention with offenders: A meta-analysis. International Journal of Offender Therapy and Comparative Criminology 47(5), 516–528.

    Article  Google Scholar 

  • Drug Strategies (1999). Making the grade: A guide to school drug prevention programs. Updated and expanded. Washington, DC: Drug Strategies.

    Google Scholar 

  • Egg, R., Pearson, F. S., Cleland, C. M. & Lipton, D. S. (2000). Evaluations of correctional treatment programs in Germany: A review and meta-analysis. Substance Use and Misuse 35, 1967–2009.

    Google Scholar 

  • Farrington, D. & Welsh, B. (2003). Family-based prevention of offending: A meta-analysis. Australian and New Zealand Journal of Criminology 36, 127–151.

    Article  Google Scholar 

  • Gallagher, C. A., Wilson, D. B., Hirschfield, P., Coggeshall, M. B. & MacKenzie, D. L. (1999). A quantitative review of the effects of sex offender treatment on sexual reoffending. Corrections Management Quarterly 3, 19–29.

    Google Scholar 

  • Garrett, C. J. (1985). Effects of residential treatment on adjudicated delinquents: A meta-analysis. Journal of Research in Crime and Delinquency 22(4), 287–308.

    Google Scholar 

  • Gendreau, P. & Goggin, C. (1996). Principles of effective programming with offenders. Forum on Corrections Research 8(3), 38–40.

    Google Scholar 

  • Gendreau, P., Goggin, C. & Cullen, F. (1999). The effects of prison sentences on recidivism. Ottawa, ON: Solicitor General Canada.

    Google Scholar 

  • Gensheimer, L. K., Mayer, J. P., Gottschalk, R. & Davidson, W. S. (1986). Diverting youth from the juvenile justice system: A meta-analysis of intervention efficacy. In S. J. Apter & A. P. Goldstein (Eds.), Youth violence (39–57). New York: Pergamon Press.

    Google Scholar 

  • Gorman, D. M. (2003). The best of practices, the worst of practices: The making of science-based primary prevention programs. Psychiatric Services 54(8), 1087–1089.

    Article  Google Scholar 

  • Gottschalk, R., Davidson W. S. II, Gensheimer, L. K. & Mayer, J. P. (1987). Community based interventions. In H. C. Quay (Ed.), Handbook of juvenile delinquency (266–89). New York: Wiley and Sons.

    Google Scholar 

  • Hall, G. C. N. (1995). Sexual offender recidivism revisited: A meta-analysis of recent treatment studies. Journal of Consulting and Clinical Psychology 63(5), 802–9.

    Article  Google Scholar 

  • Hanson, R. K., Gordon, A., Harris, A. J. R., Marques, J. K., Murphy, W., Quinsey, V. L. & Seto, M. C. (2002). First report of the collaborative outcome data project on the effectiveness of psychological treatment for sex offenders. Sexual Abuse: Journal of Research and Treatment 14, 169–195.

    Google Scholar 

  • Huebner, T. (2000). Theory-based evaluation: Gaining a shared understanding between school staff and evaluators. New Directions in Evaluation 87.

  • Izzo, R. L. & Ross, R. R. (1990). Meta-analysis of rehabilitation programs for juvenile delinquents. Criminal Justice and Behavior 17(1), 134–42.

    Google Scholar 

  • Kaufman, P. (1986). Meta-analysis of juvenile delinquency prevention programs. Unpublished paper. Claremont, CA: Claremont Graduate School.

    Google Scholar 

  • Latimer, J. (2001). A meta-analytic examination of youth delinquency, family and recidivism. Canadian Journal of Criminology and Criminal Justice 43(2).

  • Latimer, J., Dowden, C. & Muise, D. (2001). The effectiveness of restorative justice practices: A meta-analysis. Ottawa: Department of Justice.

    Google Scholar 

  • Latimer, J., Dowden, C. & Morton, K. (2004). Treating youth in conflict with the law: A new meta-analysis. Ottawa: Research and Statistics Division, Department of Justice Canada.

    Google Scholar 

  • Lipsey M. W. (1992). In Cook, T. C. Cooper, H. Cordray, D. S. Hartmann, H. Hedges, L. V. Light, R. L. Louis, T. A. & Mosteller F. M. (Eds.), Meta-analysis for explanation (83–127). New York: Russell Sage.

    Google Scholar 

  • Lipsey, M. W. (1995). What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents? In McGuire J. (Ed.), What works? Reducing reoffending. New York: Wiley.

    Google Scholar 

  • Lipsey, M. W. (2003). The good, the bad, and the ugly: The potential confounding role of moderators in meta-analysis. Annals of the American Academy of Political and Social Science.

  • Lipsey, M. W. & Wilson, D. B. (1998). Effective intervention for serious juvenile offenders: A synthesis of research. In R. Loeber & D. Farrington (Eds.), Serious and violent juvenile offenders: Risk factors and successful interventions (pp. 313–345). Thousand Oaks: Sage.

    Google Scholar 

  • Lipsey, M. & Wilson, D. (2001). Practical meta-analysis. Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Lipton D. S., Pearson, F. S., Cleland, C. M. & Yee, D. (2003). The effects of therapeutic communities and milieu therapy on recidivism: Meta-analytic findings from the correctional drug abusement treatment effectiveness (CDATE) study. In J. McGuire (Ed.), Offender rehabilitation and treatment: Effective programmes and policies to reduce reoffending (pp. 39–77). London: John Wiley.

    Google Scholar 

  • Loesel, F. & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: A systematic review of randomized evaluations. Annals of the American Academy of Political and Social Science 587, 84–109.

    Google Scholar 

  • Loesel, F. & Koferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In H. Wegener, F. Losel & J. Haisch (Eds.), Criminal behavior and the justice system (334–55). New York: Springer-Verlag.

    Google Scholar 

  • Loesel, F. & Schmucker, M. (2005). The effectiveness of treatment for sexual offenders: A comprehensive meta-analysis. Journal of Experimental Criminology 1, 117–146.

    Google Scholar 

  • Love A. J. (1991). Internal evaluation: Building organizations from within. Newbury Park, CA: Sage.

    Google Scholar 

  • MacKenzie, D. L., Wison, D. B. & Kider, S. (2001). Effects of correctional boot camps on offending. Annals of the American Academy of Political and Social Science 578, 126–143.

    Google Scholar 

  • Mayer, J. P., Gensheimer, L. K., Davidson, W. S. II, & Gottschalk, R. (1986). Social learning treatment within juvenile justice: A meta-analysis of impact in the natural environment. In S. J. Apter & A. P. Goldstein (Eds.), Youth violence (24–38). New York: Pergamon Press.

    Google Scholar 

  • Pearson, F. S. & Lipton D. S. (1999). A meta-analytic review of the effectiveness of corrections-based treatments for drug abuse. The Prison Journal 79, 384–410.

    Article  Google Scholar 

  • Pearson, F., Lipton, D., Cleland, C. & O'Kane, J. (1995). Meta-analysis on the effectiveness of correctional treatment: Another approach and extension of the time frame to 1994 — A progress report. Presentation at the American Society of Criminology Annual Meeting, Boston, Massachusetts, November 15th.

  • Pearson, F. S., Lipton, D. S. & Cleland, C. M. (1996). Some preliminary findings from the CDATE Project. Presentation at the American Society of Criminology, Chicago, Illinois, November.

  • Petrosino, A. J. (1997). What works? Revisited again: A meta-analysis of randomized experiments in individually-focused crime reduction interventions. Ph.D. dissertation, Rutgers University. Ann Arbor, MI: University Microfilms.

  • Petrosino, A., Turpin-Petrosino, C. & Buehler J. (2003). Scared straight and other juvenile awareness programs for preventing juvenile deliquency: A systematic review of the randomized experimental evidence. Annals of the American Academy of Political and Social Science 589, 41–62.

    Google Scholar 

  • Redondo, S., Sánchez-Meca, J. & Garrido, V. (2001). Treatment of offenders and recidivism: Assessment of the effectiveness of programmes applied in Europe. Psychology in Spain 5, 47–62.

    Google Scholar 

  • Rincon, P. (2004). Secrecy penalizes cancer patients. Online at: http://news.bbc.co.uk/1/hi/sci/tech/3632882.stm (Last accessed on September 10, 2005).

  • Roberts, A. R. & Camasso, M. J. (1991). The effect of juvenile offender treatment programs on recidivism: A meta-analysis of 46 studies. Notre Dame Journal of Law, Ethics and Public Policy 5(2), 421–41.

    Google Scholar 

  • Rosenthal, R. (1976). Experimenter effects in behavioral research. New York: John Wiley.

    Google Scholar 

  • Shepherd, J. P. (2003). Explaining feast or famine in randomised field trials: Medical science and criminology compared. Evaluation Review 27, 290–315.

    Article  Google Scholar 

  • Sherman, L. W. (1998). Evidence-based policing. Washington, DC: Police Foundation.

    Google Scholar 

  • Sherman, L. W. (2004). Research and policing: The infrastructure and political economy of federal funding. The Annals of the American Academy of Political and Social Science 593, 156–78.

    Article  Google Scholar 

  • US Bureau of Justice Assistance, Center for Program Evaluation, 2005, Online at: http://www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_m.htm (last accessed on September 10, 2005).

  • US National Institute of Health, (1998). NIH Policy for Data Safety Monitoring, online at http://grants.nih.gov/grants/guide/notice-files/not98-084.html (last accessed on September 10, 2005).

  • Visher, C. & Weisburd, D. (1997). Identifying what works: Recent trends in crime prevention strategies. Crime, Law and Social Change 28, 223–42.

    Article  Google Scholar 

  • Weisburd, D. L. (1996). “Preface,” In L. Greene (Ed.), Policing places with drug problems. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Weisburd, D., Sherman, L. & Petrosino, A. J. (1990). Registry of randomized experiments in criminal sanctions, 1950–1983. Los Altos, CA: Sociometrics Corporation.

    Google Scholar 

  • Weisburd, D. L., Petrosino, A. J. & Mason, G. (1993). Design sensitivity in criminal justice experiments: Reassessing the relationship between sample size and statistical power. Crime & justice: An annual review of research (vol. 17). Chicago: University of Chicago Press.

    Google Scholar 

  • Wells-Parker, E., Bangert-Drowns, R., McMillen, R. & Williams, M. (1995). Final results from a meta-analysis of remedial interventions with drink/drive offenders. Addiction 90, 907–926.

    Article  Google Scholar 

  • Wilson, D. B. (2001). Effect size determination program. Online at http://mason.gmu.edu/∼dwilsonb/downloads/es_calculator.zip (last accessed Nov. 16, 2005).

  • Wilson, S. J. & Lipsey, M. W. (2000). Wilderness challenge programs for delinquent youth: A meta-analysis of outcome evaluations. Evaluation and Program Planning 23, 1–12.

    Article  Google Scholar 

  • Wilson, D. B., Gallagher, C. A. & MacKenzie, D. L. (2000). A meta-analysis of corrections-based education, vocation, and work programs for adult offenders. Journal of Research in Crime and Delinquency 37, 347–368.

    Google Scholar 

  • Whitehead, J. T. & Lab, S. P. (1989). A meta-analysis of juvenile correctional treatment. Journal of Research in Crime and Delinquency 26, 276–295.

    Google Scholar 

  • Woolfenden, S. R., Williams, K. & Peat, J. (2003). Family and parenting interventions in children and adolescents with conduct disorder and delinquency aged 10–17. Cochrane Review. Oxford, UK: Update Software.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Learning Innovations at WestEd, 200 Unicorn Park Drive, 4th floor, Woburn, MA, 01801-3324, USA

    Anthony Petrosino

  2. USC School of Social Work, 669 West 34th Street, Los Angeles, CA, 90089-0411, USA

    Haluk Soydan

Authors
  1. Anthony Petrosino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haluk Soydan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anthony Petrosino.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Petrosino, A., Soydan, H. The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research. J Exp Criminol 1, 435–450 (2005). https://doi.org/10.1007/s11292-005-3540-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-005-3540-8

Key words

Search

Navigation