Skip to main content
SpringerLink
Log in

No effects in independent prevention trials: can we reject the cynical view?

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Recent studies suggest that the reported effect sizes of prevention and intervention trials in criminology are considerably larger when program developers are involved in a study than when trials are conducted by independent researchers. This paper examines the possibility that these differences may be due to systematic bias related to conflict of interest. A review of the evidence shows that the possibility of a substantial problem cannot be currently rejected. Based on a theoretical model about how conflict of interest may influence research findings, the paper proposes several strategies to examine empirically the extent of systematic bias related to conflict of interest. It also suggests that, in addition to improved standards for conducting and publishing future experimental studies, more research is needed on the extent of systematic bias in the existing body of literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Al-Marzouki, S., Evans, S., Marshall, T., & Roberts, I. (2005a). Are these data real? Statistical methods for the detection of data fabrication in clinical trials. BMJ, 331, 267–270.

    Article  Google Scholar 

  • Al-Marzouki, S., Roberts, I., Marshall, T., & Evans, S. (2005b). The effect of scientific misconduct on the results of clinical trials: a Delphi survey. Contemporary Clinical Trials, 26(3), 331–337.

    Article  Google Scholar 

  • Altman, D. G. (1996). Better reporting of randomised controlled trials: the consort statement. BMJ, 313(7057), 570–571.

    Google Scholar 

  • Babcock, L., Loewenstein, G., Issacharoff, S., & Camerer, C. (1995). Biased judgments of fairness in bargaining. The American Economic Review, 85(5), 1337–1343.

    Google Scholar 

  • Bauer, N. S., Lozano, P., & Rivara, F. P. (2007). The Effectiveness of the Olweus bullying prevention program in public middle schools: a controlled trial. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 40(3), 266–274.

    Google Scholar 

  • Beelmann, A., & Lösel, F. (2006). Child social skills training in developmental crime prevention: effects on antisocial behavior and social competence. Psycothema, 18(3), 603–610.

    Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA, 273(14), 1106–1112.

    Article  Google Scholar 

  • Campbell, M. K., Elbourne, D. R., & Altman, D. G. (2004). Consort statement: extension to cluster randomised trials. BMJ, 328(7441), 702–708.

    Article  Google Scholar 

  • Cho, H., Hallfors, D. D., & Sánchez, V. (2005). Evaluation of a high school peer group intervention for at-risk youth. Journal of Abnormal Child Psychology, 33(3), 363–374.

    Article  Google Scholar 

  • Cornish, D. B., & Clarke, R. V. G. (Eds.). (1986). The reasoning criminal: Rational choice perspectives on offending. New York: Springer.

  • Dana, J., & Loewenstein, G. (2003). A social science perspective on gifts to physicians from industry. JAMA, 290(2), 252–255.

    Article  Google Scholar 

  • de Graaf, I., Speetjens, P., Smit, F., de Wolff, M., & Tavecchio, L. (2008). Effectiveness of the Triple P positive parenting program on behavioral problems in children: a meta-analysis. Behav Modif, 32(5), 714–735.

    Article  Google Scholar 

  • Diekmann, A. (2007). Not the first digit! Using Benford’s Law to detect fraudulent scientific data. Journal of Applied Statistics, 34(3), 321–329.

    Article  Google Scholar 

  • DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002a). Effectiveness of mentoring programs for youth: a meta-analytic review. American Journal of Community Psychology, 30, 157–197.

    Article  Google Scholar 

  • DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002b). Effectiveness of mentoring programs for youth: a meta-analytic review. American Journal of Community Psychology, 30(2), 157–197.

    Article  Google Scholar 

  • Eggert, L. L., Seyi, C. D., & Nicholas, L. J. (1990). Effects of a school-based prevention program for potential high school dropouts and drug abusers. Substance Use & Misuse, 25(7), 773–801.

    Article  Google Scholar 

  • Eggert, L. L., Thompson, E. A., Herting, J. R., Nicholas, L. J., & Dickers, B. G. (1994). Preventing adolescent drug abuse and high school dropout through an intensive social network development program. American Journal of Health Promotion, 8(2), 202–215.

    Google Scholar 

  • Eisner, M., & Ribeaud, D. (2008). Markt, Macht und Wissenschaft; Kritische Überlegungen zur Deutschen Präventionsforschung. In E. Marks & W. Steffen (Eds.), Starke Jugend - Starke Zukunft (Ausgewählte Beiträge des 12. Deutschen Präventionstages, 18. und 19. Juni 2007) (pp. 173–191). Mönchengladbach: Forum Verlag Godesberg.

  • Eisner, M., Ribeaud, D., Jünger, R., & Meidert, U. (2007). Frühprävention von Gewalt und Aggression; Ergebnisse des Zürcher Interventions- und Präventionsprojektes an Schulen. Zürich: Rüegger.

    Google Scholar 

  • Ellickson, P. L. (1998). Preventing adolescent substance abuse: lessons from the project alert program. In J. Crane (Ed.), Social programs that work (pp. 201–257). New York: Russell Sage.

    Google Scholar 

  • Ellickson, P. L., & Bell, R. M. (1990). Drug prevention in junior high: a multi-site longitudinal test. Science, 247, 1299–1305.

    Article  Google Scholar 

  • Ellickson, P. L., McCaffrey, D. F., Ghosh-Dastidar, B., & Longshore, D. L. (2003). New inroads in preventing adolescent drug use: results from a large-scale trial of project ALERT in middle schools. American Journal of Public Health, 93(11), 1830–1836.

    Article  Google Scholar 

  • Elliott, D. S., & Mihalic, S. F. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5(1), 47–53.

    Article  Google Scholar 

  • Etzkowitz, H. (2003). Research groups as ‘Quasi-Firms’: the invention of the entrepreneurial university. Research Policy, 32(1), 109–121.

    Article  Google Scholar 

  • Farrington, D. P. (2003). Methodological quality standards for evaluation research. Annals of the American Academy of Political and Social Sciences, 587, 49–68.

    Article  Google Scholar 

  • Farrington, D. P., & Welsh, B. (2003). Family-based prevention of offending: a meta-analysis. Australian and New Zealand Journal of Criminology, 36(2), 127–151.

    Article  Google Scholar 

  • Felson, M. (1994). Crime and everyday life; insight and implications for society. Thousand Oaks: Pine Forge Press.

    Google Scholar 

  • Friedman, L. S., & Richter, E. D. (2004). Relationship between conflicts of interest and research results. Journal of General Internal Medicine, 19(1), 51–56.

    Article  Google Scholar 

  • Gandhi, A. G., Murphy-Graham, E., Petrosino, A., Chrismer, S. S., & Weiss, C. H. (2007). The devil is in the details: examining the evidence for “proven” school-based drug abuse prevention programs. Evaluation Review, 31(1), 43–74.

    Article  Google Scholar 

  • Gilbert, N. (1997). Advocacy research and social policy. Crime and Justice: A Review of Research, 22, 101–148.

    Article  Google Scholar 

  • Gorman, D. M. (2003). Alcohol & drug abuse: the best of practices, the worst of practices: the making of science-based primary prevention programs. Psychiatric Services, 54(8), 1087–1089.

    Article  Google Scholar 

  • Gorman, D. M. (2005a). Does measurement dependence explain the effects of the life skills training program on smoking outcomes? Preventive Medicine, 40(4), 479–487.

    Article  Google Scholar 

  • Gorman, D. M. (2005b). Drug and violence prevention: rediscovering the critical rational dimension of evaluation research. Journal of Experimental Criminology, 1(1), 39–62.

    Article  Google Scholar 

  • Gorman, D. M. (2006). Conflicts of interest in the evaluation and dissemination of drug use prevention programs. In J. Kleinig, & S. Einstein (Eds.), Intervening in drug use: Ethical challenges (pp. 171–187). Huntsville: Office of International Criminal Justice.

    Google Scholar 

  • Gorman, D. M., & Conde, E. (2007). Conflict of interest in the evaluation and dissemination of “Model” school-based drug and violence prevention programs. Evaluation and Program Planning, 30(4), 422–429.

    Article  Google Scholar 

  • Heinrichs, N., Hahlweg, K., Bertram, H., Kuschel, A., Naumann, S., & Harstick, S. (2006). Die langfristige Wirksamkeit eines Elterntrainings zur universellen Prävention kindlicher Verhaltensstörungen: Ergebnisse aus Sicht der Mütter und Väter. Zeitschrift für klinische Psychologie und Psychotherapie, 35(2).

  • Henggeler, S. W., Cinningham, P. B., Pickrel, S. G., Schoenwald, S. K., & Brondino, M. J. (1996). Multisystemic therapy: an effective violence prevention approach for serious juvenile offenders. Journal of Adolescence, 19(1), 47–61.

    Article  Google Scholar 

  • Hewitt, C. E., Mitchell, N., & Torgerson, D. J. (2008). Listen to the data when results are not significant. BMJ, 336(7634), 23–25.

    Article  Google Scholar 

  • Holyoak, K. J., & Simon, D. (1999). Bidirectional reasoning in decision making by constraint satisfaction. Journal of Experimental Psychology; General, 128(1), 3–31.

    Article  Google Scholar 

  • Kaptchuk, T. J. (2003). Effect of interpretive bias on research evidence. BMJ, 326(7404), 1453–1455.

    Article  Google Scholar 

  • Lipsey, M. W. (1995). What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents? In J. McGuire (Ed.), What works? Reducing reoffending (pp. 63–78). New York: Wiley.

    Google Scholar 

  • Lipsey, M. W., & Cullen, F. T. (2007). The effectiveness of correctional rehabilitation: a review of systematic reviews. Annual Review of Law and Social Science, 3, 297–320.

    Google Scholar 

  • Lipsey, M. W., Petrie, C., Weisburd, D., & Gottfredson, D. (2006). Improving evaluation of anti-crime programs: summary of a national research council report. Journal of Experimental Criminology, 2(3), 271–307.

    Article  Google Scholar 

  • Littell, J. (2005). Lessons from a systematic review of effects of multisystemic therapy. Children and Youth Services Review, 27(4), 445–463.

    Article  Google Scholar 

  • Littell, J., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for Social, Emotional, and Behavioral Problems in Youth Aged 10–17 (Report for the Campbell Collaboration) (Electronic Version). Accessed 14 January 2008 from http://www.sfi.dk/graphics/Campbell/Dokumenter/MST_Review/MULTISYSTEMIC%20TERAPHY%20-%20REVIEW.pdf.

  • Lock, S., Wells, F., & Farthing, M. (Eds.). (2001). Fraud and misconduct in medical research. London: BMJ Publishing Group.

  • Lösel, F., & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: a systematic review of randomized evaluations. The ANNALS of the American Academy of Political and Social Science, 587(1), 84–109.

    Article  Google Scholar 

  • Lösel, F., & Koferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In H. Wegener, F. Lösel, & J. Haisch (Eds.), Criminal behavior and the justice system: Psychological perspectives. New York: Springer.

    Google Scholar 

  • Lösel, F., & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: a systematic review of randomized evaluations. The ANNALS of the American Academy of Political and Social Science, 587(1), 84–109.

    Article  Google Scholar 

  • Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., et al. (1999). The researcher’s own therapy allegiances: a “Wild Card” in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6(1), 95–106.

    Article  Google Scholar 

  • Lundahl, B., Risser, H. J., & Lovejoy, M. C. (2006). A meta-analysis of parent training: moderators and follow-up effects. Clinical Psychology Review, 26(1), 86–104.

    Article  Google Scholar 

  • MacCoun, R. J. (1998). Biases in the interpretation and use of research statistics. Annual Review of Psychology, 49, 259–287.

    Article  Google Scholar 

  • MacCoun, R. J. (2005). Conflicts of interest in public policy research. In D. Moore, D. M. Cain, G. Loewenstein, & M. H. Bazerman (Eds.), Conflicts of interest; challenges and solutions in business, law, medicine, and public policy (pp. 233–262). Cambridge: Cambridge University Press.

    Google Scholar 

  • Martinson, B. C., Anderson, M. S., & de Vries, R. (2005). Scientists behaving badly. Nature, 435(9 June 2005), 737–738.

    Article  Google Scholar 

  • Messick, D. M., & Sentis, K. P. (1979). Fairness and preference. Journal of Experimental Social Psychology, 15(4), 418–434.

    Article  Google Scholar 

  • Moore, D., Tetlock, P. E., Tanlu, L., & Bazerman, M. H. (2006). Conflicts of interest and the case of auditor independence: moral seduction and strategic issue cycling. Academy of Management Review, 31(1), 10–29.

    Google Scholar 

  • Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of Triple P-positive parenting program using hierarchical linear modeling: effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11(3), 114–144.

    Article  Google Scholar 

  • Ogden, T., & Halliday-Boykins, C. A. (2004). Multisystemic treatment of antisocial adolescents in Norway: replication of clinical outcomes outside of the us. Child and Adolescent Mental Health, 9, 77–83.

    Article  Google Scholar 

  • Ogden, T., & Hagen, K. A. (2006). Multisystemic treatment of serious behaviour problems in youth: sustainability of effectiveness two years after intake. Child and Adolescent Mental Health, 11(3), 142–149.

    Article  Google Scholar 

  • Okike, K., Kocher, M. S., Mehlman, C. T., & Bhandari, M. (2007). Conflict of interest in orthopaedic research. an association between findings and funding in scientific presentations. J Bone Joint Surg Am, 89(3), 608–613.

    Article  Google Scholar 

  • Olweus, D. (1994). Bullying at school: basic facts and effects of a school based intervention program. Journal of Child Psychology and Psychiatry, 35(7), 1171–1190.

    Article  Google Scholar 

  • Perlis, R. H., Perlis, C. S., Wu, Y., Hwang, C., Joseph, M., & Nierenberg, A. A. (2005). Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. American Journal of Psychiatry, 162(10), 1957–1960.

    Article  Google Scholar 

  • Petrosino, A., & Soydan, H. (2005). The impact of program developers as evaluators on criminal recidivism: results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology, 1(4), 435–450.

    Article  Google Scholar 

  • Ranstam, J., Buyse, M., George, S. L., Evans, S., Geller, N. L., Scherrer, B., et al. (2000). Fraud in medical research: an international survey of biostatisticians. Controlled Clinical Trials, 21(5), 415–427.

    Article  Google Scholar 

  • Resnik, D. B. (2000). Financial interests and research bias. Perspectives on Science, 8(3), 255–285.

    Article  Google Scholar 

  • Russo, J. E., Carlson, K. A., Meloy, M., & Yong, K. (2007). The goal of consistency as a cause of information distortion. Johnson School Research Paper Series No. 04-07.

  • Sanchez, V., Steckler, A., Nitirat, P., Hallfors, D., Cho, H., & Brodish, P. (2007). Fidelity of implementation in a treatment effectiveness trial of reconnecting youth. Health Education Research, 22(1), 95–107.

    Article  Google Scholar 

  • Sanders, M. R. (1992). Every parent: A positive guide to children’s behavior. Sydney: Addison-Wesley.

    Google Scholar 

  • Sanders, M. R. (1999). Triple P-positive parenting program: towards an empirically validated multilevel parenting and family support strategy for the prevention of behaviour and emotional problems in children. Clinical Child and Family Psychology Review, 2(2), 71–89.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.

    Google Scholar 

  • Sherman, L. W. (2006). “to develop and test:” the inventive difference between evaluation and experimentation. Journal of Experimental Criminology, 2(3), 393–406.

    Article  Google Scholar 

  • Sherman, L. W., Farrington, D. P., Welsh, B. C., & MacKenzie, D. L. (Eds.). (2002). Evidence-based crime prevention. London: Routledge.

  • St Pierre, T. L., Osgood, D. W., Mincemoyer, C. C., Kaltreider, D. L., & Kauh, T. J. (2006). Results of an independent evaluation of project alert delivered in schools by cooperative extension. Prevention Science, 6(4), 305–317.

    Article  Google Scholar 

  • Sundell, K., Hansson, K., Löfholm, C. A., Olsson, T., Gustle, L. H., & Kadasjo, C. (2008). The transportability of multisystemic therapy to Sweden: short-term results from a randomized trial of conduct disordered youth. Journal of Family Psychology, 22(4), 550–560.

    Article  Google Scholar 

  • Wilson, S. J., & Lipsey, M. W. (2007). School-based interventions for aggressive and disruptive behavior: update of a meta-analysis. American Journal of Preventive Medicine, 33(2, Supplement 1), S130–S143.

    Article  Google Scholar 

  • Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003a). The effects of school-based intervention programs on aggressive behavior: a meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136–149.

    Article  Google Scholar 

  • Wilson, S. J., Lipsey, M. W., & Soydan, H. (2003b). Are mainstream programs for juvenile delinquency less effective with minority youth than majority youth? a meta-analysis of outcomes research. Research on Social Work Practice, 13(1), 2–26.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Criminology, Sidgwick Avenue, University of Cambridge, Cambridge, CB3 9DT, UK

    Manuel Eisner

Authors
  1. Manuel Eisner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel Eisner.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Eisner, M. No effects in independent prevention trials: can we reject the cynical view?. J Exp Criminol 5, 163–183 (2009). https://doi.org/10.1007/s11292-009-9071-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-009-9071-y

Keywords

Search

Navigation