Skip to main content
Log in

Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

This paper presents new methods for synthesizing results from subgroup and moderation analyses across different randomized trials. We demonstrate that such a synthesis generally results in additional power to detect significant moderation findings above what one would find in a single trial. Three general methods for conducting synthesis analyses are discussed, with two methods, integrative data analysis and parallel analyses, sharing a large advantage over traditional methods available in meta-analysis. We present a broad class of analytic models to examine moderation effects across trials that can be used to assess their overall effect and explain sources of heterogeneity, and present ways to disentangle differences across trials due to individual differences, contextual level differences, intervention, and trial design.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Our argument below provides a partial justification due to space; the complete proof involves formulas for power based on noncentrality parameters, which in turn depend on sample size.

  2. The development in this part of the text is limited to interactions involving a binary covariate. The power for detecting a linear interaction with a continuous baseline measure can be compared to that of the main effect once a common calibration of “effect size” is established. Our choice is to scale the treatment variable to have the same variance as that of the continuous variable. The regression coefficient of the interaction term measures the difference in response under intervention and control for two covariate values separated by 1 standard deviation, i.e. \( E{S_{{Inter}}} = E(Y|T = 1,X = 1) - E(Y|T = 0,X = 1) - \left\{ {E(Y|T = 1,X = 0) - E(Y|T = 0) - E(Y|T = 0,X = 0)} \right\} \).To achieve the same power for detecting an effect size, ES ME for the main effect in a trial with equal allocations to intervention and control, we require \( E{S_{{Inter}}} = 2E{S_{{ME}}} \) . This is the identical result for the case of a dichotomous moderator variable presented in the text.

  3. In this argument we have ignored the differences in smaller degrees of freedom needed to test for this interaction effect across trials; nevertheless, the relationship ICC < 4(M-1)/N is still a very conservative bound.

References

  • Asparouhov, T., & Muthén, B. O. (2008). Multilevel mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 27–51). Charlotte, NC: Information Age.

    Google Scholar 

  • Barker, L. E., Luman, E. T., McCauley, M. M., & Chu, S. Y. (2002). Assessing equivalence: An alternative to the use of difference tests for measuring disparities in vaccination coverage. American Journal of Epidemiology, 156, 1056–1061.

    Article  PubMed  Google Scholar 

  • Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125.

    Article  PubMed  Google Scholar 

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate—a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B-(Statistical Methodology), 57, 289–300.

    Google Scholar 

  • Berlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A., & Feldman, H. I. (2002). Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 21, 371–387.

    Article  PubMed  Google Scholar 

  • Bridge, J. A., Iyengar, S., Salary, C. B., Barbe, R. P., Birmaher, B., Pincus, H. A., et al. (2007). Clinical response and risk for reported suicidal ideation and suicide attempts in pediatric antidepressant treatment: A meta-analysis of randomized controlled trials. Journal of the American Medical Association, 297, 1683–1696.

    Article  PubMed  CAS  Google Scholar 

  • Brown, C. H. (1993). Analyzing preventive trials with generalized additive models. American Journal of Community Psychology, 21, 635–664.

    Article  PubMed  CAS  Google Scholar 

  • Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiology paradigm. American Journal of Community Psychology, 27, 673–710.

    Article  PubMed  CAS  Google Scholar 

  • Brown, E. C., Hawkins, J. D., Arthur, M. W., Briney, J. S., & Abbott, R. D. (2007). Effects of Communities That Care on prevention services systems: Findings from the community youth development study at 1.5 years. Prevention Science, 8, 180–191.

    Article  PubMed  Google Scholar 

  • Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., et al. (2008a). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, 95, S74–S104.

    Article  Google Scholar 

  • Brown, C. H., Wang, W., & Sandler, I. (2008b). Examining how context changes intervention impact: The use of effect sizes in multilevel meta-analysis. Child Development Perspectives, 2, 198–205.

    Article  Google Scholar 

  • Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., et al. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48, 1013–1022.

    Article  PubMed  CAS  Google Scholar 

  • Cook, T. D. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage.

    Google Scholar 

  • Cooper, H. (1999). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles: Sage.

    Google Scholar 

  • Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176.

    Article  PubMed  Google Scholar 

  • Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage.

    Google Scholar 

  • Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.

    Article  PubMed  Google Scholar 

  • Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., et al. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.

    Article  PubMed  Google Scholar 

  • DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.

    Article  PubMed  CAS  Google Scholar 

  • Dillman Carpentier, F. R., Mauricio, A. M., Gonzales, N. A., Millsap, R. E., Meza, C. M., Dumka, L. E., et al. (2007). Engaging Mexican origin families in a school-based preventive intervention. Journal of Primary Prevention, 28, 521–546.

    Article  PubMed  Google Scholar 

  • Dishion, T. J., Spracklen, K. M., Andrews, D. W., & Patterson, G. R. (1996). Deviancy training in male adolescents friendships. Behavior Therapy, 27, 373–390.

    Article  Google Scholar 

  • Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm. Peer groups and problem behavior. American Psychologist, 54, 755–764.

    Article  PubMed  CAS  Google Scholar 

  • Dishion, T. J., Burraston, B., & Poulin, F. (2001). Peer group dynamics associated with iatrogenic effects in group interventions with high-risk young adolescents. In C. Erdley & D. W. Nangle (Eds.), Damon’s new directions in child development: The role of friendship in psychological adjustment (pp. 79–92). San Francisco: Jossey-Bass.

    Google Scholar 

  • Dolan, L. J., Kellam, S. G., Brown, C. H., Werthamer-Larsson, L., Rebok, G. W., Mayer, L. S., et al. (1993). The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. Journal of Applied Developmental Psychology, 14, 317–345.

    Article  Google Scholar 

  • Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. American Journal of Community Psychology, 25, 115–152.

    Article  PubMed  CAS  Google Scholar 

  • Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.

    Article  PubMed  Google Scholar 

  • Ennett, S. T., Tobler, N. S., Ringwalt, C. L., & Flewelling, R. L. (1994). How effective is drug abuse resistance education? A meta-analysis of Project DARE outcome evaluations. American Journal of Public Health, 84, 1394–1401.

    Article  PubMed  CAS  Google Scholar 

  • Faggiano, F., Vigna-Taglianti, F., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2005). School-based prevention for illicit drugs’ use. Cochrane Database of Systematic Reviews, 2, Art. No.: CD003020.

  • Faggiano, F., Vigna-Taglianti, F. D., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2008). School-based prevention for illicit drugs use: A systematic review. Preventive Medicine, 46, 385–396.

    Article  PubMed  Google Scholar 

  • Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.

    Article  PubMed  Google Scholar 

  • Fournier, J. C., DeRubeis, R. J., Hollen, S. D., Dimdjian, S., Amsterdam, J. D., Shelton, R. C., et al. (2010). Antidepressant drug effects and depression severity: Patient-level meta-analysis. Journal of the American Medical Association, 303, 47–53.

    Article  PubMed  CAS  Google Scholar 

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.

    Google Scholar 

  • Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.

    Google Scholar 

  • Glass, T. A., & McAtee, M. J. (2006). Behavioral science at the crossroads in public health: Extending horizons, envisioning the future. Social Science & Medicine, 62, 1650–1671.

    Article  Google Scholar 

  • Glass, G. V., & Smith, M. L. (1978). Meta-analysis of research on the relationship of class-size and achievement: The Class Size and Instruction Project. National Institute of Education (DHEW), Washington, DC.

  • Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

    Google Scholar 

  • Hastie, T., & Tibshirani, R. (1990). Generalized additive models (1st ed.). London: Chapman and Hall.

    Google Scholar 

  • Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic.

    Google Scholar 

  • Higgins, J. P. T., & Green, S. (Eds.). (2008). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]: The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org

  • Higgins, J. P., Whitehead, A., Turner, R. M., Omar, R. Z., & Thompson, S. G. (2001). Meta-analysis of continuous outcome data from individual patients. Statistics in Medicine, 20, 2219–2241.

    Article  PubMed  CAS  Google Scholar 

  • Hofer, S. M., & Piccinin, A. M. (2009). Integrative data analysis through coordination of measurement and analysis protocol across independent longitudinal studies. Psychological Methods, 14, 150–164.

    Article  PubMed  Google Scholar 

  • Horowitz, J. L., & Garber, J. (2006). The prevention of depressive symptoms in children and adolescents: A meta-analytic review. Journal of Consulting and Clinical Psychology, 74, 401–415.

    Article  PubMed  Google Scholar 

  • Howe, G. W., Reiss, D., & Yuh, J. (2002). Can prevention trials test theories of etiology? Development and Psychopathology, 14, 673–693.

    Article  PubMed  Google Scholar 

  • Hussong, A., Cai, L., Curran, P., Flora, D., Chassin, L., & Zucker, R. (2008). Disaggregating the distal, proximal, and time-varying effects of parent alcoholism on children’s internalizing symptoms. Journal of Abnormal Child Psychology, 36, 335–346.

    Article  PubMed  CAS  Google Scholar 

  • Ialongo, N. S., Werthamer, L., Kellam, S. G., Brown, C. H., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599–641.

    Article  PubMed  CAS  Google Scholar 

  • Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional & Behavioral Disorders, 9, 146–160.

    Article  Google Scholar 

  • Jané-Llopis, E., Hosman, C., Jenkins, R., & Anderson, P. (2003). Predictors of efficacy in depression prevention programmes. Meta-analysis. British Journal of Psychiatry, 183, 384–397.

    Article  PubMed  Google Scholar 

  • Kellam, S. G., & Langevin, D. J. (2003). A framework for understanding “evidence” in prevention research and programs. Prevention Science, 4, 137–153.

    Article  PubMed  Google Scholar 

  • Kellam, S. G., Koretz, D., & Moscicki, E. K. (1999). Core elements of developmental epidemiologically based prevention research. American Journal of Community Psychology, 27, 463–482.

    Article  PubMed  CAS  Google Scholar 

  • Kellam, S. G., Brown, C. H., Poduska, J. M., Ialongo, N. S., Wang, W., Toyinbo, P., et al. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95, S5–S28.

    Article  PubMed  Google Scholar 

  • Klein, A., & Moosbrugger, H. (2000). Maximum likelihood estimation of latent interaction effects with the LMS method. Psychometrika, 65, 457–474.

    Article  Google Scholar 

  • Klein, A. G., & Muthén, B. (2006). Modeling heterogeneity of growth depending on initial status. Journal of Educational and Behavioral Statistics, 31, 357–375.

    Article  Google Scholar 

  • Kraemer, H. C., Wilson, G. T., Fairburn, C. G., Agras, W. S., Kraemer, H. C., Wilson, G. T., et al. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883.

    Article  PubMed  Google Scholar 

  • Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.

    Article  Google Scholar 

  • Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review of Public Health, 17, 1–23.

    Article  PubMed  CAS  Google Scholar 

  • Muthén, B. (2001). Latent variable mixture modeling. In G. A. Marcoulides & R. E. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 1–33). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Muthén, B., & Asparouhov, T. (2003a). Advances in latent variable modeling, Part I: Integrating multilevel and structural equation modeling using Mplus. Manucript in Preparation.

  • Muthén, B., & Asparouhov, T. (2003b). Modeling interactions between latent and observed continuous variables using maximum-likelihood estimation in Mplus. Mplus Web Notes, #6. Retrieved from http://www.statmodel.com/download/webnotes/webnote6.pdf

  • Muthén, B. O., & Asparouhov, T. (2008). Growth mixture modeling: Analysis with non-Gaussian random effects. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis: A handbook of modern statistical methods (pp. 143–165). Boca Raton, FL: Chapman & Hall/CRC.

    Google Scholar 

  • Muthén, B. O., & Asparouhov, T. (2009). Multilevel regression mixture analysis. Journal of the Royal Statistical Society, 172, 639–657.

    Article  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1999). Mplus user’s guide version 2 (2nd ed.). Los Angeles: Authors.

    Google Scholar 

  • Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical & Experimental Research, 24, 882–891.

    Article  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (2007). Mplus: Statistical analysis with latent variables: User’s guide (4th ed.). Los Angeles: Authors.

    Google Scholar 

  • Muthén, B. O., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55, 463–469.

    Article  PubMed  Google Scholar 

  • O’Connell, M. E., Boat, T., & Warner, K. E. (Eds.). (2009). Mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academy.

    Google Scholar 

  • Shadish, W. R., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883–893.

    Article  PubMed  Google Scholar 

  • Shrout, P. E. (2009). Short and long views of integrative data analysis: Comments on contributions to the special issue. Psychological Methods, 14, 177–181.

    Article  PubMed  Google Scholar 

  • Sloboda, Z., Stephens, R. C., Stephens, P. C., Grey, S. F., Teasdale, B., Hawthorne, R. D., et al. (2009). The Adolescent Substance Abuse Prevention Study: A randomized field trial of a universal substance abuse prevention program. Drug and Alcohol Dependence, 102, 1–10.

    Article  PubMed  Google Scholar 

  • Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.

    Article  PubMed  CAS  Google Scholar 

  • Tein, J.-Y., Sandler, I. N., MacKinnon, D. P., & Wolchik, S. A. (2004). How did it work? Who did it work for? Mediation in the context of a moderated prevention effect for children of divorce. Journal of Consulting & Clinical Psychology, 72, 617–624.

    Article  Google Scholar 

  • Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537–567.

    Google Scholar 

  • Van Horn, M. L., Jaki, T., Masyn, K., Ramey, S. L., Smith, J. A., & Antaramian, S. (2009). Detecting differential effects: Applying regression mixture models to identify variations in the influence of family resources on academic achievement. Developmental Psychology, 45, 1298–313.

    Article  PubMed  Google Scholar 

  • Wilson, D. B., & Lipsey, M. W. (2003). The role of method in treatment effectiveness research: Evidence from meta-analysis. In A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research (3rd ed., pp. 589–615). Washington, DC: American Psychological Association.

    Google Scholar 

  • Wolchik, S., Sandler, I., Weiss, L., & Winslow, E. (2007). New Beginnings: An empirically-based intervention program for divorced mothers to help children adjust to divorce. In J. M. Briesmeister & C. E. Schaefer (Eds.), Handbook of parent training: Helping parents prevent and solve problem behaviors (3rd ed., pp. 25–66). Hoboken, NJ: John Wiley & Sons.

    Google Scholar 

Download references

Acknowledgements

We would like to thank our colleagues in the Prevention Science and Methodology Group (PSMG) for their suggestions in the development of this paper. This project was funded by a National Institute on Drug Abuse supplement to the Prevention Science and Methodology Group for the US-EU Drug Abuse Prevention Project (R01MH040859; Brown, Sloboda, Muthén, Masyn, Wang), Robert Wood Johnson Foundation (No. 039223, 040371) for Sloboda, Stephens, Grey, Teasdale. The EU-Drug Abuse Prevention Project is funded by the European Commission (European Public Health programme 2002 grant # SPC 2002376, Faggiano, Vigna-Taglianti), and the parallel data analyses supported by the European Monitoring Centre for Drugs and Drug Addiction (Burkhart, CT.09.RES.005.1.0: Keller). We would also like to thank J G Perpich LLC for their support in the use of the NIDA International Virtual Collaboratory funded through N44DA000000-00409, for their logistical support in developing our international collaboration. A version of this paper was presented by the first author at the “Foundational Issues in Examining Subgroup Effects in Experiments” Interagency Federal Methodological Meeting: Subgroup Analysis in Prevention and Intervention Research, Washington, DC.

Author information

Authors and Affiliations

Authors

Consortia

Corresponding author

Correspondence to C. Hendricks Brown.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Brown, C.H., Sloboda, Z., Faggiano, F. et al. Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials. Prev Sci 14, 144–156 (2013). https://doi.org/10.1007/s11121-011-0207-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-011-0207-8

Keywords

Navigation