Skip to main content
SpringerLink
Log in

Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design

  • Original Paper
  • Published:
Child & Youth Care Forum Aims and scope Submit manuscript

Abstract

Background

HealthWise South Africa: Life Skills for Adolescents (HW) is an evidence-based substance use and sexual risk prevention program that emphasizes the positive use of leisure time. Since 2000, this program has evolved from pilot testing through an efficacy trial involving over 7,000 youth in the Cape Town area. Beginning in 2011, through 2015, we are undertaking a new study that expands HW to all schools in the Metro South Education District.

Objective

This paper describes a research study designed in partnership with our South African collaborators that examines three factors hypothesized to affect the quality and fidelity of HW implementation: enhanced teacher training; teacher support, structure and supervision; and enhanced school environment.

Methods

Teachers and students from 56 schools in the Cape Town area will participate in this study. Teacher observations are the primary means of collecting data on factors affecting implementation quality. These factors address the practical concerns of teachers and schools related to likelihood of use and cost-effectiveness, and are hypothesized to be “active ingredients” related to high-quality program implementation in real-world settings. An innovative factorial experimental design was chosen to enable estimation of the individual effect of each of the three factors.

Results

Because this paper describes the conceptualization of our study, results are not yet available.

Conclusions

The results of this study may have both substantive and methodological implications for advancing Type 2 translational research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Adelman, H. S., & Taylor, L. (2003). On sustainability of project innovations as systemic change. Journal of Educational and Psychological Consultation, 14, 1–25.

    Article  Google Scholar 

  • August, G. J., Winters, K. C., Realmuto, G. M., Tarter, R., Perry, C., & Hektner, J. M. (2004). Moving evidence-based prevention programs from basic science to practice: “Bridging the efficacy-effectiveness interface”. Substance Use and Misuse, 39, 2017–2053.

    Article  PubMed  Google Scholar 

  • Bales, E. A., & Soren, S. A. (2000). Managing clinical knowledge for health care improvement. Yearbook of Medical Informatics, (0943-4747), 65–70.

    Google Scholar 

  • Bradley, S. A., Smith, E. A., Lai, M. H., Graham, J. W., & Caldwell, L. L. (2010, June). Sexual coercion-risk in South African adolescent male youth: Risk correlates and HealthWise program impacts. In E. A. Smith (Chair), Sex, drugs, and deviance: Understanding prevention in South Africa. Symposium at the Society for Prevention Research annual meeting, Denver, CO.

  • Bradshaw, C. P., Koch, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.

    Article  PubMed  Google Scholar 

  • Bronfenbrenner, U. (1995). Developmental ecology through space and time: A future perspective. In P. Moen, G. H. Elder Jr., & K. Luscher (Eds.), Examining lives in context: Perspectives on the ecology of human development (pp. 619–647). Washington, DC: The American Psychological Association.

    Chapter  Google Scholar 

  • Bumbarger, B. K., & Perkins, D. F. (2008). After randomized trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services, 3(2), 55–64.

    Google Scholar 

  • Burns, B. J., & Hoagwood, K. (Eds). (2005). Evidence-based practices Part II: Effecting change. Child and Adolescent Psychiatric Clinics of North America, 14(2), xv–xvii.

  • Caldwell, L. L., Baldwin, C. K., Walls, T., & Smith, E. A. (2004). Preliminary effects of a leisure education program to promote healthy use of free time among middle school adolescents. Journal of Leisure Research, 36(3), 310–335.

    Google Scholar 

  • Caldwell, L. L., Patrick, M., Smith, E. A., Palen, L., & Wegner, L. (2010). Influencing adolescent leisure motivation: Intervention effects of HealthWise South Africa. Journal of Leisure Research, 42, 203–220.

    Google Scholar 

  • Coffman, D. L., Smith, E. A., Flisher, A. J., & Caldwell, L. L. (2011). Effects of HealthWise South Africa on condom use self-efficacy. Prevention Science, 12, 162–172.

    Article  PubMed  Google Scholar 

  • Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., et al. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208–226.

    Article  PubMed  Google Scholar 

  • Collins, L. M., Chakraborty, B., Murphy, S. A., & Strecher, V. (2009a). Comparison of a phased experimental approach and a single randomized clinical trial for developing multicomponent behavioral interventions. Clinical Trials, 6, 5–15.

    Article  PubMed  Google Scholar 

  • Collins, L. M., Dziak, J. R., & Li, R. (2009b). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202–224.

    Article  PubMed  Google Scholar 

  • Collins, L. M., Murphy, S. A., Nair, V., & Strecher, V. (2005). A strategy for optimizing and evaluating behavioral interventions. Annals of Behavioral Medicine, 30, 65–73.

    Article  PubMed  Google Scholar 

  • Collins, L. M., Murphy, S. A., & Strecher, V. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent e-health interventions. American Journal of Preventive Medicine, 32, S112–S118.

    Article  PubMed  Google Scholar 

  • Conduct Problems Prevention Research Group. (2007). The fast track randomized controlled trial to prevent externalizing psychiatric disorders: Findings from grades 3 to 9. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 1263–1272.

    Article  Google Scholar 

  • Conduct Problems Prevention Research Group. (2010). The effects of a multi-year randomized clinical trial of a universal social-emotional learning program: The role of student and school characteristics. Journal of Consulting and Clinical Psychology, 78(2), 156–168.

    Article  Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation efforts out of control? Clincial Pschology Review, 18(1), 23–45.

    Article  Google Scholar 

  • Department of Basic Education. (2011). Curriculum and Assessment Policy Statement (CAPS). Life Orientation Grades 7–9, Final Draft. Pretoria: Department of Basic Education.

    Google Scholar 

  • Department of Health. (2000). National guidelines for the development of health promoting schools/sites in South Africa. Pretoria, South Africa: Author.

    Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1(3), 6–28.

    Google Scholar 

  • Dorrington, R., Johnson, L., Bradshaw, D., & Daniel, T. J. (2006). The demographic Impact of HIV/AIDS in South Africa: National and provincial indicators for 2006. Cape Town, South Africa: Centre for Actuarial Research, South African Medical Research Council, and Actuarial Society of South Africa.

    Google Scholar 

  • Drummond, M., O’Brien, B., Stoddart, G., & Torrance, G. (1997). Methods for economic evaluation of health care programmes. Oxford, NY: Oxford Medical Publications.

    Google Scholar 

  • Durlak, J. A., & DuPre, E. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

    Article  PubMed  Google Scholar 

  • Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.

    Article  PubMed  Google Scholar 

  • Dusenbury, L., Brannigan, R., Falco, M., & Lake, A. (2004). An exploration of fidelity of implementation in drug abuse prevention among five professional groups. Journal of Alcohol & Drug Education, 47, 4–19.

    Google Scholar 

  • Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313.

    Article  PubMed  Google Scholar 

  • Elliott, D. S., & Mihalik, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.

    Article  PubMed  Google Scholar 

  • Ennett, S. T., Ringwalt, C. L., Thorne, J., Rohrbach, L. A., Vincus, A., Simons-Rudolph, A., et al. (2003). A comparison of current practice in school-based substance use prevention programs with meta-analytic findings. Prevention Science, 4, 1–14.

    Article  PubMed  Google Scholar 

  • Fagan, A. A., Hanson, K., Hawkins, J. D., & Arthur, M. W. (2008). Bridging science to practice: Achieving prevention program implementation fidelity in the community youth development study. American Journal of Community Psychology, 41, 235–249.

    Article  PubMed  Google Scholar 

  • Fisher, R. A. (1935). The design of experiments. Oxford, England: Oliver & Boyd.

    Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

    Google Scholar 

  • Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon Model of Parent Management Training (PMTO). Behavior Therapy, 36, 3–14.

    Article  PubMed  Google Scholar 

  • Furr-Holden, C. D. M., Ialongo, N., Anthony, J. C., Petras, H., & Kellam, S. (2004). Developmentally inspired drug prevention: Middle school outcomes in a school-based randomized prevention trial. Drug and Alcohol Dependence, 73, 149–158.

    Article  PubMed  Google Scholar 

  • Gager, P. J., & Elias, M. J. (1997). Implementing prevention programs in high-risk environments: Application of the resiliency paradigm. American Journal of Orthopsychiatry, 67(3), 363–373.

    Article  PubMed  Google Scholar 

  • Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915–945.

    Article  Google Scholar 

  • Giles, S., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008). Measuring quality of delivery in a substance use prevention program. Journal of Primary Prevention, 29(6), 489–501.

    Article  PubMed  Google Scholar 

  • Goddard, R. D., & Goddard, Y. L. (2001). A multilevel analysis of teacher and collective efficacy. Teaching and Teacher Education, 17, 807–818.

    Article  Google Scholar 

  • Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research on Crime and Delinquency, 39, 3–35.

    Article  Google Scholar 

  • Graham, J. W., Flay, B. R., Johnson, C. A., Hansen, W. B., & Collins, L. M. (1984). Group comparability: A multiattribute utility measurement approach to the use of random assignment with small numbers of aggregated units. Evaluation Review, 8, 247–260.

    Article  Google Scholar 

  • Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. (2005). The study of implementation in school-based preventive interventions: Theory, research and practice. Rockville, MD: Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention.

    Google Scholar 

  • Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679.

    Article  PubMed  Google Scholar 

  • Harshbarger, C., Simmons, G., Coelho, H., Sloop, K., & Collins, C. (2006). An empirical assessment of implementation, adaptation, and tailoring: The evaluation of CDC’s national diffusion of VOICES/VOCES. AIDS Education and Prevention, 18(Suppl. A), 184–197.

    Article  PubMed  Google Scholar 

  • Jenson, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy of Adolescent Psychiatry, 42, 527–530.

    Article  Google Scholar 

  • Joint United Nations Programme on HIV/AIDS (UNAIDS). (2010). UNAIDS report on the global AIDS epidemic | 2010. UNAIDS/10.11E | JC195BE. UNAIDS. ISBN 978-92-9173-871-7.

  • Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Belmont, CA, US: Thomson Brooks/Cole Publishing.

    Google Scholar 

  • Knutson, N. M., Forgatch, M. S., & Rains, L. A. (2003). Fidelity of implementation rating system (FIMP): The training manual for PMTO [revised]. Eugene, OR: Oregon Social Learning Center.

    Google Scholar 

  • Kulis, S., Marsiglia, F. F., Elek, E., Dustman, P., Wagstaff, D. A., & Hecht, M. L. (2005). Mexican/Mexican American adolescents and keepin’ it REAL: An evidence-based substance use prevention program. Children and Schools, 3, 133–145.

    Article  Google Scholar 

  • Kumpfer, K. L., Alvarado, R., & Whiteside, H. O. (2003). Family-based interventions for substance use and misuse prevention. Substance Use and Misuse, 11–13, 1759–1787.

    Article  Google Scholar 

  • Leach, D. J., & Conto, H. (1999). The additional effects of process and outcome feedback following brief in-service teacher training. Educational Psychology, 19, 441–462.

    Article  Google Scholar 

  • Meintjies, H. (2010). Orphans of the AIDS epidemic? The extent, nature and circumstances of child-lead households in South Africa. AIDS Care, 22, 40–49.

    Article  Google Scholar 

  • Mihalic, S., Irwin, K., Fagan, A., Ballard, D., & Elliott, D. (2004). Successful program implementation: Lessons learned from Blueprints. Washington, DC: U.S. Department of Justice, Office of Justice Programs. Retrieved from www.ojp.usdo.gov/ojjdp.

  • National Conference on Health Promoting Schools. (2006). Executive summary of conference proceedings. South Africa: University of the Western Cape.

    Google Scholar 

  • Palen, L., Smith, E. A., Caldwell, L. L., Mathews, C., & Vergnani, T. (2009). Transitions to substance use and sexual intercourse among South African high school students. Substance Use and Misuse, 44(13), 1872–1887.

    Article  PubMed  Google Scholar 

  • Palen, L., Smith, E. A., Flisher, A. J., Caldwell, L. L., & Mpofu, E. (2006). Substance use and sexual behavior among South African eighth grade students. Journal of Adolescent Health, 39(5), 761–763.

    Article  PubMed  Google Scholar 

  • Patrick, M. E., Collins, L. M., Smith, E., Caldwell, L., Flisher, A., & Wegner, L. (2009). A prospective longitudinal model of substance use onset among South African adolescents. Substance Use & Misuse, 44, 647–662. PMCID: PMC2796627.

    Article  Google Scholar 

  • Payne, A. A., Gottfredson, D. C., & Gottfredson, G. D. (2006). School predictors of the intensity of implementation of school-based prevention programs. Prevention Science, 7, 225–237.

    Article  PubMed  Google Scholar 

  • Pettifor, A. E., Rees, H. V., Steffenson, A., Hlongwa-Madikizela, L., MacPahil, C., Vermaak, K., et al. (2004). HIV and sexual behavior among young South Africans: A national survey of 15–24 year olds. Johannesburg, South Africa: Reproductive Health Research Unit, University of the Witwatersrand.

    Google Scholar 

  • Reddy, S. P., Sewpaul, J. S., Koopman, F., Funani, N. I., Sifunda, S., Josie, J., et al. (2010). Umthente Uhlaba Usamila—The South African Youth Risk Behaviour Survey 2008. Cape Town: South African Medical Research Council.

    Google Scholar 

  • Rohrbach, L. A., Grana, R., Sussman, S., & Valente, T. W. (2006). Type II translation. Transporting prevention interventions from research to real-world settings. Evaluation and the Health Professions, 29, 302–333.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton, Mifflin & Co.

    Google Scholar 

  • Shisana, O., Rehle, T., Simbayi, L. C., Parker, W., Zuma, K., Bhana, A., et al. (2005). South African national HIV prevalence, HIV incidence, behaviour and communication survey, 2005. Cape Town, South Africa: HSRC Press.

    Google Scholar 

  • Smith, E. A., Palen, L., Caldwell, L. L., Graham, J. W., Flisher, A. J., Wegner, L., et al. (2008). Substance use and sexual risk prevention in Cape Town, South Africa: An evaluation of the HealthWise program. Prevention Science, 9(4), 311–321.

    Article  PubMed  Google Scholar 

  • Snyder, J., Reid, J., Stoolmiller, M., Howe, G., Brown, H., Dagne, G., et al. (2006). The role of behavior observation in measurement systems for randomized prevention trials. Prevention Science, 7, 43–56.

    Article  PubMed  Google Scholar 

  • Storr, C., Ialongo, N., Anthony, J., & Kellam, S. (2002). A randomized prevention trial of early onset tobacco use by school and family-based interventions implemented in primary school. Drug and Alcohol Dependence, 66, 51–60.

    Article  PubMed  Google Scholar 

  • Strecher, V. J., McClure, J. B., Alexander, G. W., Chakraborty, B., Nair, V. N., Konkel, J. M., et al. (2008). Web-based smoking cessation programs: Results of a randomized trial. American Journal of Preventive Medicine, 34, 373–381.

    Article  PubMed  Google Scholar 

  • Tibbits, M. K., Caldwell, L. L., Smith, E. A., & Wegner, L. (2009). The relation between profiles of leisure activity participation and substance use among South African youth. World Leisure Journal, 51, 150–159. NIHMSID: NIHMS163569.

    PubMed  Google Scholar 

  • Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: The Interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41, 171–181.

    Article  PubMed  Google Scholar 

  • Wegner, L., Flisher, A. J., Caldwell, L., Vergnani, T., & Smith, E. (2008). HealthWise South Africa: Cultural adaptation of a school-based risk prevention programme. Health Education Research, 23, 1085–1096.

    Article  PubMed  Google Scholar 

  • Woolf, S. H. (2008). The meaning of translational research and why it matters. JAMA, 299, 211–213.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

HealthWise was supported by NIDA grant 1R01DA029084-01A1. Preparation of this article was supported by NIDA Center Grant P50 DA100075, NIDA 1R01DA029084-01A1, and F31 DA028155.

Conflict of interest

We declare that our research team has no conflict of interest with our association with the Metro South Education District (MSED). Our relationship with the district does not involve grants (other than the one secured by the U.S. National Institutes of Health/National Institute on Drug Abuse for this study which the MSED endorsed), employment, consultancy, shared ownership, or any close relationship that would influence or alter interests, financial or otherwise, that would be affected by the publication of this paper.

Ethics statement

This study has been granted human subjects approval by the IRB boards at both Penn State University and the University of the Western Cape. We have followed and will follow all requirements for the protection of human subjects as stipulated by the IRB application and approval.

Author information

Authors and Affiliations

  1. The Pennsylvania State University, University Park, PA, USA

    Linda L. Caldwell, Edward A. Smith, Linda M. Collins, John W. Graham & Mary Lai

  2. The University of the Western Cape, Cape Town, South Africa

    Lisa Wegner, Tania Vergnani & Joachim Jacobs

  3. The University of Cape Town, Cape Town, South Africa

    Catherine Matthews

  4. South African Medical Research Council, Cape Town, South Africa

    Catherine Matthews

Authors
  1. Linda L. Caldwell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edward A. Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linda M. Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John W. Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mary Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lisa Wegner
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tania Vergnani
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Catherine Matthews
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Joachim Jacobs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linda L. Caldwell.

Additional information

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Drug Abuse (NIDA) or the National Institutes of Health (NIH).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Caldwell, L.L., Smith, E.A., Collins, L.M. et al. Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design. Child Youth Care Forum 41, 119–136 (2012). https://doi.org/10.1007/s10566-011-9164-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10566-011-9164-4

Keywords

Search

Navigation