Abstract
In the rising quest for evidence-based interventions, recent research often does not give adequate attention to “nonspecific therapeutic factors,” including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and “dose” of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically:
-
1.
Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention;
-
2.
Planned (rather than post hoc) analyses to explore whether and how a given treatment’s specific ingredients or the overall intensity of its “dose” are related to treatment outcomes;
-
3.
More studies conducting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change
-
4.
When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client’s treatment expectations, changes in self-efficacy), and
-
5.
When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).
Similar content being viewed by others
References
Abikoff, H., Hechtman, L., Klein, R. G., Weiss, G., Fleiss, K., Etcovitch, J., et al. (2004). Symptomatic improvement in children with ADHD treated with long-term methylphenidate and multimodal psychosocial treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 802–811.
Barrett, P. M. (1998). Evaluation of cognitive-behavioral group treatments for childhood anxiety disorders. Journal of Clinical Child Psychology, 27, 459–468.
Barrett, P., Healy-Farrell, L., & March, J. S. (2004). Cognitive-behavioral family treatment of childhood obsessive-compulsive disorder: A controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 46–62.
Barrett, P. M., Dadds, M. R., & Rapee, R. M. (1996). Family treatment of childhood anxiety: A controlled trial. Journal of Consulting and Clinical Psychology, 64, 333–342.
Baskin, T. W., Tierney, S. C., Minami, T., & Wampold, B. E. (2003). Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71, 973–979.
Beidel, D. C., Turner, S. M., & Morris, T. L. (2000). Behavioral treatment of childhood social phobia. Journal of Consulting and Clinical Psychology, 68, 1072–1080.
Bickman, L., Heflinger, C. A., Northrup, D., Sonnichsen, S., & Schilling, S. (1998). Long-term outcomes of family caregiver empowerment. Journal of Child and Family Studies, 7, 269–282.
Birmaher, B., Brent, D. A., Kolko, D., Baugher, M., Bridge, J., Holder, D., et al. (2000), Clinical outcome after short-term psychotherapy for adolescents with major depressive disorder. Archives of General Psychiatry, 57, 29–36.
Brent, D. A., Holder, D., Kolko, D., Birmaher, B., Baugher, M., Roth, C., et al. (1997). A clinical psychotherapy trial for adolescent depression comparing cognitive, family, and supportive therapy. Archives of General Psychiatry, 54, 877–885.
Chadwick, O., Momcilovic, N., Rossiter, R., Stumbles, E., & Taylor, E. (2001). A randomized trial of brief individual versus group parent training for behaviour problems in children with severe learning disabilities. Behavioural and Cognitive Psychotherapy, 29, 151–167.
Chamberlain, P., & Reid, J. B. (1998). Comparison of two community alternatives to incarceration for chronic juvenile offenders. Journal of Consulting and Clinical Psychology, 66, 624–633.
Chambless, D., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.
Clarke, G. N., Hornbrook, M., Lynch, F., & Polen, M. (2002). Group cognitive-behavioral treatment for depressed adolescent offspring of depressed parents in a health maintenance organization. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 305–313.
Clarke, G. N., Rohde, P., Lewinsohn, P. M., Hops, H., & Seeley, J. R. (1999). Cognitive-behavioral treatment of adolescent depression: Efficacy of acute group treatment and booster sessions. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 272–279.
Cobham, V. E., Dadds, M. R., & Spence, S. H. (1998). The role of parental anxiety in the treatment of childhood anxiety. Journal of Consulting and Clinical Psychology, 66, 893–905.
Cohen, J. (1988). Statistical power analyses for the behavioral sciences (2nd edn.). Hillsdale, NJ: Lawrence Erlbaum.
Cohen, J. A., & Mannarino, A. P. (1996). A treatment outcome study for sexually abused preschool children. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 42–50.
Cohen, N. J., Muir, E., Parker, C. J., Brown, M., Lojkasek, M., Muir, R., et al. (1999). Watch, wait and wonder: Testing the effectiveness of a new approach to mother-infant psychotherapy. Infant Mental Health Journal, 20, 429–451.
Connolly, L., Sharry, J., & Fitzpatrick, C. (2001). Evaluation of a group treatment programme for parents of children with behavioural disorders. Child and Adolescent Mental Health, 6, 159–165.
Dadds, M. R., Spence, S. H., Holland, D. E., Barrett, P. M., & Laurens, K. R. (1997). Prevention and early intervention for anxiety disorders: A controlled trial. Journal of Consulting and Clinical Psychology, 65, 627–635.
Dishion, T. J., & Andrews, D. W. (1995). Preventing escalation in problem behaviors with high-risk young adolescents: immediate and 1-year outcomes. Journal of Consulting and Clinical Psychology, 63, 538–548
dos Santos, E. L. C., Marturano, E. M., de Almeida Motta, A. M., & Giurlana, A. G. (2003). Treating boys with low school achievement and behavior problems: Comparison of two kinds of intervention. Psychological Reports, 92, 105–116.
Driskill, J. D. (2000). Structured child and parent groups with ADHD children: Evaluation of varying levels of parent involvement. Dissertation Abstracts International: B, 60(9–B), 4884.
Eddy, J. M., & Chamberlain, P. (2000). Family management and deviant peer association as mediators of the impact of treatment condition on youth antisocial behavior. Journal of Consulting and Clinical Psychology, 68, 857–863.
Eddy, J. M., Whaley, R. B., & Chamberlain, P. (2004). The prevention of violent behavior by chronic and serious male juvenile offenders: A 2-year follow-up of a randomized clinical trial. Journal of Emotional and Behavioral Disorders, 12, 2–8.
Eisler, I., Dare, C., Hodes, M., Russell, G., Dodge, E., & Le Grange, D. (2000). Family therapy for adolescent anorexia nervosa: The results of a controlled comparison of two family interventions. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 727–736.
Flannery-Schroeder, E. C., & Kendall, P. C. (2000). Group and individual cognitive-behavioral treatments for youth with anxiety disorders: A randomized clinical trial. Cognitive Therapy and Research, 24, 251–278.
Forgatch, M. S., & DeGarmo, D. S. (1999). Parenting through change: An effective prevention program for single mothers. Journal of Consulting and Clinical Psychology, 67, 711–724.
Hebert, D. A. (2002). Evaluating cognitive-behavioral group treatment for disruptive adolescents in a special school. Dissertation Abstracts International: B, 63(1–B), 528.
Heflinger, C. A., Bickman, L., Northrup, D., & Sonnichsen, S. (1997). A theory-driven intervention and evaluation to explore family caregiver empowerment. Journal of Emotional and Behavioral Disorders, 5, 184–191.
Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821–833.
Hoagwood, K., Burns, B., Kiser, L., Ringeisen, H., & Schoenwald, S. K. (2001). Evidence-based practice in child and adolescent mental health services. Psychiatric Services, 52, 1179–1189.
Hoath, F. E., & Sanders, M. R. (2002). A feasibility study of enhanced group triple P—Positive Parenting Program for parents of children with attention-deficit(hyperactivity disorder. Behaviour Change, 19, 191–206.
Huey, S. J., Jr., Henggeler, S. W., Rowland, M. D., Halliday-Boykins, C. A., Cunningham, P. B., Pickrel, S. G., et al. (2004) Multisystemic therapy effects on attempted suicide by youths presenting psychiatric emergencies. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 83–90
Jensen, P. S., Hoagwood, K., & Trickett, E. (1999). Ivory Towers or Earthen Trenches? Community collaborations to foster real world research. Applied Developmental Science, 3, 306–212.
Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with autism: A randomized controlled trial to evaluate a caregiver-based intervention program in community day-care centers. Journal of Developmental and Behavioral Pediatrics, 19, 326–334.
Kaminer, Y., Blitz, C., Burleson, J. A., Kadden, R. M., & Rounsaville, B. J. (1998). Measuring treatment process in cognitive-behavioral and interactional group therapies for adolescent substance abusers. Journal of Nervous and Mental Diseases, 186, 407–413.
Kaminer, Y., Burleson, J. A., Blitz, C., Sussman, J., & Rounsaville, B. J. (1998). Psychotherapies for adolescent substance abusers: A pilot study. Journal of Nervous and Mental Diseases, 186, 684–690.
Kazdin, A. E. (2000). Developing a research agenda for child and adolescent psychotherapy. Archives of General Psychiatry, 57, 829–837.
Kazdin, A. E., & Bass, D. (1989). Power to test differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147.
Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: Methodological issues and research recommendations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 44(8), 1116–1129.
Kazdin, A. E., & Whitley, M. K. (2003). Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology, 71, 504–515.
King, N. J., Tonge, B. J., Mullen, P., Myerson, N., Heyne, D., Rollings, S., et al. (2000). Treating sexually abused children with posttraumatic-stress symptoms. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 1347–1355.
Last, C. G., Hansen, C., & Franco, N. (1998). Cognitive-behavioral treatment of school phobia. Journal of the American Academy of Child and Adolescent Psychiatry, 37, 404–411.
Liddle, H. A., Dakof, G. A., Parker, K., Diamond, G. S., Barrett, K., & Tejeda, M. (2001). Multidimiensional family therapy for adolescent drug abuse: Results of a randomized clinical trial. American Journal of Drug and Alcohol Abuse, 27, 651–688.
McArdle, P., Moseley, D., Quibell, T., Johnson, R., Allen, A., Hammal, D., et al. (2002). School-based indicated prevention: a randomized trial of group therapy. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43, 705–712.
Mendlowitz, S. L., Manassis, K., Bradley, S., Scapillato, D., Miezitis, S., & Shaw, B. F. (1999). Cognitive behavioral group treatments in childhood anxiety disorders: The role of parental involvement. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1223–1229.
Merkel, K. M. (2000). Effect of treatment of distress symptoms in child victims of violent crime: A comparison of supportive therapy and cognitive-behavioral therapy. Dissertation Abstracts International: B, 60(9-B), 4897.
Montgomery, P., Stores, G., & Wiggs, L. (2004). The relative efficacy of two brief treatments for sleep problems in young learning disabled (mentally retarded) children: A randomised controlled trial. Archives of Disease in Childhood, 89, 125–130.
Muris, P., Meesters, C., & van Melick, M. (2002). Treatment of childhood anxiety disorders; A preliminary comparison between cognitive-behavioral group therapy and a psychological placebo intervention. Journal of Behavior Therapy and Experimental Psychiatry, 33, 143–158.
Murray, L., Cooper, P. J., Wilson, A., & Romaniuk, H. (2003). Controlled trial of the short- and long-term effect of psychological treatment of post-partum depression. 2. Impact on the mother–child relationship and child outcome. British Journal of Psychiatry, 182, 420–427.
Nicholson, J. M., & Sanders, M. R. (1999). Randomized controlled trial of behavioral family intervention for the treatment of child behavior problems in stepfamilies. Journal of Divorce and Remarriage, 30, 1–23.
Ost, L. G., Svensson, L., Hellstrom, K., & Lindwall, R. (2001). One-session treatment of specific phobias in youths: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 69, 814–824.
Robin, A. R., Siegel, P. T., Moye, A. W., Gilroy, M., Dennis, A. B., & Sikand, A. (1999). A controlled comparison of family versus individual therapy for adolescents with anorexia nervosa. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1482–1489.
Rossello, J., & Bernal, G. (1999). The efficacy of cognitive, behavioral, and interpersonal treatments for depression in Puerto Rican adolescents. Journal of Consulting and Clinical Psychology, 67, 734–745.
Scott, S., Spender, Q., Doolan, M., Jacobs, B., & Aspland, H. (2001). Multicentre controlled trial of parenting groups for childhood antisocial behaviour in clinical practice. British Medical Journal, 323, 194.
Silverman, W. K., Kurtines, W. M., Ginsburg, G. S., Weems, C. F., Rabian, B., & Sarafini, L. T. (1999). Contingency management, self-control, and education support in the treatment of childhood phobic disorders: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 67, 675–687.
Snyder, K. V., Kymissis, P., & Kessler, K. (1999). Anger management for adolescents. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1409–1416.
Sonuga-Barke, E. J. S., Daley, D., Thompson, M., Laver-Bradbury, C., & Weeks, A. (2001). Parent-based therapies for preschool attention-deficity(hyperactivity disorder: A randomized, controlled trial with a community sample. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 402–408.
Spence, S. H., Donovan, C., & Brechman-Toussaint, M. (2000). The treatment of childhood social phobia: The effectiveness of a social-skills-training-based, cognitive-behavioral intervention, with and without parental involvement. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 713–726.
Spoth, R. L., Redmond, C., & Shin, C. (2001). Randomized trial of brief family interventions for general populations: Adolescent substance-use outcomes four years following baseline. Journal of Consulting and Clinical Psychology, 69, 662– 642.
Storr, C. L., Ialongo, N. S., Kellam, S. G., & Anthony, J. C. (2002). A randomized controlled trial of two primary intervention strategies to prevent early onset tobacco smoking. Drug and Alcohol Dependence, 66, 51–60.
Taylor, T. K., Schmidt, F., Pepler, D., & Hodgins, C. (1998). A comparison of eclectic treatment with Webster-Stratton’s parents and children series in a children’s mental health center: A randomized controlled trial. Behavior Therapy, 29, 221–240.
Verduyn, C., Barrowclough, C., Roberts, J., Tarrier, N., & Harrington, R. (2003). Maternal depression and child behaviour problems: Randomised placebo-controlled trial of a cognitive-behavioural group intervention. British Journal of Psychiatry, 183, 342–348.
Wayland, L. A. L. (2003). Treatment of children with problem behaviours: The efficacy of conjoint behavioural consultation versus videotape therapy and the impact on parent–teacher collaboration. Dissertation Abstracts International: Section B: The Sciences and Engineering, 64(1-B), 404.
Weersing, V. R., & Weisz, J. R. (2002). Mechanisms of action of youth psychotherapy. Journal of Child Psychology & Psychiatry and Allied Disciplines, 43, 3–29.
Weisz, J. R., & Jensen, P. S. (1999). Efficacy and effectiveness of psychotherapy and pharmacotherapy with children and adolescents. Mental Health Services Research, 1, 125–158.
Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. American Psychologist, 47, 1578–1585.
Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in clinical trials. Psychological Bulletin, 130, 631–663.
Wiggs, L., & Stores, G. (1999). Behavioural treatment for sleep problems in children with severe learning disabilities and challenging daytime behaviour: Effect on daytime behaviour. Journal of Child Psychology and Psychiatry (Allied Disciplines, 40, 627–635.
Wolchik, S. A., Sandler, I. N., Millsap, R. E., Plummer, B. A., Greene, S. M., Anderson, E. R., et al. (2002). Six-year follow-up of preventive interventions for children of divorce: a randomized controlled trial. JAMA, 288, 1874–1881.
Wood, A., Harrington, R., & Moore, A. (1996). Controlled trial of a brief cognitive-behavioural intervention in adolescent patients with depressive disorders. Journal of Child Psychology and Psychiatry and Allied Disciplines, 37, 737–746.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jensen, P.S., Weersing, R., Hoagwood, K.E. et al. What Is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbelly. Ment Health Serv Res 7, 53–74 (2005). https://doi.org/10.1007/s11020-005-1965-3
Issue Date:
DOI: https://doi.org/10.1007/s11020-005-1965-3