Limitations of Randomized Control Designs in Psychotherapy Research

Shean, Glenn

doi:https://doi.org/10.1155/2014/561452

Advances in Psychiatry

On this page

Abstract Introduction Background Conclusion References Copyright Related Articles

Review Article | Open Access

Volume 2014 | Article ID 561452 | https://doi.org/10.1155/2014/561452

Limitations of Randomized Control Designs in Psychotherapy Research

Glenn Shean¹

Academic Editor: Christine M. Blasey

Received18 Jun 2014

Revised15 Sept 2014

Accepted15 Oct 2014

Published06 Nov 2014

Abstract

Despite the growing influence of lists of empirically supported therapies (ESTs) there are concerns about the design and conduct of this body of research. These concerns include limitations inherent in the requirements of randomized control trials (RCTs) that favor those psychotherapies that define problems and outcome in terms of uncomplicated symptoms. Additional concerns have to do with criteria for patient selection, lack of integration with research on psychotherapy process and effectiveness studies, limited outcome criteria, and lack of controls for experimenter bias. RCT designs have an important place in outcome research; however it is important to recognize that these designs also place restrictions on what and how psychotherapy can be studied. There is a need for large scale psychotherapy outcome research based on designs that allow for inclusion of process variables and the study of the effects of those idiographic approaches to therapy that do not lend themselves to RCT designs. Interpretative phenomenological analysis may provide a useful method for the evaluation of the effectiveness of idiographic approaches to psychotherapy where outcome is not understood solely in terms of symptom reduction.

1. Introduction

There are several issues related to the external validity of lists of empirically supported psychotherapies (ESTs) that should be considered before broad policy changes are instituted based on this body of research. First, the design requirements of randomized control trial (RCT) studies do not allow for adequate understanding of the multifaceted nature of many mental health problems or the complex interplay of individual differences, interpersonal processes, and range of potential outcomes that are inherent in psychotherapy practice. Second, the EST literature is based on studies characterized by unacknowledged sampling issues, methodological constraints, researcher bias, and limited outcome criteria [1–3].

2. Background

A white paper issued by a Task Force of the American Psychological Association in 1995 stated that “in order to remain competitive in the mental health services marketplace psychologists must provide evidence of therapeutic efficacy in the form of lists of empirically supported psychotherapies (ESTs) for specific disorders” [4]. The APA paper specified RCTs as the ideal design for studies of psychotherapy efficacy, in which “a treatment is manualized and demonstrated to be more effective than other treatments or placebo” [4]. A large number of efficacy studies have been published that are the basis for lists of ESTs [5]. These lists are the basis for proposals to differentiate between “psychological treatments” and “psychotherapy” [6]. As proposed by Barlow and Carl, for example, [6], the term “psychological treatment” would be restricted to ESTs designed to address specific diagnoses within a health-care context. “Psychotherapy” in turn would be limited to references to processes and procedures designed to resolve issues such as interpersonal and relationship problems and personal growth concerns and administered outside of the health-care system. Given the broad policy and social implications of this proposal, it is worthwhile to more closely examine the external validity of lists of ESTs.

2.1. RCT Designs

Double-blind randomized control trial designs are widely considered to be the “gold standard” for treatment efficacy studies [7]. RCTs are the most direct way of determining whether a cause-effect relation exists between treatment and outcome [8]. RCT designs include the following features [8]. (1) The first item is random allocation of participants to intervention and control groups, in order to minimize allocation bias and to balance known and unknown prognostic factors in the assignment of treatments. (2) Both patients and clinicians must remain unaware of which treatment (e.g., placebo control or a previously tested treatment positive control study) was given until the study is completed. (3) All intervention groups are treated identically except for the experimental treatment. (4) Patients are analyzed within the group to which they were allocated. (5) Analyses are designed to estimate the size of the difference in predicted outcomes between intervention and control groups. Valid studies using RCT designs also require representative samples of adequate size to support planned statistical analyses and close matching of treatment and control group patients based on defined criteria.

2.1.1. RCT Advantages and Limitations

The practice of patient selection based on uncomplicated DSM based symptoms, common to most RCT design EST studies, allows for enhanced matching of treatment and control groups and clarity of outcome criteria. On the other hand, this requirement restricts the population studied to a portion of the individuals who seek psychotherapy services, since many patients seek psychotherapy services for issues that transcend specific symptoms, and comorbidity of symptoms is common. The use of therapy manuals in RCT studies allows for enhanced control over the experimental treatment under study but restricts the types of psychotherapeutic interventions that can be studied. The requirements of RCT designs in effect help to ensure the reliability of reported effects but limit the manner in which problems, treatments, and outcome criteria are defined, understood, and assessed [3, 9–11]. The effects of therapy moderators are assumed to be randomized in RCT design studies based on the assumption that randomization of group assignment takes into account potential effects of known moderators of therapeutic effects such as level of distress, social impairment, comorbidity, self-reflectiveness, readiness for change, openness to experience, level of engagement, ability to verbalize feelings, social support, and coping [1–3, 7, 10, 11]. Differences in dispositional traits that are correlated with DSM diagnoses (e.g., negative affect, submissiveness, low affect tolerance, and introversion) are also assumed to be randomized in RCT designs [2, 3]. The assumption that randomization of patient assignment controls for the effects of therapy moderators and process variables is valid only if study sample sizes are large enough to allow for adequate levels of power for valid statistical comparisons. Many EST studies are based on relatively small sample sizes, justified by power estimates based on questionable effect sizes reported in previous non-double-blind psychotherapy efficacy studies. These estimates are likely to be inflated by the effects of “therapist allegiance.”

In contrast to most manualized psychotherapies now listed as ESTs, idiographic psychotherapies emphasize the importance of process variables as moderators of therapeutic effectiveness and tailor the application of therapeutic strategies and techniques to the particular characteristics and circumstances of each individual patient. These approaches cannot be manualized in a manner that can be applied in the same manner across patients. The goals of idiographic therapies are typically described in terms of changes that are more in line with the concept of recovery than symptom reduction. Idiographic therapies emphasize the multifaceted origins and nature of mental disorders and understand recovery as an ongoing process that requires flexibility on the part of the therapists, as well as responsibility, commitment, and motivation to change on the part of the patient. The goals of idiographic therapies are not limited to symptom reduction and include enhanced connection of memories, emotions, changes in patterns of problematic experiences, and relating in ways that enable patients to live more satisfying and effective symptom-free lives, to have richer and more satisfying relationships, and to find new ways to become more effective agents in their lives [1, 7, 11, 12].

2.2. Issues Related to ESTs

2.2.1. Sampling Practices

The practice of limiting RCT psychotherapy studies to populations with a single diagnosis and few if any comorbid problems does not mirror the realities of most clinical practice. Individuals who meet single uncomplicated symptom criteria are estimated to represent no more than about 20% of the overall population seeking psychotherapy services [2, 3, 13]. Surveys indicate that at least one-third to one-half of people seeking mental health treatment do not meet criteria for any one diagnostic category [2, 14]. When specific symptoms are the initial focus of treatment about one-half of the patients add new target complaints or change their complaints over the course of treatment [10, 14]. It is estimated that between 40% and 70% of individuals who present for treatment in clinical practice settings are excluded from EST studies because they do not meet the restrictive criteria required by RCT designs [3, 14]. Finally, many EST studies are conducted in university affiliated related clinics [13]. Participants are often enrolled in studies of behavioral or cognitive-behavioral therapy (CBT). The individuals selected for the study are not likely to be blind to the treatment condition and likely were referred because of prior information about the lead therapist’s reputation. In short, the patient samples in university affiliated studies are not representative of the population at large nor are they randomly selected.

2.2.2. Length of Therapy

The average number of sessions for published EST studies is sixteen [2, 3]. The assumption that mental health problems can be resolved in an average of 16 or fewer sessions is not consistent with evidence for a significant psychotherapy dose-response relationship [3, 9] or with field studies of CBT for depressed patients that report an average of 69 sessions [3]. In contrast to the EST literature on treatment for depression, the research team conducted by the National Institute of Mental Health Treatment of Depression Collaborative Research Program concluded that 16 weeks of cognitive-behavioral or interpersonal therapy is insufficient for most patients to achieve full recovery and lasting remission [15]. Follow-up assessments of participants in the NIMH study, conducted at 6, 12, and 18 months after treatment, indicated that the percent of patients who were rated as recovered following the end of treatment and remained well during follow-up did not differ between the four treatment groups: about 30% for cognitive-behavioral therapy, interpersonal therapy, imipramine plus clinical management, and placebo plus clinical management. Long-term studies of patients treated by ESTs indicate high rates of relapse and seeking of alternative therapies [3, 15]. Evidence indicates that short-term therapies produce benefits more quickly than more individually focused therapies but long-term therapy is superior to short-term therapies at three- and five-year follow-up [16, 17].

2.2.3. Therapy Manuals

The use of manuals in RCT outcome studies is based on the assumption that psychotherapy can be formulated as a standardized set of procedures that can be applied across matched individuals without significant variation related to differences between either therapists or patients. There is logic in attempting to describe as systematically as possible the various principles and strategies involved in each therapeutic approach but it is important to acknowledge that this requirement limits the scope and range of problems that can be treated as well as the therapies that can be studied. The use of therapy manuals biases psychotherapy outcome research in favor of those therapies that can be operationalized [13, 14, 18]. Idiographic therapies do not lend themselves to manualization or standardized application across patients [19].

Finally, evidence indicates therapists within treatments vary significantly in outcomes so that some people respond to certain types of psychotherapy provided by some practitioners and others do not, and moderators of treatment effectiveness for one outcome indicator may not be the same as moderators of treatment for another [7, 11]. A more fruitful direction for future research would be to design studies that seek to identify the characteristics of patients that are associated with positive responses to different approaches to psychotherapy, with adequate safeguards for researcher bias.

2.2.4. Therapist Allegiance

It is often not practical to conduct true double-blind RCT studies of psychotherapy since it is difficult for both researchers and patients to remain blind to the therapeutic approaches included in the study. Efforts have been made to control for this source of error variance in non-double-blind RCT psychotherapy studies but evidence consistently indicates that the “same” therapy performs better in studies conducted by researchers who are committed to the approach than it does in studies conducted by others [20, 21]. Therapist allegiance accounts for about 69% of the variance in psychotherapy outcome studies [21]. Double-blind studies of psychotherapy are admittedly difficult to conduct; however, controls for the effects of therapy allegiance are not that difficult to implement. Luborsky and colleagues have published guidelines for how to control for the effects of therapist allegiance but these suggestions are ignored in the EST literature [20, 21]. The guidelines suggested by Luborsky et al. represent minimal standards for valid psychotherapy outcome research. (1) Comparative treatment studies should be conducted using raters of varied theoretical persuasion, and researchers who have minimal allegiances to the approaches studied should be involved in conducting the study. (2) Therapists for each treatment mode should be selected and supervised by those who represent the same treatment mode, and therapists should be assigned to each mode of treatment on the basis of ratings of their effectiveness. (3) Outcome criteria developed based on the input of the therapists of all persuasions under study and long-term functional follow-up evaluations should be conducted using consistent outcome criteria. (4) If all else fails efficacy studies include researcher/therapist allegiance as a variable in all analyses.

2.2.5. Idiographic Psychotherapy Effectiveness and Process

Numerous psychotherapy studies have been published that support the effectiveness of idiographic approaches but these studies are ignored in lists of ESTs [6, 22–24]. Meta-analytic analyses of comparative outcome studies also indicate that large differences in effectiveness are not observed among the major approaches to psychotherapy (e.g., cognitive-behavioral, interpersonal, behavioral activation, psychodynamic, problem solving, or social skills training) for many problems [25]. Researchers have identified practitioner related moderators that contribute to therapeutic outcome across approaches to therapy, including the quality of the therapeutic alliance, fostering a sense of hope and self-efficacy in the patient, and that encourage relevant emotional expression in combination with meaningful cognitive and emotional processing and integration [26–29]. Process variables, such as the quality of the therapeutic relationship estimated to account for between 15 and 30% of the variance in outcome; therapist techniques about 15%; expectancy (hope), therapist credibility about 15%; and environmental and patient characteristics (e.g., readiness for change, openness, engagement, active participation, and ability to verbalize feelings) about 40%, are ignored in RCT studies [12, 29].

3. Conclusion

The use of RCT designs for the evaluation of psychotherapy effectiveness works best for those symptom focused approaches to psychotherapy that can be manualized and applied in essentially the same manner across patients who share a certain uncomplicated diagnosis. RCT designs are ideal when cause and effect relationships can be clearly defined, interventions can be applied in a similar manner across patients, and outcome criteria are applicable across all study participants. RCT designed studies have resulted in useful lists of ESTs that can be applied effectively to address specific DSM symptoms. On the other hand the requirements of RCT designs limit how problems are understood, psychotherapy can be practiced, and outcome can be defined and assessed. These design requirements disadvantage idiographic approaches to psychotherapy. Research is needed on the effectiveness of idiographic approaches to psychotherapy that are characterized by broad descriptions of patients’ problems and allow for flexibility in how therapeutic processes are described, applied, and assessed in individual cases and indicators of recovery are defined and measured over extended follow-up intervals [19]. Not everyone seeking psychotherapy services is interested to, able to, or motivated to commit to the demands and relative ambiguity of idiographic therapies, just as not everyone seeks targeted approaches to therapy for the resolution of specific uncomplicated symptoms. RCT design requirements do not allow for research on the effectiveness of idiographic psychotherapies, nor do they allow researchers to address the full range of problems for which people seek psychotherapy.

Advances in the application of phenomenological psychology appear to hold promise for the study of the effects of idiographic psychotherapies [30]. Phenomenological psychology is rooted in attempts to construct a philosophical science of consciousness that includes both hermeneutics (theory of interpretation) and symbolic interactionism (posits that individual meanings are of central concern and are only accessible through an interpretive process) [30, 31]. Methods have been developed that allow for reliable idiographic understandings of patients, their social realities, and what it means to live with symptoms in a particular situation. IPA allows us to situate and validate observations developed through discourse, subjective engagement, and careful and sympathetic attention to how another person cares for and about certain “things,” how they are involved and/or distressed by them, and how they are involved and tied up with certain aspects of their lives [31]. These methods lack the potential for clear delineation of cause and effect relationships in outcome research but hold promise for research on the effects of idiographic therapies in particular because they allow for (1) descriptions of the patients’ concerns and cares and their orientation to the world and (2) contextualization of phenomenological claims within the patient’s social, cultural, and physical environments in ways that make sense of the mutually constitutive relationship between “person” and “world” within a psychological framework that describes what this means to this person in this context [31].

IPA, in combination with methodological safeguards against researcher bias, can allow for more appropriate ways of evaluating idiographic approaches to psychotherapy and broaden our understanding of psychotherapy process while minimizing the risks of ungrounded theoretical relativism and authoritarianism. RCT designs are the “gold standard” for determining cause and effect relationships in studies of treatment effectiveness but it is important to recognize that not all approaches to psychotherapy can be adequately described, understood, and evaluated within RCT design requirements. Alternative research approaches are necessary in order to evaluate the effectiveness of idiographic approaches to psychotherapy.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

References

B. Wampold, “Do therapies designated as ESTs for specific disorders produce outcomes superior to non-EST therapies? Not a scintilla of evidence to support ESTs as more effective than other treatments,” in Evidence-Based Practices in Mental Health: Debate and Dialogue on the Fundamental Questions, J. C. Norcross, L. E. Beutler, and R. F. Levant, Eds., pp. 299–308, American Psychological Association, Washington, DC, USA, 2005.
View at: Google Scholar
D. Westen and K. Morrison, “A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: an empirical examination of the status of empirically supported therapies,” Journal of Consulting and Clinical Psychology, vol. 69, no. 6, pp. 875–899, 2001.
View at: Publisher Site | Google Scholar
D. Westen, C. M. Novotny, and H. Thompson-Brenner, “The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials,” Psychological Bulletin, vol. 130, no. 4, pp. 631–663, 2004.
View at: Publisher Site | Google Scholar
American Psychological Association Task Force on Psychological Intervention Guidelines, Template for Developing Guidelines: Interventions for Mental Disorders and Psychosocial Aspects of Physical Disorders, American Psychological Association, Washington, DC, USA, 1995.
D. L. Chambless and T. H. Ollendick, “Empirically supported psychological interventions: controversies and evidence,” Annual Review of Psychology, vol. 52, pp. 685–716, 2001.
View at: Publisher Site | Google Scholar
D. Barlow and J. Carl, “The future of clinical psychology: promises, perspectives, and predictions,” in The Oxford Handbook of Clinical Psychology, D. H. Barlow and P. E. Nathan, Eds., pp. 891–911, Oxford University Press, New York, NY, USA, 2011.
View at: Google Scholar
P. A. Arean and H. C. Kraemer, High Duality Psychotherapy Research: From Conception to Piloting to National Trials, Oxford University Press, New York, NY, USA, 2013.
K. F. Schulz, L. Chalmers, R. J. Hayes, and D. G. Altman, “Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials,” The Journal of the American Medical Association, vol. 273, no. 5, pp. 408–412, 1995.
View at: Publisher Site | Google Scholar
F. Richardson, B. Fowers, and C. Guignon, Renewing Psychology: Beyond Scientisim and Constructivisim, Jossey-Bass, San Francisco, Calif, USA, 1999.
R. Frie, Understanding Experience: Psychology and Postmodernism, Routledge, New York, NY, USA, 2003.
J. Martin and J. Thompson, “Psychotherapy as the interpretation of being: hermeneutic perspectives on psychotherapy,” Journal of Constructivist Psychology, vol. 16, no. 1, pp. 1–16, 2003.
View at: Publisher Site | Google Scholar
E. Ansell, A. Pinto, M. O. Edelen et al., “The association of personality disorders with the prospective 7-year course of anxiety disorders,” Psychological Medicine, vol. 41, no. 5, pp. 1019–1028, 2011.
View at: Publisher Site | Google Scholar
B. E. Wampold, “Psychotherapy: the Humanistic (and Effective) Treatment,” The American Psychologist, vol. 62, no. 8, pp. 857–873, 2007.
View at: Publisher Site | Google Scholar
D. Westen, “Are research patients and clinical trials representative of clinical practice?” in Evidence-Based Practices in Mental Health: Debate and Dialogue on the Fundamental Questions, J. C. Norcross, L. E. Beutler, and R. F. Levant, Eds., pp. 161–171, 317–319, American Psychological Association, Washington, DC, USA, 2003.
View at: Google Scholar
M. T. Shea, I. Elkin, S. D. Imber et al., “Course of depressive symptoms over follow-up: findings from the national institute of mental health treatment of depression collaborative research program,” Archives of General Psychiatry, vol. 49, no. 10, pp. 782–787, 1992.
View at: Publisher Site | Google Scholar
P. Knekt, O. Lindfors, T. Härkänen et al., “Randomized trial on the effectiveness of long-and short-term psychodynamic psychotherapy and solution-focused therapy on psychiatric symptoms during a 3-year follow-up,” Psychological Medicine, vol. 38, no. 5, pp. 689–703, 2008.
View at: Publisher Site | Google Scholar
P. Knekt, O. Lindfors, M. A. Laaksonen, R. Raitasalo, P. Haaramo, and A. Järvikoski, “Effectiveness of short-term and long-term psychotherapy on work ability and functional capacity—a randomized clinical trial on depressive and anxiety disorders,” Journal of Affective Disorders, vol. 107, no. 1–3, pp. 95–106, 2008.
View at: Publisher Site | Google Scholar
B. Wampold, The Great Psychotherapy Debate: Models, Methods and Findings, Erlbaum, Mahwah, NJ, USA, 2001.
B. Wampold, S. Hollon, and C. Hill, “Unresolved questions and future directions in psychotherapy research,” in The Real Relationship in Psychotherapy: The Hidden Foundation of Change, C. J. Gelso, Ed., chapter 11, pp. 333–355, American Psychological Association, Washington, DC, USA, 2011.
View at: Google Scholar
L. Luborsky, L. Diguer, D. A. Seligman et al., “The researcher's own therapy allegiances: a “wild card” in comparisons of treatment efficacy,” Clinical Psychology: Science and Practice, vol. 6, no. 1, pp. 95–106, 1999.
View at: Publisher Site | Google Scholar
L. Luborsky, M. Barrett, and D. Antonuccio, “What else materially influences what is represented and published as evidence?” in Evidence-Based Practices in Mental Health: Debate and Dialogue on the Fundamental Questions, J. C. Norcross, L. E. Beutler, and R. F. Levant, Eds., pp. 257–298, American Psychological Association, Washington, DC, USA, 2006.
View at: Google Scholar
F. Leichsenring, “Are psychodynamic and psychoanalytic therapies effective? A review of empirical data,” International Journal of Psychoanalysis, vol. 86, no. 3, pp. 841–868, 2005.
View at: Publisher Site | Google Scholar
J. Shedler, “The efficacy of psychodynamic psychotherapy,” American Psychologist, vol. 65, no. 2, pp. 98–109, 2010.
View at: Publisher Site | Google Scholar
R. Levy and S. Ablon, Handbook of Evidence-Based Psychodynamic Psychotherapy: Bridging the Gap Between Science and Practice, Humana Press, New York, NY, USA, 2008.
L. Luborsky, R. Rosenthal, and L. Diguer, “The dodo bird verdict is alive and well-mostly,” Clinical Psychology Science and Practice, vol. 9, no. 1, pp. 2–12, 2002.
View at: Publisher Site | Google Scholar
M. Lambert, “The individual therapist's contribution to psychotherapy process and outcome,” Clinical Psychology Review, vol. 9, no. 4, pp. 469–485, 1989.
View at: Publisher Site | Google Scholar
A. Horvath and R. Bedi, “The alliance,” in Psychotherapy Relationships That Work: Therapist Contributions and Responsiveness to Patients, J. C. Norcross, Ed., pp. 37–70, Oxford University Press, New York, NY, USA, 2002.
View at: Google Scholar
D.-M. Kim, B. E. Wampold, and D. M. Bolt, “Therapist effects in psychotherapy: a random-effects modeling of the national institute of mental health treatment of depression collaborative research program data,” Psychotherapy Research, vol. 16, no. 2, pp. 161–172, 2006.
View at: Publisher Site | Google Scholar
E. Heinonen, O. Lindfors, M. A. Laaksonen, and P. Knekt, “Therapists' professional and personal characteristics as predictors of outcome in short- and long-term psychotherapy,” Journal of Affective Disorders, vol. 138, no. 3, pp. 301–312, 2012.
View at: Publisher Site | Google Scholar
D. Biggerstaff and A. R. Thompson, “Interpretative Phenomenological Analysis (IPA): a qualitative methodology of choice in healthcare research,” Qualitative Research in Psychology, vol. 5, no. 3, pp. 214–224, 2008.
View at: Publisher Site | Google Scholar
M. Larkin, S. Watts, and E. Clifton, “Giving voice and making sense in interpretative phenomenological analysis,” Qualitative Research in Psychology, vol. 3, no. 2, pp. 102–120, 2006.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2014 Glenn Shean. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

17413

Downloads

668

Citations