Introduction
Cognitive and behavior therapies are often considered “first-line” treatments for a number of psychiatric disorders, with various meta-analyses demonstrating the efficacy of these therapies for conditions such as anxiety disorders (Hofmann and Smits
2008; Otto et al.
2004), depression (Dobson
1989; Spek et al.
2007), and substance-use disorders (Duttra et al.
2008). While cognitive and behavior therapies have been established on theoretical foundations, the efficacy of these interventions may lie in their strong history of utilizing homework assignments as a mechanism toward producing beneficial treatment outcomes. That is, practice of skills outside of therapy (i.e., homework) allows clients to master the skills believed necessary to affect symptoms, generalize these skills to their natural settings, and promote prolonged symptom improvement through extending therapeutic aspects of treatment beyond the completion of therapy (Kazantzis and Lampropoulos
2002).
Indeed, the importance of homework for producing positive therapy outcome was demonstrated in a previous meta-analysis (Kazantzis et al.
2000). In their analysis, a Pearson
r effect size of .22 was reported for the relationship between homework compliance and therapy outcome in a sample of 1,327 subjects across 27 studies. These results suggest that greater compliance with homework is associated with beneficial treatment outcome, with the strength of the association falling between Cohen’s small and medium effect size cutoffs (Cohen
1988; Kraemer et al.
2003).
Kazantzis et al. (
2000) analysis was the first study to examine the type of homework activity and the nature of the client’s presenting problem as moderating variables of homework effectiveness. The presenting problems were categorized as depression, anxiety-related disorders, and other outpatient. The results of this meta-analysis showed the following mean effect sizes for problem type: depression (.22), anxiety (.24), and other outpatient (.17), with homework effects being significantly greater for the treatment of depression than the “other outpatient” sample. Additionally, results indicated that effect sizes were robust across the type of homework completed (no single type, relaxation, or social skills) and time of homework compliance assessment (regular intervals or posttreatment), but differed by the source of homework compliance assessment. Specifically, studies that utilized client and therapist ratings had a significantly lower mean effect size relative to those using objective measures of homework compliance.
In the 8 years since Kazantzis, Deane, and Ronan’s meta-analysis on the effects of homework assignments on treatment outcome, homework has continued to remain “both a traditional and integral component of contemporary manual-based cognitive-behavioral therapy (CBT) approaches” (Coon and Thompson
2003, p. 53). Further, there continues to be support for the effectiveness of cognitive-behavioral interventions to prevent the onset, relapse, and recurrence of a number of psychological disorders (Hollon
2003). The meta-analysis conducted by Kazantzis et al. (
2000) included homework-related studies spanning from 1980, 1 year following Beck’s emphasis on regularly using homework in cognitive-behavioral therapy for depression (Beck et al.
1979), through 1998, a time when homework in therapy had been incorporated into a more diverse range of clinical conditions (Kazantzis et al.
2000). Therefore, a significant amount of variance as a function of time may exist within this analysis.
The present study is an updated meta-analysis of the relationship between homework compliance and treatment outcome. We hypothesized that greater homework compliance would be significantly associated with improved treatment outcome. Given that the previous meta-analysis found some evidence that targeted symptoms and source of homework ratings may moderate the effect of homework compliance, we further examined whether treatment target (e.g., symptoms of anxiety, depression, etc.) and source of rating (e.g., therapist, objective) moderated the relationship between homework compliance and therapy outcome. A novel aspect of this meta-analysis is that we examine the moderating effect of rating type (e.g., Likert rating, percentage of homeworks completed).
Discussion
This meta-analysis examined the relationship between homework compliance and treatment outcome across 23 studies and over 2,000 participants. Similar to results found by Kazantzis et al. (
2000), greater homework compliance was associated with improved treatment outcome (
r = .27). These results were consistent across a variety of target symptoms including symptoms of anxiety (
r = .22), depression (
r = .24), and substance use (
r = .27), suggesting that compliance with homework is an important component of psychotherapy regardless of the target symptoms. Indeed, this finding is consistent with cognitive and behavioral theories, which suggest that mastery of skills learned in therapy via practice of such skills is important for producing positive treatment outcomes (i.e., improving symptoms).
In the present study, the two most common sources of homework ratings were therapists and objective ratings (e.g., counting the number or percentage of homework turned in), and we found that the source of homework ratings moderated the relationship between homework compliance and treatment outcome. Specifically, when both clients and their therapists provided homework ratings, effect sizes were significantly higher (
r = .35) than when objective ratings were used (
r = .16). However, because only two studies utilized both client and therapist ratings, these results should be interpreted with caution. Indeed, the two studies that utilized therapist and patient ratings of compliance used quite different methods for assessing homework compliance and had quite different sample sizes. Moreover, our analysis averaged the therapist and patient rating of homework compliance, despite the fact that these ratings may not always be strongly correlated. Indeed, the study by Westra and Dozois (
2006) reported only a modest correlation between therapist and client compliance ratings. Again, given the small number of studies utilizing this method and the limitations mentioned here, readers should take caution about interpreting these findings as particularly meaningful.
These findings might be interpreted in a number of different ways. First, they may suggest that future studies of this relationship should utilize both types of ratings, at least on the assumption that this effect size discrepancy is real. Alternatively, this discrepancy in findings might highlight the inherent limitations of using “subjective” ratings as a means of assessing homework compliance. For example, therapists who provide homework ratings may give better scores to those who are doing better in therapy (i.e., “he’s doing better, so he must be doing his homework”).
There were no significant differences between groups when comparing other sources of homework ratings. However, although objective ratings did not differ from client alone or therapist alone ratings, it is interesting to note that our findings differ from those of Kazantzis et al. (
2000), who found that objective ratings had a higher overall correlation with treatment outcome. This may be due to the difference in defining “objective” assessment between the two meta-analyses. Specifically, whereas Kazantzis defined “objective” as an electronic marker of homework compliance, our analysis considered “objective” to mean studies that counted the number of homeworks turned into therapists.
Studies that used Likert scales to rate homework compliance had a significantly higher mean effect size (r = .31) than those rating the percentage of homeworks completed (r = .17). Further, studies using Likert scales were higher, but not significantly so, than studies using the number of homeworks completed. While this finding is difficult to explain, it may be due to the fact that Likert ratings might inadvertently reflect quality and quantity ratings, whereas a summary variable such as percent or total homeworks completed reflect quantity only. For example, during the course of therapy, clients may be asked to regularly (e.g., once each day) practice homework. However, they may present at the next therapy session and describe one excellent (and extremely beneficial) example of how he/she practiced homework over the past week. Therapists who rated client homework from 0 (poor) to 6 (outstanding) might rate this compliance relatively high on the scale. In contrast, clients who report doing homework every day but who had difficulty with the assignment or who described it as unhelpful might be rated relatively lower in terms of compliance. Further, Likert scales provide the therapist and the client with a range to rank homework completion. This can be opposed to percentage of homeworks completed and number of homeworks completed, which are often scored on a dichotomous (completed or did not complete) scale. If a client completes part of a homework assignment, the client is given some credit for compliance, even if the effort is minimal.
Further, a “timing effect” was found for contemporaneous versus retrospective ratings of homework completion in that retrospective ratings were a significantly better predictor of outcome than contemporaneous ratings. This may have been due to a bias effect for retroactive ratings. For example, it is possible that patients who have appeared to have done well in therapy could have been rated by their therapist or themselves as more compliant with homework assignments. These results may provide insight into differences in objective versus subjective ratings (i.e., higher effect size for subjective ratings than objective assessments), in that objective ratings are most typically contemporaneous by nature (e.g., paperwork that was turned into and/or discussed with the therapist), and therefore appear more reliable in assessing compliance than retroactive or subjective ratings of compliance.
These issues (objective vs. subjective; Likert vs. non-Likert) highlight the important issue of how we define homework compliance. Specifically, they highlight the important issue of the purpose of conducting a homework analyses, which is to discover the “true nature” of the relationship between homework compliance and treatment outcome, not findings ways of manipulating methods to demonstrate larger effects. Determining the true effect indeed involves finding increasingly “objective”, or bias-free methods of assessing homework compliance. To this end, Kazantzis et al. (
2004) has described novel methods of assessing homework in therapy research (e.g., the Homework Rating Scale), which include the assessment of homework quality. However, there has yet to be any consistent use of these methods. We strongly recommend new research incorporate these new methods of assessing homework compliance, as well as develop more objective and accurate means of assessing homework quantity and quality in treatment research and outcome.
There are several limitations to the current review. As previously mentioned, there have been problems with the objective assessment of homework compliance. Additionally, the current review did not examine demographic moderators (i.e., age, gender, ethnicity, education) or the severity of psychopathology (e.g., Major Depressive Disorder vs. Dysthymia; Substance Abuse vs. Substance Dependence) that could contribute to homework compliance. These variables were not included in the current study’s moderator analysis as they were not examined in the results of the studies reviewed. Research has found that clients comply less with homework directives if they have greater and/or more long-lasting symptomology (Worthington
1986). In addition to demographic moderators and severity of psychopathology, other things to keep in mind when considering the relationship between a client’s homework compliance and therapeutic gain are pharmacotherapy (e.g., is the client on antidepressant medications?), if the client is involved in another form of treatment (e.g., social skills training), and use of coping mechanisms for dealing with stress (e.g., does the client take action in response to stress or become less productive? Addis and Jacobson
2000). The results, however, demonstrate a more generalized view of the effects of homework compliance on therapy outcome across a span of different psychological diagnoses and diverse demographic characteristics.
A further limitation of the current review is that it did not take into account the client-therapist relationship. Research has found that a positive and trusting client-therapist relationship may aid recovery in mental illness (Green et al.
2008) regardless of homework. Additionally, the strength of the relationship between the client and the therapist could contribute to homework compliance, with a stronger working relationship leading to increased homework compliance. Without looking at the client-therapist relationship as a moderator between homework compliance and treatment outcome, there is a possibility that the relationship alone contributed to the improvements seen in the clients. However, as mentioned by Kazantzis et al. (
2000), there exists an abundance of research that demonstrates the positive effects of the use of homework in therapy on treatment outcome.
Finally, the current review did not examine the client’s attitude towards homework. A negative attitude towards homework, even if the homework is completed, could potentially limit the likelihood that the client will continue to practice the skills learned once therapy is completed. Motivation, lack of effort, and readiness to change are other variables that were not explored in the current study, which are factors that have been found to be correlated with homework compliance (Neimeyer et al.
2008; Yovel and Safren
2007). Addis and Jacobson (
2000) examined the relationship between clients acceptance of the treatment rationale and the degree to which clients completed homework, and concluded that the ability to provide a convincing treatment rationale may be one of the crucial skills which determines the success of CBT in real-world clinical settings. Further studies would benefit from exploring these areas in regard to homework compliance.
In sum, the results of this meta-analysis suggest that on the whole, greater compliance with homework is related to improved treatment outcome, and this relationship is robust across a variety of treatment targets (e.g., depression, anxiety, and substance use). However, this study also highlights discrepancies in effect sizes surrounding the method of assessing homework compliance (e.g., objective vs. subjective). Specifically, higher effect sizes were found when therapists and clients both evaluate homework compliance. On one hand, clinicians may desire making homework compliance a collaborative part of treatment (e.g., to structure therapy whereby review of homework is an integral part of sessions). On the other hand, these discrepancies may highlight the inherent limitations in using subjective assessments of homework compliance. To this regard, it may be increasingly important for more standardized and objective methods of assessing homework compliance that are less prone to bias and that capture the true nature of the relationship between homework compliance and treatment outcome. In this vein, suggestions on incorporating homework into therapy and improving compliance are available in the literature (Beck
1995; Tompkins
2004), as are forms for measuring multiple aspects of homework compliance (Kazantzis et al.
2004).