Predictors of Dissemination of Family Psychoeducation in Community Mental Health Centers in Maine and Illinois

In its plan to improve mental health services, the National Advisory Mental Health Council asked investigators to translate their research into practice by disseminating their findings to the providers of those services (1). In response, the Maine Family Support Action Initiative (MFSAI) and a similar effort in Illinois were undertaken to translate the findings of abundant clinical research on family psychoeducational treatment methods into routine clinical practice in public mental health systems.

The family psychoeducation model defines schizophrenia as a brain disorder that is sensitive to the social environment (2,3,4,5,6,7,8,9). The results of experimental trials of family psychoeducational intervention methods for treating schizophrenia have been remarkable. Outcome studies by numerous researchers (10,11,12,13,14,15,16,17,18,19) have shown a reduction in annual relapse rates for medicated, community-based patients of as much as 60 percent as a result of a variety of educational, supportive, cognitive, and behavioral techniques of at least nine months' duration. Several studies have shown significant effects on functional outcomes and family well-being (20,21,22). With sixteen studies from four countries in the literature, family intervention has accumulated one of the most consistent records of clinical effectiveness for psychotic disorders (16).

However, the national record of nonapplication and nondissemination of family intervention is also unequalled, especially for such a remarkably cost-effective approach for treating so disabling and costly a disorder. A three-year project conducted by the New York State Office of Mental Health focused on a longer-term implementation and dissemination strategy, emphasizing the multifamily group version of psychoeducation (23). Unfortunately, a change of administration and a reversal of policy ended that effort before the strategy had been implemented. Until the effort we describe here was undertaken, New Jersey was the only state to fully and continuously subscribe to dissemination of family psychoeducation programs (24). Few clinical services or community support organizations can be found that systematically offer empirically proven family support services.

Dissemination of this approach, as with other new psychosocial methods, presents complex challenges, because it satisfies only one of Rogers' (25,26) criteria for an innovation that is likely to be diffused rapidly; that is, it offers a relative advantage. It is not compatible with the theoretical training of many clinicians, it is more complex than standard medication treatment, it cannot readily be tried in a brief or easily evaluated form, and the outcomes—by design—emerge over the long term and are therefore not readily observable. The problems of disseminating new psychiatric treatments in existing systems have been described by Fairweather (27), who showed that an effective dissemination program requires simultaneous but various approaches to changing the practices of administrative and clinical personnel.

The initiatives in Maine and Illinois set a goal of disseminating psychoeducational treatment across each state. These programs attempted to use and extend Fairweather's approach by incorporating what was learned from the New York initiative. The approach was based on distinct dissemination methods similar to those used with clinical family interventions (23).

The Maine Family Support Action Initiative

The MFSAI was a joint endeavor of the Maine Medical Center and the executive directors of key community mental health services, all of whom are members of the Maine Association of Mental Health Services. It proceeded with the concurrence of the State of Maine Department of Mental Health, Mental Retardation, and Substance Abuse Services and was sponsored by the Center for Mental Health Services. The objectives of the initiative were to provide psychoeducational support services for at least two years in a multifamily or single-family format by trained and supervised staff and in collaboration with an integrated team of mental health practitioners.

The project team pursued a three-step process to achieve dissemination. First, the team wanted to build consensus within the local and regional mental health community for adopting the psychoeducational multifamily group model. Second, it wanted to ensure that the human, financial, and technical resources required to implement the model were in place. Third, the team carried out concrete implementation plans, both statewide and for each site, taking local complexities into account and making adaptations where necessary.

Consensus building

To gather statewide support, project leaders assembled a Family Services Council with representatives from such bodies as the Maine chapter of the National Alliance for the Mentally Ill, the state mental health department, the Office of Consumer Affairs, and the Bureau of Medical Services—the Medicaid administrator.

In addition, orientation workshops were held throughout the state in an effort to promote adoption of the approach through descriptions of outcomes from clinical trials and descriptions of the interventions themselves. These workshops introduced family advocates, consumers and mental health professionals, administrators, and policy leaders to the advantages of multifamily psychoeducational groups. They also served to address concerns voiced by stakeholders. Considerable effort was made to secure the agreement and cooperation of the agencies themselves before and after the new treatment approach was introduced.

Resources

The strategy to build resources for the initiative included changing the Medicaid code to accommodate single and multifamily support and treatment services, training clinicians, and supporting prospective sites as well as actively participating sites. A new rule for outpatient Medicaid-reimbursed family services was developed and was offered to the practice community. Training for clinicians began in March 1998 and was completed in December 2000.

Approximately 380 clinicians from 33 agencies, hospitals, and centers were trained. On-site consultation and clinical supervision were offered to each of 15 catchmented mental health centers and their affiliated hospitals to develop procedural adaptations specific to each service system. These adaptations have been carried out at 14 of the 15 core centers so far. Several of the 18 remaining agencies went on to implement services as well.

Implementation

The MFSAI used a four-step implementation design: needs assessments and epidemiologic profiles of populations and family availability; continued orientation and consensus building, clinical training, and on-site consultation; adaptations to accommodate unique service delivery issues and target populations; and ongoing supervision and consultation. Needs assessments were used to design services on a site-by-site basis, allowing for variations in consumer populations or resources. Adaptations included multifamily groups for bipolar disorder, major depression, adolescents, and elderly persons, each developed by use of existing methods for these disorders and populations and supervised by the implementing team. Although these groups were generally judged to be successful by clinicians, supervisors, and participating families and consumers, we were not able to formally assess outcomes.

The Illinois Family Psychoeducation Dissemination Project

The team from the Maine Medical Center that carried out the MFSAI received funding and a mandate from the Illinois mental health department to disseminate the model to clinicians in mental health centers and agencies throughout Illinois. The Illinois effort was initiated by the state mental health department to improve the quality of care at agencies with which the department had major contracts as well as at some state-operated agencies and facilities.

This initiative began with the identification by the statewide advisory council on the mental health service system of training in multifamily psychoeducational groups as a priority for statewide dissemination. The advisory council consists of consumers, providers, advocates, and representatives of academia from across the state. Before implementation several meetings were held to create an awareness of multifamily psychoeducational groups and to build commitment on the part of mental health service agencies to participate in the training. A presentation of the model was combined with discussion about the utility of implementing such groups. Executive directors of community agencies were invited to three regional meetings around the state.

In addition, a presentation was held in conjunction with a statewide meeting of one of the largest trade associations representing mental health providers in Illinois. Training exercises as well as technical assistance using the same set of presentations that were used in Maine were offered in nine service areas. Clinical staff at 51 of approximately 200 mental health agencies participated in training workshops.

State differences

The efforts in Maine and Illinois differed in five significant ways. First, in Illinois dissemination was initiated by the state's mental health department, and the participating agencies were independent, nonprofit mental health centers that were the targets of the dissemination effort. The central administration promoted adoption of the model through regional directors and encouraged voluntary participation. Financial incentives were provided to the implementing centers. The state's mental health trade association, which is composed of the directors of all the major mental health centers, was not involved.

Second, although the Illinois effort attempted to reach multiple staff levels and various geographic units, it placed markedly less emphasis on local consensus building and on acceptance by clinicians, families, and consumers than the Maine initiative. At one site, a family advocacy group was actually opposed to the initiative, believing that the proposed new services would compete with their family education program. Third, few sites in Illinois used consultation and supervision after the training workshops. Fourth, direct implementation expenditures in Maine were 25 cents per capita and in Illinois were only one cent per capita. Fifth, the project's director for both states (the first author) is a resident of Maine and a member of the Maine Association of Mental Health Services.

In this article we report the results of a third-party process evaluation of these efforts to disseminate family psychoeducation throughout the public mental health systems in two states. Our purpose was to use the results of the evaluation to identify characteristics both of agencies and of clinicians that were associated with implementation and nonimplementation of family psychoeducation services. We also review the evaluation and present several observations that may explain the differences in outcomes between the two states. The comparison between states was not a deliberate experiment; rather, the differences emerged during the process.

Methods

Data collection

An independent evaluation team conducted a two-part evaluation. A written survey was administered immediately after each orientation and training workshop. The survey included seven domains: trainees' demographic characteristics, satisfaction, knowledge acquisition, acceptance of and intentions to use the new approach, beliefs about the agency's ability and intention to implement the initiative, perceived obstacles to implementation, and the perceived value of various resources in helping the agency implement the program.

The instrument used to evaluate the perceived likelihood of implementation was developed specifically for evaluating multifamily psychoeducation clinical training exercises in collaboration with the University of Maryland's Schizophrenia Patient Outcomes Research Team (PORT). The instrument was originally adapted from a similar instrument that was first used in a dissemination evaluation concucted in the Netherlands and then modified for the PORT dissemination of multifamily psychoeducation in Ohio (28). The domains and their constituent items are listed in Tables 1 and 2. Participants responded to the survey items on 3-point to 9-point Likert scales, depending on the particular item. The surveys and data collection procedures used in Maine and Illinois were identical.

The second component of the evaluation consisted of follow-up interviews by an independent evaluation firm, Hornby Zeller Associates, nine months after the training sessions. Those of us who were not involved in the training (H.H. and E.C.) interviewed key informants at each agency by using semistructured, open-ended protocols that permitted the participants to lead the discussions in directions that were important to them. Progress in implementing the model was rated on a 5-point scale, where 1 represented no attempt to implement services and 5 represented full implementation of at least one multifamily psychoeducational group service.

Data analysis

Participant level. Mean scores for the survey items were calculated for each state and for the states combined. Between-state comparisons of mean responses were conducted with use of Wilcoxon's two-sample tests because of item nonnormality. To reduce the number of perceived obstacles and helpfulness items that were used in the analysis, principal components analyses were conducted separately on the two sets of data, yielding two principal components each: expected limitations of the model and expected resource limitations for obstacles, and inducements to implement and availability of help for helpfulness items. The principal components were rotated to varimax criterion to enhance interpretability. Cronbach's alpha coefficients for expected resource limitations, expected model limitations, availability of help, and inducements to implement were .74, .90, .90, and .94, respectively.

Site level. Site-level variables were created by aggregating trainees' demographic data within each site—the mean proportion of women trainees, the mean age of trainees, and the mean number of years the trainees had been in the mental health field; program characteristics—the mean rating by trainees of short-term and long-term priority of implementing the intervention, where 1 represented no priority and 5 represented very high priority; the mean rating by trainees of the likelihood of implementation at three months and nine months, where 1 represented none at all and 5 represented a very high likelihood; the mean rating by trainees of consistency of the intervention with the methods and philosophy of the site, where 1 represented not consistent and 5 represented very consistent; the mean ratio of clinicians to administrators among the trainees; and the four principal components averaged across the site.

State (Maine versus Illinois) was also entered as a predictor of implementation. Multiple regression analysis was used to predict each site's progress toward implementation as assessed in follow-up interviews. Progress was measured on a scale of 1 to 5, on which 1 represented "no steps to implement" and 5 represented "currently functioning group"; half-point scores were assigned when appropriate. In addition to regression parameter estimates, semipartial r² coefficients for each predictor were calculated to assess the unique contribution of each predictor, with all other predictors controlled for, to the variance in implementation progress. This calculation permitted an examination of multiple associations between various site characteristics and progress toward implementation when each site characteristic was controlled for all others in the model.

Results

Survey data

A total of 537 trainees completed surveys. These respondents came from 69 sites, with a mean±SD number of respondents of 5.5±6.7 per site (median, three respondents; range, one to 27 respondents). The sites of eight respondents were unknown because of missing data. A total of 103 Illinois trainees and 47 Maine trainees were from sites that did not provide follow-up data and so were not included in the regression analyses. A total of 53 of the 69 sites had both follow-up data and complete predictor data.

The demographic characteristics of the 537 trainees were as follows: 376, or 70 percent, were women; 322, or 60 percent, were under the age of 45 years; 328, or 61 percent, were clinicians; and 199, or 37 percent, were administrators, family members, or consumers. More than half (279 participants, or 52 percent) had been in the mental health field for ten years or less, and 299, or 54 percent, were from sites in Maine. Thus a disproportionate number of trainees were from Maine, given the number of agencies involved. A total of 381 trainees, or 71 percent, believed that their agencies had set aside space for multifamily psychoeducation; 328, or 61 percent, believed that funding was available for this type of treatment; and 317, or 59 percent, believed that the agency had a designated implementer.

The most common perceived obstacles to implementation, in decreasing order, were intense pressures on staff as a result of growing caseloads and less time per case; excessive current demands; a perceived lack of reimbursement for a nontraditional, though empirically proven, treatment; the longer duration—one to two years—of this treatment approach; and uncertainty about financing for the agency as a whole (Table 1).

With respect to the helpfulness of support and inducements for implementation, the respondents reported, in decreasing order, the availability of outside consultants, an ongoing link to trainers, early involvement of participants, step-by-step implementation, and personal contact with the treatment developers. Least helpful was evidence of the model's relative advantage over existing programs—for example, literature on outcome studies.

Predictors of implementation

The overall regression analysis yielded a model that significantly predicted implementation of family psychoeducation at nine months (R²=.7307, adjusted R²=.6215, p<.001, N=53). Table 2 shows the regression scores, semipartial r², and probabilities for the principal components and demographic data, with site as the unit of analysis. The home state of the program was the strongest single predictor of progress; on average, Maine sites made more progress toward implementation than Illinois sites. Sites where respondents expected that limitations of the model would be less important reported more progress.

Counterintuitive findings were that sites where trainees rated inducements as less helpful made greater progress, and sites where trainees rated multifamily psychoeducation as less consistent with current methods made greater progress. It is noteworthy that the trainees were able to accurately predict whether their sites would succeed in implementation.

The relationships between expected resource limitations and outcome were less strong: sites where trainees expected resource limitations to be a greater obstacle, again seemingly paradoxically, made more progress toward implementation. A higher ratio of clinicians to administrators at the orientation and training sessions was a modest predictor of progress. The availability of help did not predict implementation. The least influential factor was "evidence of advantage over other programs"—that is, empirical evidence of the model's superiority. The items are listed in order of influence in Table 1.

Between-state comparison of outcomes and predictors

As for actual implementation outcomes, the MFSAI focused on 15 key mental health centers or core service agencies that provide catchmented and state-contracted services to the vast majority of the population with severe and persistent mental illness. As of December 2000, project leaders had initiated multifamily psychoeducation and single-family support services at 14 of those sites, or 93 percent. The participating patients and families were from rural, small-town, and urban settings. In Illinois, of the 51 core mental health centers with roles comparable to those in Maine, five, or 10 percent, had implemented services, and a small number were in the process of developing services. Fourteen of 15 sites in Maine used consultation, supervision, or both; almost none of the sites in Illinois used these supports.

The survey data reflected these outcomes: the strongest predictor was the location of the involved agency. Compared with their counterparts in Illinois, trainees in Maine rated the availability of help as much more useful or important. The largest difference for a single item was in the perceived value of an ongoing link to trainers. The same was true, but to a lesser extent, for inducements. Illinois trainees rated the expected limitations of the model as more critically important obstacles (Table 1).

In particular, large differences were found in the belief that family members would not support the approach, the belief that there were too few patients and families to warrant implementation, and skepticism about the assumptions and the value of multifamily psychoeducation. Maine trainees were less concerned about the need to adopt a medical model for major mental illness and were more positively predisposed to accepting help and inducements—for example, observing multifamily psychoeducation in action and getting information about the model's advantages. The Maine trainees also thought that having family members as collaborators requires less of a change in attitude. Thus before the implementation efforts began, the Illinois trainees were significantly more skeptical about the family psychoeducation model and gave lower ratings of the usefulness of further assistance after training.

Discussion

To understand the outcomes of this complex dissemination program, we first comment on the empirical outcomes and predictors and then, using baseline survey data, follow-up interview data, and best-estimate observations, attempt to explain the marked differences between these states in the degree to which they have adopted a major best-practice approach.

Empirical findings across both states

The most striking finding is that it was possible, by controlling for between-state differences, to predict the likelihood of an agency's adopting this psychosocial treatment on the basis of data gathered from potential implementers of the initiative after training. Because the respondents were trained and surveyed at different times and because local experience with implementation may have biased those who were trained later, these results should be interpreted with caution.

Nevertheless, the prospective association between predictors—in particular, the principal components—and progress ratings can be used to identify which sites have been successful after the fact. This approach may allow prediction in advance or at least at an early stage and guide intensive efforts toward addressing local obstacles suggested by the survey data. The survey itself could be used to assess the likelihood that sites will implement the treatment model once cutoff scores have been derived from this and further applications in implementation research.

Principal components

Not surprisingly, sites where staff rated family psychoeducation as having many inherent limitations were less likely to implement this treatment approach and presumably were more resistant to changes in practice. This finding is consistent with the PORT findings (29). However, the finding appears largely to reflect perception or bias, because agencies where trainees did not believe that the treatment model had limitations proceeded to implement it successfully—that is, they did not experience actual limitations—and in follow-up interviews they were enthusiastic about the effects of the initiative and the process of multifamily psychoeducation.

An unexpected finding was that the more the trainees saw fiscal and operational factors as impediments and the less useful they thought inducements would be, the more progress their agencies made. The successful agencies were apparently those in which staff viewed fiscal and operational barriers as significant, dealt with or neutralized those barriers, and went on to implement multifamily psychoeducation. These staff members rated inducements as less important, apparently seeing themselves as already well suited for adopting multifamily psychoeducation and perhaps as more inclined to use better therapies for ethical or professional reasons. The administrations and staff of these agencies may have had more confidence in, or commitment to, addressing barriers and improving the quality of care.

Single items

The participants' own sense of the likelihood of their agency's implementing multifamily psychoeducation was predictive by itself, adding additional weight to the validity of the principal components analyses. This finding implies that trainees decided after the orientation or training sessions whether their agency—and perhaps they themselves—would adopt this new practice. Knowing about trainees' perceptions and attitudes during or very soon after training might allow supervisors to explore potential problems associated with implementing multifamily psychoeducation and to address these problems on the basis of previous positive experience elsewhere.

Furthermore, staff at sites that implemented multifamily psychoeducation seemed to appreciate that this treatment model was not consistent with their existing approaches and that they would have to change their methods and perhaps even their attitudes toward families. These findings emphasize the large role that is seemingly played by subjective factors, as opposed to more objective judgments about the empirical superiority of best practices. The findings also suggest that simply asking trainees how likely it is that their agency will implement multifamily psychoeducation could prove to be a useful approximation of the actual likelihood.

Empirical proof in the literature of the effectiveness of multifamily psychoeducation—the component "advantage over other programs"—ranked lowest among the helpfulness items. This finding is disturbing, because this component is the only one of Rogers' predicting factors (25) that is common to this and other forms of psychosocial treatment.

This finding, if generalizable, would go far in explaining why randomized controlled clinical trials have had little impact on dissemination across the mental health service system. Empirical proof, even when emphatically presented to clinicians, apparently is not as influential as the treatment's philosophy, methods, emotional appeal, feasibility of implementation, or early experiences in working with families. On the other hand, it is likely that many of the clinicians who are now successfully leading multifamily psychoeducation groups would not have implemented this approach if they or their administrators had not seen improvement in outcomes as a major incentive.

Between-state differences

To date, the strategies applied in Maine to achieve and build on a consensus have been successful. All but one of the major mental health centers—the two largest hospitals that provide psychiatric services and several major case management, outreach, crisis, housing, and children's provider agencies—are offering empirically validated family psychoeducational services, primarily multifamily groups. Momentum and interest have increased over the past few years—one successful implementation has been conducive to another.

Once clinicians at a site have actually conducted multifamily psychoeducation groups, skepticism or previously perceived limitations are generally replaced with acceptance and often enthusiasm. The Maine sites that implemented multifamily psychoeducation and led a group for more than six months decided to add additional groups and train additional staff. Some of these sites have implemented the model at satellite clinics. In Illinois, implementation has been limited.

Several factors might explain the relatively large difference in implementation success between the two states to date. Compared with their counterparts in Illinois, the trainees in Maine were significantly more positive about the treatment model, were much more interested in receiving supervision and consultation, and were less influenced by perceived obstacles at the outset. Thus it appears that Maine trainees were better informed about the model and more personally or professionally predisposed to implement it.

We believe, but cannot prove with existing data, that the difference in outcomes also reflects the different strategies used in the two states. Maine made an elaborate effort to reach several stakeholder groups, including family members and family advocates, consumers and their advocates, paraprofessionals and professionals, supervisors, and, especially, executive and clinical directors of community mental health centers and hospital psychiatric services. The project was sponsored by the trade association, whereas the state mental health authority was supportive but not actively involved.

In Illinois, the state mental health authority was the initiator and ongoing promulgator of the dissemination. Local agencies and their trade association were markedly less involved. Nevertheless, the effort has put Illinois well ahead of most states in the availability of family support services from the core mental health agencies.

One interpretation of our results is that the unilateral implementation of evidence-based practices by state mental health authorities is fraught with obstacles. Such a conclusion was not anticipated—and is not welcomed—by us or by the principal officials in Maine or Illinois. These obstacles became apparent despite the mental health authority's continuing dedication to the dissemination effort and its allocation of training resources to facilitate implementation. Clearly, if this finding is confirmed elsewhere, it represents a major barrier to the adoption of new therapies, because in nearly every state the principal fiscal beneficiary and the only centralized control system is the state mental health authority. Note also that in at least one state—New York—the state mental health authority has had the power to impede implementation. For better or for worse, this experience suggests that responsibility for implementation rests with the agencies who work directly with the affected population, as well as with the mental health authority and consumer and family stakeholder groups.

Finally, greater financial resources from the federal sponsor were available in Maine than in Illinois. As in many aspects of mental health services, better outcomes were achieved when more funding was available for consensus building before training and during supervision. However, unlike the survey data, these observations were made after the fact and are our opinions only. These observations hardly exhaust the possible differences between the states that might explain their different experiences.

Follow-up of trainees

Another source of information was the interviews with the trainees and their supervisors that were conducted periodically at sites with multifamily psychoeducation groups. Those agencies' representatives have been virtually unanimous in affirming the importance of ongoing supervision and consultation. Several leaders of multifamily psychoeducation groups stated that they had underestimated how useful—and perhaps essential—such support is, both for the ongoing needs of the groups and to meet specific challenges as they arise. However, the principal component that included clinical supervision ("availability of help") did not predict outcome.

Conclusions

Trainees' perceptions at the beginning of implementation of multifamily psychoeducation were associated with outcomes in Maine and Illinois. This finding illustrates the importance of building consensus among key stakeholders before undertaking a universal change in practice. Local agencies and clinicians should take the lead in initiating new approaches or programs, while being encouraged and supported by the state mental health authority. The approach most likely to succeed is a partnership between local agencies and the state's mental health and Medicaid administrations, with the encouragement of local and statewide advocacy organizations.

Our findings and our experience in the field suggest that the most traditional method of influencing adoption of a new and more effective approach—publication of clinical trials in prestigious journals—has a limited impact on clinicians' and agencies' practice when the individuals who have the potential to implement such approaches are not part of an academic setting. Proof of the effectiveness of psychosocial treatment models is clearly necessary but is insufficient by itself. In addition, professional training programs need to build these new treatments into their curricula and to inculcate in trainees an expectation that they will have to update their skills periodically as new knowledge and techniques are shown to be more effective, just as surgeons and internists do.

The experience in Maine and Illinois strongly suggests that outcomes data from controlled clinical trials must be abundantly supplemented with information that is professionally and personally meaningful, persuasion, consensus building, removal of barriers, consultation with the agency's administration, time to learn, and effective training and supervision. These appear to be the necessary ingredients for a change in practice on a scale that will lead to benefits for the affected population and realize the cost savings that are promised by new psychosocial treatments.

Acknowledgments

This study was supported in part by community action grant SM-52197 from the Center for Mental Health Services and a contract from the Illinois Department of Human Services.

Dr. McFarlane, Ms. Hornby, and Ms. Cimett are affiliated with the Maine Medical Center, 22 Bramhall Street, Portland, Maine 04102 (e-mail, [email protected]). Dr. Dixon and Dr. McNary are with the University of Maryland Mental Health Services Research Center in Baltimore.

Enlarge table

Enlarge table