Skip to main content

Advertisement

SpringerLink
Log in

Comparison of Assertive Community Treatment Fidelity Assessment Methods: Reliability and Validity

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Assertive community treatment is known for improving consumer outcomes, but is difficult to implement. On-site fidelity measurement can help ensure model adherence, but is costly in large systems. This study compared reliability and validity of three methods of fidelity assessment (on-site, phone-administered, and expert-scored self-report) using a stratified random sample of 32 mental health intensive case management teams from the Department of Veterans Affairs. Overall, phone, and to a lesser extent, expert-scored self-report fidelity assessments compared favorably to on-site methods in inter-rater reliability and concurrent validity. If used appropriately, these alternative protocols hold promise in monitoring large-scale program fidelity with limited resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allness, D. J., & Knoedler, W. H. (1998). The PACT model of community-based treatment for persons with severe and persistent mental illness: A manual for PACT start-up. Arlington: NAMI.

    Google Scholar 

  • Allness, D. J., & Knoedler, W. H. (2003). The PACT model of community-based treatment for persons with severe and persistent mental illness: A manual for PACT start-up. Arlington: NAMI.

    Google Scholar 

  • Bond, G. R. (2013). Self-assessed fidelity: proceed with caution. Psychiatric Services, 64(4), 393–394. doi:10.1176/appi.ps.640418.

    Article  PubMed  Google Scholar 

  • Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health, 41(2), 228–236. doi:10.1007/s10488-012-0461-5.

    Article  PubMed  Google Scholar 

  • Bond, G. R., Drake, R. E., Mueser, K. T., & Latimer, E. (2001). Assertive community treatment for people with severe mental illness: Critical ingredients and impact on patients. Disease Management & Health Outcomes, 9, 141–159.

    Article  Google Scholar 

  • Bond, G. R., McGrew, J. H., & Fekete, D. M. (1995). Assertive outreach for frequent users of psychiatric hospitals: A meta-analysis. Journal of Mental Health Administration, 22(1), 4–16.

    Article  CAS  PubMed  Google Scholar 

  • Bond, G. R., Williams, J., Evans, L., Salyers, M., Kim, H. W., Sharpe, H., & Leff, H. S. (2000). Psychiatric rehabilitation fidelity toolkit. Cambridge: Human Services Research Institute.

    Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  • Clark, P. (2004). The role of medicaid, Chicago.

  • Drake, R. E., Bond, G. R., & Essock, S. M. (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35(4), 704–713. doi:10.1093/schbul/sbp041.

    Article  PubMed Central  PubMed  Google Scholar 

  • Drake, R. E., Goldman, H. H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., & Torrey, W. C. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52(2), 179–182.

    Article  CAS  PubMed  Google Scholar 

  • Herdelin, A. C., & Scott, D. L. (1999). Experimental studies of the program of assertive community treatment (PACT): A meta-analysis. Journal of Disability Policy Studies, 10, 53–89. doi:10.1177/104420739901000105.

    Google Scholar 

  • Latimer, E. (1999). Economic impacts of assertive community treatment: A review of the literature. Canadian Journal of Psychiatry, 44, 443–454.

    CAS  PubMed  Google Scholar 

  • McGrew, J. H., & Bond, G. R. (1995). Critical ingredients of assertive community treatment: Judgments of the experts. Journal of Mental Health Administration, 22, 113–125.

    Article  CAS  PubMed  Google Scholar 

  • McGrew, J. H., & Bond, G. R. (1997). The association between program characteristics and service delivery in assertive community treatment. Administration and Policy in Mental Health and Mental Health Services, 25, 175–189.

    Article  CAS  Google Scholar 

  • McGrew, J. H., Bond, G. R., Dietzen, L. L., & Salyers, M. P. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62(4), 670–678.

    Article  CAS  PubMed  Google Scholar 

  • McGrew, J. H., Stull, L. G., Rollins, A. L., Salyers, M. P., & Hicks, L. J. (2011). A comparison of phone-based and on-site assessment of fidelity for assertive community treatment in Indiana. Psychiatric Services, 62(6), 670–674. doi:10.1176/appi.ps.62.6.670.

    Article  PubMed Central  PubMed  Google Scholar 

  • McGrew, J. H., White, L. M., & Stull, L. G. (2013a). Self-assessed fidelity: Proceed with caution: In reply. Psychiatric Services, 64(4), 394. doi:10.1176/appi.ps.640419.

    Article  PubMed  Google Scholar 

  • McGrew, J. H., White, L. M., Stull, L. G., & Wright-Berryman, J. (2013b). A comparison of self-reported and phone-administered methods of ACT fidelity assessment: A pilot study in Indiana. Psychiatric Services, 64(3), 272–276. doi:10.1176/appi.ps.001252012.

    Article  PubMed Central  PubMed  Google Scholar 

  • McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the new hampshire dual disorders study. Psychiatric Services, 50(6), 818–824.

    Article  CAS  PubMed  Google Scholar 

  • McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., & Finnerty, M. T. (2007). Fidelity outcomes in the national implementing evidence-based practices project. Psychiatric Services, 58(10), 1279–1284.

    Article  PubMed  Google Scholar 

  • Mueser, K. T., Bond, G. R., Drake, R. E., & Resnick, S. G. (1998). Models of community care for severe mental illness: A review of research on case management. Schizophrenia Bulletin, 24, 37–74.

    Article  CAS  PubMed  Google Scholar 

  • Neale, M. S., Rosenheck, R., Castrodonatti, J., Martin, A., Morrissey, J., & D’Amico, M. (2007). Mental health intensive case management (MHICM) in the department of veterans affairs: The tenth national performance monitoring report—FY 2006. West Haven: VHA NEPEC Report.

    Google Scholar 

  • Phillips, S. D., Burns, B. J., Edgar, E. R., Mueser, K. T., Linkins, K. W., Rosenheck, R. A., et al. (2001). Moving assertive community treatment into standard practice. Psychiatric Services, 52(6), 771–779.

    Article  CAS  PubMed  Google Scholar 

  • President’s New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville: Substance Abuse and Mental Health Services Administration.

  • Ransohoff, D. F., & Feinstein, A. R. (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New England Journal of Medicine, 299(17), 926–930.

    Article  CAS  PubMed  Google Scholar 

  • Rosenheck, R. A., & Neale, M. S. (1998). Cost-effectiveness of intensive psychiatric community care for high users of inpatient services. Archives of General Psychiatry, 55(5), 459–466.

    Article  CAS  PubMed  Google Scholar 

  • Rosenheck, R., Neale, M. S., Leaf, P., Milstein, R., & Frisman, L. (1995). Multisite experimental cost study of intensive psychiatric community care. Schizophrenia Bulletin, 21, 129–140.

    Article  CAS  PubMed  Google Scholar 

  • Salyers, M. P., Bond, G. R., Teague, G. B., Cox, J. F., Smith, M. E., Hicks, M. L., & Koop, J. I. (2003). Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. Journal of Behavioral Health Services and Research, 30(3), 304–320.

    Article  PubMed  Google Scholar 

  • Salyers, M. P., McKasson, R. M., Bond, G. R., McGrew, J. H., Rollins, A. L., & Boyle, C. (2007). The role of technical assistance centers in implementing evidence-based practices: Lessons learned. American Journal of Psychiatric Rehabilitation, 10(2), 85–101.

    Article  Google Scholar 

  • Stein, L. I., & Test, M. A. (1980). An alternative to mental health treatment. I: Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37, 392–397.

    Article  CAS  PubMed  Google Scholar 

  • Stemler, S. E. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation.(2004); 9(4), 66–78. Retrieved January 2, 2015 from http://PAREonline.net/getvn.asp?v=9&n=4.

  • Teague, G. B., Bond, G. R., & Drake, R. E. (1998). Program fidelity in assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68(2), 216–232.

    Article  CAS  PubMed  Google Scholar 

  • Torrey, W. C., Finnerty, M., Evans, A., & Wyzik, P. F. (2003). Strategies for leading the implementation of evidence-based practices. Psychiatric Clinics of North America, 26, 883–897.

    Article  PubMed  Google Scholar 

  • Ziguras, S., & Stuart, G. (2000). A meta-analysis of the effectiveness of mental health case management over 20 years. Psychiatric Services, 51, 1410–1421.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by VA Health Services Research and Development IIR 09-368.

Author information

Authors and Affiliations

  1. HSR&D Center for Health Information and Communication, Richard L. Roudebush VAMC, 1481 W. 10th Street, 11-H, Indianapolis, IN, 46202, USA

    Angela L. Rollins, Marina Kukla, Alan B. McGuire, Mindy E. Flanagan & Linda A. Collins

  2. Department of Psychology, Indiana University Purdue University Indianapolis, Indianapolis, USA

    Angela L. Rollins, John H. McGrew, Marina Kukla, Alan B. McGuire & Michelle P. Salyers

  3. Office of Mental Health Operations, VA Northeast Program Evaluation Center, West Haven, USA

    Marcia G. Hunt

  4. College of Medicine, Penn State University, University Park, USA

    Doug L. Leslie

  5. School of Social Work, University of Cincinnati, Cincinnati, USA

    Jennifer L. Wright-Berryman

  6. Adult & Child Center, Indianapolis, USA

    Lia J. Hicks

Authors
  1. Angela L. Rollins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John H. McGrew
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marina Kukla
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alan B. McGuire
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mindy E. Flanagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marcia G. Hunt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Doug L. Leslie
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Linda A. Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jennifer L. Wright-Berryman
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lia J. Hicks
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Michelle P. Salyers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Angela L. Rollins.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rollins, A.L., McGrew, J.H., Kukla, M. et al. Comparison of Assertive Community Treatment Fidelity Assessment Methods: Reliability and Validity. Adm Policy Ment Health 43, 157–167 (2016). https://doi.org/10.1007/s10488-015-0641-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-015-0641-1

Keywords

Search

Navigation