Skip to main content
SpringerLink
Log in

Exploring the Black Box: Measuring Youth Treatment Process and Progress in Usual Care

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

The session report form (SRF) is a 25-item self-report measure scheduled to be completed by clinicians at the end of each session. This study addresses the utility of the SRF for describing session content by exploring patterns of self-reported behavior of 235 clinicians in 7,058 sessions with 600 clients. Results indicate that the SRF has a distinct subscale of treatment process and provides an adequate range of topics related to session content while also discriminating between client and clinician influences on patterns of topics addressed. The SRF is a promising measure meeting a significant need in describing session content in usual care.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. The caseload per clinician includes only clients who were entered into the CFIT system; clinicians may have had additional clients with whom they did not use CFIT.

References

  • Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research, 6(2), 61–74.

    Article  PubMed  Google Scholar 

  • Addis, M. E., & Krasnow, A. D. (2000). A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. Journal of Consulting and Clinical Psychology, 68(2), 331–339.

    Article  CAS  PubMed  Google Scholar 

  • Angold, A., Costello, E. J., Burns, B. J., Erkanli, A., & Farmer, E. M. Z. (2000). Effectiveness of non-residential specialty mental health services for children and adolescents in the ‘real world’. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 154–160.

    Article  CAS  PubMed  Google Scholar 

  • Bearsley-Smith, C., Sellick, K., Chesters, J., & Francis, K. (2008). Treatment content in child and adolescent mental health services: Development of the treatment recording sheet. Administration and Policy in Mental Health and Mental Health Services Research, 35, 423–435. doi:10.1007/s10488-008-0184-9.

    Article  PubMed  Google Scholar 

  • Bickman, L. (1996). A continuum of care: More is not always better. American Psychologist, 51(7), 689–701.

    Article  CAS  PubMed  Google Scholar 

  • Bickman, L. (1999). Practice makes perfect and other myths about mental health services. American Psychologist, 54, 958–973.

    Article  Google Scholar 

  • Bickman, L. (2000). Improving children’s mental health: How no effects can affect policy. Report on emotional & behavioral disorders in youth: Evidence-based assessment, interventions for the real world (Vol. 3–4, pp 21–23).

  • Bickman, L., Riemer, M., Lambert, E. W., Kelley, S. D., Breda, C., Dew, S. E., et al. (Eds.) (2007). Manual of the Peabody treatment progress battery [Electronic version]. Nashville, TN: Vanderbilt University. http://peabody.vanderbilt.edu/ptpb/.

  • Brookman-Frazee, L., Garland, A. F., Taylor, R., & Zoffness, R. (2009). Therapists’ attitudes towards psychotherapeutic strategies in community-based psychotherapy with children w/disruptive behavior problems. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 1–12. doi:10.1007/s10488-008-0195-6.

    Article  PubMed  Google Scholar 

  • Brookman-Frazee, L., Haine, R. A., Baker-Ericzen, M., Zoffness, R, & Garland, A. F. (2010). Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-009-0244-9.

  • Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

    Google Scholar 

  • Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245–276.

    Article  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005a). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Service Research, 7(1), 5–20. doi:10.1007/s11020-005-1962-6.

    Article  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005b). Modularity in the design and application of therapeutic interventions. Applied and Preventive Psychology, 11, 141–156. doi:10.1016/j.appsy.2005.05.002.

    Article  Google Scholar 

  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.

    Article  CAS  PubMed  Google Scholar 

  • Crits-Christoph, P., Beebe, K. L., & Connolly, M. B. (1990). Therapist effects in the treatment of drug dependence: Implications for conducting comparative treatment studies. National Institutes on Drug Abuse Research Monograph, 104.

  • Cronbach, L. J. (1988). Internal consistency of tests: Analyses old and new. Psychometrika, 53(1), 63–70.

    Article  Google Scholar 

  • Garland, A. F., Hawley, K. M., Brookman-Frazee, L., & Hurlburt, M. S. (2008). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. Journal of the American Academy of Child & Adolescent Psychiatry, 47(5), 505–514. doi:10.1097/CHI.0b013e31816765c2.

    Article  Google Scholar 

  • Goldberger, A. S. (1991). A Course in Econometrics. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Guttman, L. (1954). Some necessary conditions for common factor analysis. Psychometrika, XIX, 149–161.

    Article  Google Scholar 

  • Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71(1), 62–70.

    Article  PubMed  Google Scholar 

  • Hepner, K. A., Azocar, F., Greenwood, G. L., Miranda, J., & Burnam, M. A. (2010). Development of a clinician report to assess psychotherapy for depression in usual care settings. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-009-0249-4.

  • Hunsley, J., & Mash, E. J. (2008). Developing criteria for evidence-based assessment: An introduction to assessments that work. In J. Hunsley & E. J. Mash (Eds.), A guide to assessments that work. Oxford series in clinical psychology (pp. 3–14). New York, NY: Oxford University Press.

    Google Scholar 

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-009-0251-x.

  • Jensen, P. S., Weersing, R., Hoagwood, K. E., & Goldman, E. (2005). What is the evidence for evidence-based treatments? A hard look at our soft underbelly. Mental Health Services Research, 7(1), 53–74.

    Article  PubMed  Google Scholar 

  • Kaiser, H. F. (1961). A note on Guttman’s lower bound for the number of common factors. British Journal of Psychology, 14, 1–2.

    Google Scholar 

  • Karver, M. S., Handelsman, J. B., Fields, S., & Bickman, L. (2005). A theoretical model of common process factors in youth and family therapy. Mental Health Services Research, 7(1), 35–51.

    Article  PubMed  Google Scholar 

  • Karver, M. S., Handelsman, J., Fields, S., & Bickman, L. (2006). Meta-analysis of therapeutic relationship variables in youth and family therapy: The evidence for different relationship variables in the child and adolescent treatment outcome literature. Clinical Psychology Review, 26(1), 50–65.

    Article  PubMed  Google Scholar 

  • Kelley, S. D., & Bickman, L. (2009). Beyond outcomes monitoring: Measurement feedback systems (MFS) in child and adolescent clinical practice. Current Opinion in Psychiatry, 22(4), 363–368. doi:10.1097/YCO.0b013e32832c9162.

    Article  PubMed  Google Scholar 

  • Kelley, S. D., Bickman, L., & Norwood, E. (2010). Evidence-based treatments and common factors in youth psychotherapy. In B. L. Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble (Eds.), The heart and soul of change: Delivering what works in therapy (2nd ed., pp. 325–355). Washington, DC: American Psychological Association.

    Google Scholar 

  • Kennedy, E. (2009). Editorial: Evaluating interventions in child mental health: The importance of care provider and contextual influences. Clinical Child Psychology and Psychiatry, 14, 163–165. doi:10.1177/1359104508100881.

    Article  PubMed  Google Scholar 

  • Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Institute.

    Google Scholar 

  • Lord, F. M., Novick, M. R., & Numerous, C. (1968). Statistical theories of mental test scores. Oxford, UK: Addison-Wesley.

    Google Scholar 

  • Miller, S., Wampold, B., & Varhely, K. (2008). Direct comparison of treatment modalities for youth disorders: A meta-analysis. Psychotherapy Research, 18(1), 5–18.

    Article  PubMed  Google Scholar 

  • Minami, T., & Wampold, B. E. (2008). Adult psychotherapy in the real world. In W. B. Walsh (Ed.), Biennial review of counseling psychology (Vol. 1, pp. 27–45). New York: Taylor & Francis.

    Google Scholar 

  • Miranda, J., Hepner, K. A., Azocar, F., Greenwood, G., Ngo, V., & Burnam, M. A. (2010). Development of a patient-report measures of psychotherapy for depression. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-009-0238-7.

  • Nakamura, B. J., Daleiden, E. L., & Mueller, C. W. (2007). Validity of treatment target progress ratings as indicators of youth improvement. Journal of Child and Family Studies, 16(5), 729–741. doi:10.1007/s10826-006-9119-9.

    Article  Google Scholar 

  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill.

    Google Scholar 

  • Phares, V., & Curley, J. (2008). Evidence-based assessment for children and adolescents. In R. G. Steele, T. D. Elkin, & M. C. Roberts (Eds.), Handbook of evidence-based therapies for children and adolescents: Bridging science and practice issues in clinical child psychology (pp. 537–549). New York, NY: Springer.

    Google Scholar 

  • Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213.

    Article  CAS  PubMed  Google Scholar 

  • Riemer, M., Rosof-Williams, J., & Bickman, L. (2005). Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America, 14, 241–254.

    Article  PubMed  Google Scholar 

  • Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 6, 145–153.

    Article  Google Scholar 

  • Shadish, W. R., Navarro, A. M., Matt, G. E., & Phillips, G. (2000). The effects of psychological therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin, 126(4), 512–529.

    Article  CAS  PubMed  Google Scholar 

  • Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press.

    Google Scholar 

  • Sufka, K. J., Feltenstein, M. W., Warnick, J. E., Acevado, E. O., Webb, H. E., & Cartwright, C. M. (2006). Modeling the anxiety-depression continuum hypothesis in domestic fowl chicks. Behavioural Pharmacology, 17(8), 681–689. doi:10.1097/FBP.0b013e3280115fac.

    Article  PubMed  Google Scholar 

  • Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Psychology, 31(2), 168–180. doi:10.1207/153744202753604458.

    Google Scholar 

  • Weiss, B., Catron, T., Harris, V., & Phung, T. M. (1999). The effectiveness of traditional child psychotherapy. Journal of Consulting and Clinical Psychology, 67(1), 82–94.

    Article  CAS  PubMed  Google Scholar 

  • Weisz, J. R., & Gray, J. S. (2008). Evidence-based psychotherapy for children and adolescents: Data from the present and a model for the future. Child and Adolescent Mental Health, 13(2), 54–65. doi:10.1111/j.1475-3588.2007.00475.x.

    Article  Google Scholar 

  • Weisz, J. R., Jensen, A. L., & McLeod, B. D. (2005). Development and dissemination of child and adolescent psychotherapies: Milestones, methods, and a new deployment-focused model. In D. Hibbs & P. S. Jensen (Eds.), Psychosocial treatments for child, adolescent disorders: Empirically based strategies for clinical practice (2nd ed.). Washington, DC: APA.

    Google Scholar 

  • Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006a). Evidence-based youth psychotherapies versus clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671–689.

    Article  PubMed  Google Scholar 

  • Weisz, J. R., McCarty, C. A., & Valeri, S. M. (2006b). Effects of psychotherapy for depression in children and adolescents: A meta-analysis. Psychological Bulletin, 132, 132–149.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Preparation of this article was partially supported by a grant from NIMH (MH068589). The authors wish to thank Warren Lambert for his expert guidance on statistical approaches used. We also wish to thank Susan Casey, Carol Hurst, Laurie Macrae, and Tasha Walsh for their expert clinical consultation.

Author information

Authors and Affiliations

  1. Center for Evaluation and Program Improvement, Vanderbilt University, Peabody #151, 230 Appleton Place, Nashville, TN, 37203-5721, USA

    Susan Douglas Kelley, Ana Regina Vides de Andrade, Emily Sheffer & Leonard Bickman

Authors
  1. Susan Douglas Kelley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ana Regina Vides de Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emily Sheffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leonard Bickman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susan Douglas Kelley.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kelley, S.D., de Andrade, A.R.V., Sheffer, E. et al. Exploring the Black Box: Measuring Youth Treatment Process and Progress in Usual Care. Adm Policy Ment Health 37, 287–300 (2010). https://doi.org/10.1007/s10488-010-0298-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-010-0298-8

Keywords

Search

Navigation