Skip to main content
SpringerLink
Log in

Psychometrics, Reliability, and Validity of a Wraparound Team Observation Measure

  • Original Paper
  • Published:
Journal of Child and Family Studies Aims and scope Submit manuscript

Abstract

Wraparound is a widely-implemented team-based care coordination process for youth with serious emotional and behavioral needs. Wraparound has a positive evidence base; however, research has shown inconsistency in the quality of its implementation that can reduce its effectiveness. The current paper presents results of three studies used to examine psychometrics, reliability, and validity of a measure of wraparound fidelity as assessed during team meetings called the Team Observation Measure (TOM). Analysis of TOM results from 1,078 team observations across 59 sites found good overall internal consistency (α = 0.80), but constrained variability, with the average team rated as having 78 % of indicators of model adherent wraparound present, 11 % absent, and 11 % not applicable. A study of N = 23 pairs of raters found a pooled Kappa statistic of 0.733, indicating substantial inter-rater reliability. Higher agreement was found between external evaluators than for pairs of raters that included an external evaluator and an internal rater (e.g., supervisor or coach). A validity study found no correlation between the TOM and an alternate fidelity instrument, the Wraparound Fidelity Index (WFI), at the team level. However, positive correlations between mean program-level TOM and WFI scores provide support for TOM validity as a summative assessment of site- or program-level fidelity. Implications for TOM users, measure refinement, and future research are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aspland, H., & Gardner, F. (2003). Observational measures of parent child interaction. Child and Adolescent Mental Health, 8, 136–144.

    Article  Google Scholar 

  • Bruns, E. J. (2008). Measuring wraparound fidelity. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.

    Google Scholar 

  • Bruns, E. J., Burchard, J., Suter, J., & Force, M. D. (2005). Measuring fidelity within community treatments for children and families. In M. Epstein, A. Duchnowski, & K. Kutash (Eds.), Outcomes for children and youth with emotional and behavioral disorders and their families (Vol. 2). Austin, TX: Pro-ED.

    Google Scholar 

  • Bruns, E. J., Burchard, J. D., Suter, J. C., Leverentz-Brady, K. M., & Force, M. M. (2004). Assessing fidelity to a community-based treatment for youth: The wraparound fidelity index. Journal of Emotional & Behavioral Disorders, 12(2), 79.

    Article  Google Scholar 

  • Bruns, E. J., Pullmann, M. P., Brinson, R. D., Sather, A., & Ramey, M. (2014). Effectiveness of wraparound vs. case management for children and adolescents: Results of a randomized study. Manuscript submitted for publication.

  • Bruns, E. J., & Sather, A. (2007). User’s manual to the wraparound team observation measure. Seattle, WA: University of Washington, Wraparound Evaluation and Research Team, Division of Public Behavioral Health and Justice Policy.

    Google Scholar 

  • Bruns, E. J., & Sather, A. (2013). Team Oversation Measure (TOM) manual for use and scoring. Wraparound fidelity assessment system (WFAS). Seattle, WA: University of Washington School of Medicine, Division of Public Behavioral Health and Justice Policy.

    Google Scholar 

  • Bruns, E. J., Sather, A., & Pullmann, M. D. (2010). The wraparound fidelity assessment system-psychometric analyses to support refinement of the wraparound fidelity index and team observation measure. Paper presented at the the 23rd Annual Children’s Mental Health Research and Policy Conference, Tampa, FL.

  • Bruns, E. J., Sather, A., Pullmann, M. D., & Stambaugh, L. F. (2011). National trends in implementing wraparound: results from the state wraparound survey. Journal of Child and Family Studies, 20(6), 726–735. doi:10.1007/s10826-011-9535-3.

    Article  Google Scholar 

  • Bruns, E. J., & Suter, J. C. (2010). Summary of the wraparound evidence base. In E. J. Bruns & J. S. Walker (Eds.), The resouce guide to wraparound. National Wraparound Initiative: Portland, OR.

    Google Scholar 

  • Bruns, E. J., Suter, J. C., Force, M. M., Sather, A., & Leverentz-Brady, K. M. (2009). Wraparound Fidelity Index 4.0: Manual for training, administration, and scoring of the WFI 4.0. Seattle: Division of Public Behavioral Health and Justice Policy, University of Washington.

    Google Scholar 

  • Bruns, E. J., Suter, J. C., & Leverentz-Brady, K. M. (2006). Relations between program and system variables and fidelity to the wraparound process for children and families. Psychiatric Services, 57(11), 1586–1593. doi:10.1176/appi.ps.57.11.1586.

    Article  PubMed  Google Scholar 

  • Bruns, E. J., Walker, J. S., Bernstein, A., Daleiden, E., Pullmann, M. D., & Chorpita, B. F. (2013). Family voice with informed choice: coordinating wraparound with research-based treatment for children and adolescents. Journal of Clinical Child & Adolescent Psychology. (Online First Publishing),. doi:10.1080/15374416.2013.859081.

    Google Scholar 

  • Bruns, E. J., Walker, J. S., & The National Wraparound Initiative Advisory Group. (2008). Ten principles of the wraparound process. In E. J. Bruns & J. S. Walker (Eds.), Resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.

    Google Scholar 

  • Bruns, E. J., Walker, J. S., Zabel, M., Estep, K., Matarese, M., Harburger, D., et al. (2010b). The wraparound process as a model for intervening with youth with complex needs and their families. American Journal of Community Psychology, 46(3–4), 314–331. doi:10.1007/s10464-010-9346-5.

    Article  PubMed Central  PubMed  Google Scholar 

  • Burchard, J. D., Bruns, E. J., & Burchard, S. N. (2002). The wraparound process. In B. J. Burns, K. Hoagwood, & M. English (Eds.), Community-based interventions for youth (pp. 69–90). New York: Oxford University Press.

    Google Scholar 

  • Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7–18.

    Article  PubMed  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

    Article  Google Scholar 

  • Cooper, J. L., Aratani, Y., Knitzer, J., Douglas-Hall, A., Masi, R., Banghart, P., et al. (2008). Unclaimed children revisited: The status of children’s mental health policy in the United States. New York: National Center for Children. in Poverty.

    Google Scholar 

  • Cox, K., Baker, D., & Wong, M. A. (2010). Wraparound retrospective: Factors predicting positive outcomes. Journal of Emotional & Behavioral Disorders, 18(1), 3–13.

    Article  Google Scholar 

  • De Vries, H., Elliott, M. N., Kanouse, D. E., & Teleki, S. S. (2008). Using pooled kappa to summarize interrater agreement across many Items. Field Methods, 20(3), 272–282.

    Article  Google Scholar 

  • Eames, C. C., Daley, D. D., Hutchings, J. J., Hughes, J. C., Jones, K. K., Martin, P. P., et al. (2008). The leader observation tool: A process skills treatment fidelity measure for the incredible years parenting programme. Child: Care, Health and Development, 34(3), 391–400. doi:10.1111/j.1365-2214.2008.00828.x.

    Google Scholar 

  • Effland, V. S., Walton, B. A., & McIntyre, J. S. (2011). Connecting the dots: Stages of implementation, wraparound fidelity and youth outcomes. Journal of Child and Family Studies, 20(6), 736–746. doi:10.1007/s10826-011-9541-5.

    Article  Google Scholar 

  • Epstein, M. H., Jayanthi, M., McKelvey, J., Frankenberry, E., Hary, R., Potter, K., et al. (1998). Reliability of the wraparound observation form: An instrument to measure the wraparound process. Journal of Child and Family Studies, 7, 161–170.

    Article  Google Scholar 

  • Epstein, M. H., & Sharma, J. M. (1998). Behavioral and emotional rating scale: A strength-based approach to assessment. Austin, TX: PRO-ED.

    Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

    Google Scholar 

  • Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse and Neglect, 22(5), 401–421. doi:10.1016/S0145-2134(98)00005-2.

    Article  PubMed  Google Scholar 

  • Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press.

    Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  PubMed  Google Scholar 

  • Lyons, J. S., Weiner, D. A., & Lyons, M. B. (2004). Measurement as communication in outcomes management: The child and adolescent needs and strengths (CANS). The Use of Psychological Testing for Treatment Planning and Outcomes Assessment, 3, 461–476.

    Google Scholar 

  • Miles, P., Brown, N., & The National Wraparound Initiative Implementation Work Group. (2011). The wraparound implementation guide: A handbook for administrators and managers. Portland, OR: National Wraparound Initative.

    Google Scholar 

  • Mitchell, P. F. (2011). Evidence-based practice in real-world services for young people with complex needs: New opportunities suggested by recent implementation science. Children and Youth Services Review, 33, 207–216. doi:10.1016/j.childyouth.2010.10.003.

    Article  Google Scholar 

  • Nordness, P. D., & Epstein, M. H. (2003). Reliability of the wraparound observation form-second version: An instrument designed to assess the fidelity of the Wraparound approach. Mental Health Services Research, 5, 89–96.

    Article  PubMed  Google Scholar 

  • Proctor, E. K., Landsverk, J., Aarons, G. A., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy In Mental Health, 36(1), 24–34.

    Article  PubMed  Google Scholar 

  • Pullmann, M., Bruns, E. J., & Sather, A. (2013). Evaluating fidelity to the wraparound service model for youth: Application of item response theory to the wraparound fidelity index. Psychological Assessment, 25(2), 583–598.

    Article  PubMed Central  PubMed  Google Scholar 

  • Sather, A., & Bruns, E. J. (2008). Wraparound fidelity index 4.0: Interviewer training toolkit. Seattle: Division of Public Behavioral Health and Justice Policy, University of Washington.

    Google Scholar 

  • Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s … fidelity measurement in the real world. American Journal of Orthopsychiatry, 76, 304–311.

    Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43.

    Article  PubMed Central  PubMed  Google Scholar 

  • Singh, N. N., Curtis, W. J., Wechsler, H. A., Ellis, C. R., & Cohen, R. (1997). Family friendliness of community-based services for children and adolescents with emotional and behavioral disorders and their families: An observational study. Journal of Emotional and Behavioral Disorders, 5(2), 82–92. doi:10.1177/106342669700500203.

    Article  Google Scholar 

  • Snyder, E. H., Lawrence, N., & Dodge, K. A. (2012). The impact of system of care support in adherence to wraparound principles in child and family teams in child welfare in North Carolina. Children and Youth Services Review, 34(4), 639–647.

    Article  PubMed Central  PubMed  Google Scholar 

  • Stroul, B. A. (2002). Issue brief—system of care: A framework for system reform in children’s mental health. Washington, DC: Georgetown University Child Development Center, National Technical Assistance Center for Children’s Mental Health.

    Google Scholar 

  • Stroul, B. A., & Friedman, R. M. (1986). A system of care for severely emotionally disturbed children and youth. Tampa, FL: University of South Florida, Tampa Research Training Center for Improved Services for Seriously Emotionally Disturbed Children and Georgetown Univ. Child Development Center, Washington D. C. Cassp Technical Assistance Center.

    Google Scholar 

  • Suter, J. C., & Bruns, E. J. (2009). Effectiveness of the wraparound process for children with emotional and behavioral disorders: A meta-analysis. Clincial Child and Family Psychology Review, 12(4), 336–351.

    Article  Google Scholar 

  • United States Public Health Service (USPHS). (1999). Mental health: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

    Google Scholar 

  • van Dijk, J. (1990). Delphi method as a learning instrument: Bank employees discussing an automation project. Technological Forecasting and Social Change, 37, 399–407.

    Article  Google Scholar 

  • VanDenBerg, J., Bruns, E., & Burchard, J. (2003). History of the wraparound process. Focal Point: A National Bulletin on Family Support and Children’s Mental Health: Quality and Fidelity in Wraparound, 17(2), 4–7.

    Google Scholar 

  • Vetter, J. & Strech, G (2012, March). Using the ohio scales for assessment and outcome measurement in a statewide system of care. Paper presented at the 25th Annual Children’s Mental Health Research and Policy Conference, Tampa, FL.

  • Walker, J. S., & Bruns, E. J. (2006). Building on practice-based evidence: Using expert perspectives to define the wraparound process. Psychiatric Services, 57, 1579–1585.

    Article  PubMed  Google Scholar 

  • Walker, J. S., Bruns, E. J., Conlan, L., & LaForce, C. (2011). The National Wraparound Initiative: A community-of-practice approach to building knowledge in the field of children’s mental health. Best Practices in Mental Health, 7(1), 26–46.

    Google Scholar 

  • Walker, J. S., Bruns, E. J., & Penn, M. (2008a). Individualized services in systems of care: The wraparound process. In B. A. Stroul & G. M. Blau (Eds.), The system of care handbook: Transforming mental health services for children, youth, and families. Baltimore, MD: Paul H. Brookes Publishing Company.

    Google Scholar 

  • Walker, J. S., Bruns, E. J., & The National Wraparound Initiative Advisory Group. (2008b). Phases and activities of the wraparound process. In E. J. Bruns & J. S. Walker (Eds.), Resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.

    Google Scholar 

  • Webster-Stratton, C., & Hancock, L. (1998). Training for parents of young children with conduct problems: Contents, methods and therapeutic processes. In G. E. Schaefer & J. M. Breismeister (Eds.), Handbook of parent training (pp. 98–152). New York: Wiley.

    Google Scholar 

  • Williams, N. J., & Glisson, C. (2013). Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: A united states national study. Child Abuse & Neglect (Online first publication),. doi:10.1016/j.chiabu.2013.09.003.

    Google Scholar 

Download references

Acknowledgments

This study was supported in part by grant R34 MH072759 from the National Institute of Mental Health. We would like to thank our national collaborators and the dozens of trained TOM observers in these wraparound initiatives nationally. Thanks also to the wraparound initiatives in Clark County, Nevada and King County, Washington for their collaboration on the inter-rater reliability studies.

Author information

Authors and Affiliations

  1. Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine, 2815 Eastlake Ave. E, Suite 200, Seattle, WA, 98102, USA

    Eric J. Bruns, Ericka S. Weathers, Spencer Hensley, Michael D. Pullmann & April Sather

  2. Center on Disability and Community Inclusion, University of Vermont, Burlington, VT, USA

    Jesse C. Suter

Authors
  1. Eric J. Bruns
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ericka S. Weathers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jesse C. Suter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Spencer Hensley
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael D. Pullmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. April Sather
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric J. Bruns.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bruns, E.J., Weathers, E.S., Suter, J.C. et al. Psychometrics, Reliability, and Validity of a Wraparound Team Observation Measure. J Child Fam Stud 24, 979–991 (2015). https://doi.org/10.1007/s10826-014-9908-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10826-014-9908-5

Keywords

Search

Navigation