Skip to main content

Advertisement

SpringerLink
Log in

Inter-rater Reliability of Clinician-rated Outcome Measures in Child and Adolescent Mental Health Services

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

This study investigated the inter-rater reliability when 169 out of 171 clinicians working in 10 Norwegian child and adolescent mental health services rated 20 written vignettes using the following outcome measures: Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA), Children’s Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD). Three clinicians rated both patients and vignettes. On vignettes the intraclass correlation coefficient (ICC) for the HoNOSCA total score was 0.81 (single scales 0.47–0.96), for the CGAS 0.61 and for the GAPD 0.60. The reliability was not lower on patients. The rater’s profession, experience or clinic did not have effect on the scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • American Psychiatric Association (2000). Diagnostic and statistical manual of mental disorders (4th ed., Text Revision ed.). Washington, DC: American Psychiatric Association.

  • Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.). Prentice Hall International.

  • Australian Mental Health Outcomes and Classification Network (2005). Child & Adolescent National Outcomes & Casemix Collection Standard Reports, Version 1.1. Brisbane, Queensland.

  • Bates, M. P. (2001). The child and adolescent functional assessment scale (CAFAS): Review and current status. Clinical Child & Family Psychology Review, 4(1), 63–84.

    Article  CAS  Google Scholar 

  • Bickman, L., Guthrie, P. R., Foster, E. M., Lambert, E. W., Summerfelt, W., Breda, C. S., et al. (1995). Evaluating managed mental health services: The Fort Bragg experiment. New York, NY, US: Plenum Press.

    Google Scholar 

  • Bilenberg, N. (2003). Health of the nation outcome scales for children and adolescents (HoNOSCA). Results of a Danish field trial. European Child & Adolescent Psychiatry, 12(6), 298–302.

    Article  Google Scholar 

  • Brann, P., Coleman, G., & Luk, E. (2001). Routine outcome measurement in a child and adolescent mental health service: An evaluation of HoNOSCA. The Australian and New Zealand Journal of Psychiatry, 35(3), 370–376.

    Article  CAS  PubMed  Google Scholar 

  • CAFAS in Ontario (2005). Child and adolescent functional assessment scale. http://www.cafasinontario.ca/html/home.asp, access date 12.4.2007.

  • CAMHS Outcome Research Consortium (2007). CORC homepage. http://www.corc.uk.net/, access date 19.3.2007.

  • Dunn, G. (1989). Design and analysis of reliability studies: The statistical evaluation of measurement errors. New York: Oxford University Press.

    Google Scholar 

  • Dyrborg, J., Larsen, F. W., Nielsen, S., Byman, J., Nielsen, B. B., & Gautre-Delay, F. (2000). The children’s global assessment scale (CGAS) and global assessment of psychosocial disability (GAPD) in clinical practice—substance and reliability as judged by intraclass correlations. European Child & Adolescent Psychiatry, 9(3), 195–201.

    Article  CAS  Google Scholar 

  • Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall.

  • Epstein, M. H., Kutash, K., & Duchnowski, A. J. (2005). Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices (2nd ed.). Austin, TX: Pro-Ed.

  • Fonagy, P. (2002). Outcome measurement in children and adolescents. In W. W. IsHak, T. Burt, & L. I. Sederer (Eds.), Outcome measurement in psychiatry: a critical review (pp. 59–75). Washington, DC: American Psychiatric Publishing, Inc.

    Google Scholar 

  • Ford, T., Tingay, K., Wolpert, M., & the CORC Steering Group. (2006). CORC’s survey of routine outcome monitoring and national CAMHS dataset developments: A response to Johnston and Gowers. Child and Adolescent Mental Health, 11(1), 50–52.

    Article  Google Scholar 

  • Garralda, M. E., Yates, P., & Higginson, I. (2000). Child and adolescent mental health service use. HoNOSCA as an outcome measure. British Journal of Psychiatry, 177, 52–58.

    Article  CAS  PubMed  Google Scholar 

  • Gowers, S. G., Harrington, R. C., Whitton, A., Beevor, A., Lelliott, P., Jezzard, R., et al. (1999a). Health of the nation outcome scales for children and adolescents (HoNOSCA). Glossary for HoNOSCA score sheet. British Journal of Psychiatry, 174, 428–431.

    CAS  PubMed  Google Scholar 

  • Gowers, S. G., Harrington, R. C., Whitton, A., Lelliott, P., Beevor, A., Wing, J., et al. (1999b). Brief scale for measuring the outcomes of emotional and behavioural disorders in children. Health of the nation outcome scales for children and adolescents (HoNOSCA). British Journal of Psychiatry, 174, 413–416.

    CAS  PubMed  Google Scholar 

  • Gowers, S., Bailey-Rogers, S. J., Shore, A., & Levine, W. (2000). The health of the nation outcome scales for child & adolescent mental health (HoNOSCA). Child Psychology & Psychiatry Review, 5(2), 50–56.

    Article  Google Scholar 

  • Green, B., Shirk, S., Hanze, D., & Wanstrath, J. (1994). The children’s global assessment scale in clinical practice: An empirical evaluation. Journal of the American Academy of Child and Adolescent Psychiatry, 33(8), 1158–1164.

    Article  CAS  PubMed  Google Scholar 

  • Hoagwood, K., Jensen, P. S., Petti, T., & Burns, B. J. (1996). Outcomes of mental health care for children and adolescents: I. A comprehensive conceptual model. Journal of the American Academy of Child and Adolescent Psychiatry, 35(8), 1055–1063.

    Article  CAS  PubMed  Google Scholar 

  • Hodges, K., & Gust, J. (1995). Measures of impairment for children and adolescents. Journal of Mental Health Administration, 22(4), 403–413.

    Article  CAS  PubMed  Google Scholar 

  • Hodges, K., Wong, M. M., & Latessa, M. (1998) Use of the child and adolescent functional assessment scale (CAFAS) as an outcome measure in clinical settings. Journal of Behavioral Health Services & Research, 25(3), 325–336.

    Article  CAS  Google Scholar 

  • Hodges, K., & Wotring, J. (2004) The role of monitoring outcomes in initiating implementation of evidence-based treatments at the state level. Psychiatric Services, 55(4), 396–400.

    Article  PubMed  Google Scholar 

  • HoNOSCA Project (2003). Health of the nation outcome scales for children and adolescents. http://www.liv.ac.uk/honosca/Home.htm, access date 16.4.2007.

  • Jensen, P. S., Hoagwood, K., & Petti, T. (1996). Outcomes of mental health care for children and adolescents: II. Literature review and application of a comprehensive model. Journal of the American Academy of Child and Adolescent Psychiatry, 35(8), 1064–1077.

    Article  CAS  PubMed  Google Scholar 

  • Johnston, C., & Gowers, S. (2005). Routine outcome measurement: A survey of UK child and adolescent mental health services. Child and Adolescent Mental Health, 10(3), 133–139.

    Article  Google Scholar 

  • Kazdin, A. E. (2003). Psychotherapy for children and adolescents. Annual Review of Psychology, 54, 253–276.

    Article  PubMed  Google Scholar 

  • MH-SMART (2007). The mental health standard measures of assessment and recovery summary. http://www.tepou.co.nz/page/tepou_11.php, access date 19.3.2007.

  • National Mental Health Strategy. (2004). Australian mental health outcomes and classification network. http://www.mhnocc.org/amhocn/, access date 14.3.2007.

  • Norcini, J. (2004) Back to the future: clinical vignettes and the measurement of physician performance. Annals of Internal Meicine, 141(10), 813–814.

    Google Scholar 

  • Peabody, J. W., Luck, J., Glassman, P., Jain, S., Hansen, J., Spell, M., et al. (2004). Measuring the quality of physician practice by using clinical vignettes: A prospective validation study. Annals of Internal Medicine, 141(10), 771–780.

    PubMed  Google Scholar 

  • Pirkis, J. E., Burgess, P. M., Kirk, P. K., Dodson, S., Coombs, T. J., & Williamson, M. K. (2005). A review of the psychometric properties of the health of the nation outcome scales (HoNOS) family of measures. Health and Quality of Life Outcomes, 3, 76.

  • Rey, J. M., Starling, J., Wever, C., Dossetor, D. R., & Plapp, J. M. (1995). Inter-rater reliability of global assessment of functioning in a clinical setting. Journal of Child Psychology and Psychiatry, 36(5), 787–792.

    Article  CAS  PubMed  Google Scholar 

  • Schorre, B. E., & Vandvik, I. H. (2004). Global assessment of psychosocial functioning in child and adolescent psychiatry. A review of three unidimensional scales (CGAS, GAF, GAPD). European Child & Adolescent Psychiatry, 13(5), 273–286.

    Article  Google Scholar 

  • Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., et al. (1983). A children’s global assessment scale (CGAS). Archives of General Psychiatry, 40(11), 1228–1231.

    CAS  PubMed  Google Scholar 

  • Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research, 7(3), 301–317.

    Article  CAS  PubMed  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

    Article  PubMed  CAS  Google Scholar 

  • Sitter, M., & Hagen, H. (2004). Opptrappingsplanens mål: Status psykisk helsevern for barn og unge. In V. Halsteinli (Ed.), Samdata psykisk helsevern. Sektorrapport 2003. Sammenligningsdata for psykisk helsevern. Sintef Helse.

  • Vatnaland, T., Vatnaland, J., Friis, S., & Opjordsmoen, S. (2007). Are GAF scores reliable in routine clinical use? Acta Psychiatrica Scandinavica, 115(4), 326–330.

    Article  CAS  PubMed  Google Scholar 

  • Weiss, B. (1998). Annotation: Routine monitoring of the effectiveness of child psychotherapy. Journal of Child Psychology and Psychiatry, 39(7), 943–950.

    Article  CAS  PubMed  Google Scholar 

  • Williams, R., & Kerfoot, M. (2005). Child and adolescent mental health services. Strategy, planning, delivery and evaluation. Oxford University Press.

  • Winters, N. C., Collett, B. R., & Myers, K. M. (2005). Ten-year review of rating scales, VII: Scales assessing functional impairment. Journal of the American Academy of Child and Adolescent Psychiatry, 44(4), 309–338.

    Article  PubMed  Google Scholar 

  • Wolpert, M., Thompson, M., & Tingay, K. (2005). Data collection, clinical audit, and measuring outcomes. In R. Williams & M. Kerfoot (Eds.), Child and adolescent mental health services. Strategy, planning, delivery and evaluation (pp. 519–533). Oxford University Press.

  • World Health Organization (1994). International statistical classification of diseases and related health problems 10th revision (ICD-10).

  • World Health Organization (1996). Multiaxial classification of child and adolescent psychiatric disorders. Cambridge University Press.

  • Yates, P., Garralda, M. E., & Higginson, I. (1999). Paddington complexity scale and health of the nation outcome scales for children and adolescents. British Journal of Psychiatry, 174, 417–423.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

The study was financially supported by The Research Council of Norway, and by Centre for Child and Adolescent Mental Health, Eastern and Southern Norway. The authors thank all the raters. Disclosure: The authors report no conflicts of interests.

Author information

Authors and Affiliations

  1. Centre for Child and Adolescent Mental Health, Eastern and Southern Norway, P.O. Box 4623, Nydalen, NO-0405, Oslo, Norway

    Ketil Hanssen-Bauer & Sonja Heyerdahl

  2. Department of Biostatistics, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway

    Odd O. Aalen

  3. Division of Mental Health Services, Akershus University Hospital, Lørenskog, Norway

    Torleif Ruud

  4. Faculty Division Akershus University Hospital, Faculty of Medicine, University of Oslo, Oslo, Norway

    Torleif Ruud

Authors
  1. Ketil Hanssen-Bauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Odd O. Aalen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Torleif Ruud
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sonja Heyerdahl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ketil Hanssen-Bauer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hanssen-Bauer, K., Aalen, O.O., Ruud, T. et al. Inter-rater Reliability of Clinician-rated Outcome Measures in Child and Adolescent Mental Health Services. Adm Policy Ment Health 34, 504–512 (2007). https://doi.org/10.1007/s10488-007-0134-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-007-0134-y

Keywords

Search

Navigation