Children's Global Assessment Scale (CGAS) in a naturalistic clinical setting: Inter-rater reliability and comparison with expert ratings

https://doi.org/10.1016/j.psychres.2010.02.006Get rights and content

Abstract

The Children's Global Assessment Scale (CGAS) is a tool to assess the overall level of functioning of children in Child and Adolescent Mental Health Services (CAMHS). Even though the use of this rating scale requires trained raters, it is commonly deployed without prior training in clinical settings. The aim of this study was to investigate the reliability and the agreement of CGAS ratings with an expert rating, in a clinical setting with untrained raters. Five experienced clinicians rated five vignettes to provide expert ratings. These vignettes were then rated by 703 health-care professionals representing 33 Swedish CAMHS. The health-care professionals rated the vignettes significantly higher (showing better global functioning) than the expert ratings. There was a wide range between the minimum and maximum ratings. The intraclass correlation coefficient was 0.73, which indicates moderate inter-rater reliability. Neither clinical experience nor earlier experience of using CGAS influenced the agreement with the expert ratings. The inter-rater reliability is moderate when CGAS is used in a clinical setting with untrained raters. Further, the untrained raters differed substantially from the experts. This stresses the importance of proper training in conjunction with the introduction of new rating scales.

Introduction

Child and Adolescent Mental Health Services (CAMHS) have increasingly recognised the importance of assessing patients’ overall level of functioning with reliable and valid instruments. To this end, many multi- and unidimensional rating scales have been developed (COMHWA, 2003, KITH, 2007, MH-SMART, 2009, MH-SMART, 2009). In the UK, for instance, the CAMHS Outcome Research Consortium (CORC) recommends a core evaluation set consisting of three questionnaires including the Children's Global Assessment Scale (CGAS), the Strengths and Difficulties Questionnaire (SDQ) and the Commission for Health Improvement-Experience of Service Questionnaire (CHI-ESQ) (Wolpert et al., 2007).

Even though the rationale for the use of these instruments is based on the need for objective measures in the planning and evaluation of individual treatment regimens, they are also valuable for describing clinical and epidemiological populations (Bird et al., 1996, COMHWA, 2003, Shaffer et al., 1999, Winters et al., 2005). These tools might also be used by health-care providers to prioritise health-care resources and to compare different clinical units in quality assurance programmes (KITH, 2007, MH-SMART, 2009, MHNOCC, 2004, Wolpert et al., 2007).

Many clinics routinely use unidimensional rating scales of global functioning, with which the clinician assesses the patient's overall level of functioning during a particular time period, usually on a continuum on a 100-point scale (American Psychiatric Association, 1994, KITH, 2007, Wolpert et al., 2007). These instruments have been designed for easy use and they normally take only a few minutes to complete, if the clinician has adequate information available. One such unidimensional tool is the CGAS, which is derived from the Global Assessment Scale (GAS) and adapted for use in children and adolescents (Endicott et al., 1976, Shaffer et al., 1983). Another similar tool that is commonly used in adult psychiatry and in some child and adolescent psychiatric settings is the Global Assessment of Functioning (GAF), which is a hybrid of GAS and CGAS that was developed in conjunction with the Diagnostic and Statistical manual of Mental Disorders (DSM) III. The CGAS is currently well established and used in nationwide clinical settings. In 2008, the Stockholm County Council in Sweden launched a remunerative incentive scheme to increase the use of the CGAS. There were no quality requirements in terms of reliability measures, or formal training, coupled to the reimbursement programme.

Despite the widespread use of CGAS, it has been evaluated mainly in settings involving a relatively small number of homogenous raters (Bird et al., 1996, Bird et al., 1987, Green et al., 1994, Rey et al., 1995, Shaffer et al., 1983, Steinhausen, 1987, Weissman et al., 1990). The results from these studies are therefore not generalisable to the current large-scale clinical use of CGAS, which involves raters of different professions with different degree of experience and training – if any – in the use of CGAS. Hence, the reliability and validity of CGAS is unknown in the setting in which it is most commonly used.

Moreover, it is not obvious according to which standard the validity for CGAS should be assessed. In the studies cited above, the ‘true’ ratings are defined post hoc as the mean rating of the group. Using this definition, CGAS correlates fairly well with the Child Behaviour Check List (CBCL) in epidemiological and clinical settings (Bird et al., 1987, Green et al., 1994); significantly lower CGAS scores have been shown in a group of children referred to mental health services compared with a group of children that was not referred, as well as in a group of ‘cases’ compared with ‘non-cases’ (Bird et al., 1987). An alternative approach to assess validity is to compare the individual CGAS rating to a consensus rating defined by experienced clinicians. The latter method has been employed in a Norwegian study involving a total of 15 raters from five countries, where each national group made a consensus rating (Hanssen-Bauer et al., 2007a).

The aim of the present study was to investigate the inter-rater reliability of CGAS ratings in a large-scale naturalistic clinical setting, and to evaluate the agreement with expert ratings. To this end, we recruited 703 health-care workers who rated five case vignettes each. Their ratings were compared with the expert rating, which was defined by five clinicians with extensive training and experience in the use of psychiatric rating scales. The use of case vignettes instead of real-life patients offers the advantage of providing all raters with the same information, which makes them useful for assessing differences in clinical judgement (Peabody et al., 2000). To emulate the common practice when rating scales are introduced top-down on demand by health administrators, the raters had no or limited experience in the use of CGAS.

Section snippets

Methods

CGAS is a clinician-rated tool to assess the overall functioning of the child, taking into account all available information. The scoring ranges from 1 (the most impaired level) to 100 (superior level of functioning). The scale is separated into 10-point sections that are headed with a description of the level of functioning and followed by examples matching the interval.

CGAS was translated into Swedish in 2001 by Per Gustafsson and Madeleine Helgesson. The translation was revised and

Characteristics of the participants

In total, 703 raters, most of them working at outpatient units in Stockholm, participated in the study (Table 1). The raters were psychologists, social workers, medical doctors and other staff members (nurses, psychiatric technicians, special education teachers, occupational therapists, etc.). The vast majority had no experience using CGAS, but a majority had used GAF.

Comparisons with expert ratings

Table 2 shows that the untrained raters’ median and mean scoring were significantly higher than the expert rating for all five

Discussion

This is the first naturalistic study of the inter-rater reliability of CGAS in a large-scale clinical setting, involving more than 700 raters with different professional backgrounds. To emulate naturalistic conditions, the participants received no training prior to the ratings. The main finding was that the inter-rater reliability is moderate when CGAS is used in a naturalistic setting by professionals without prior training in the use of the instrument.

Acknowledgements

This study was made possible thanks to support from Child and Adolescent Mental Health Services, Stockholm County Council, Sweden, and through grants from the Capio Research Foundation in Sweden, and from the Swedish Medical Research Council (K2008-62x-14647-06-3). We wish to thank Prudence W. Fisher, Ph.D., Yanling Huo, M.S., David Shaffer, F.R.C.P. (Lond), F.R.C. (Lond) and J. Blake Turner, Ph.D. at the Division of Child and Adolescent Psychiatry, New York State Psychiatric Institute/Columbia

References (29)

  • B. Green et al.

    The Children's Global Assessment Scale in clinical practice: an empirical evaluation

    Journal of the American Academy of Child and Adolescent Psychiatry

    (1994)
  • H.C. Steinhausen

    Global assessment of child psychopathology

    Journal of the American Academy of Child and Adolescent Psychiatry

    (1987)
  • M.M. Weissman et al.

    Applying impairment criteria to children's psychiatric diagnosis

    Journal of the American Academy of Child and Adolescent Psychiatry

    (1990)
  • Diagnostic and statistical manual of mental disorders : DSM-IV

    (1994)
  • L.W. Bates et al.

    Effects of brief training on application of the Global Assessment of Functioning Scale

    Psychological Reports

    (2002)
  • H.R. Bird et al.

    Global measures of impairment for epidemiologic and clinical use with children and adolescents

    International Journal of Methods in Psychiatric Research

    (1996)
  • H.R. Bird et al.

    Further Measures Of The Psychometric Properties Of The Childrens' Global Assessment Scale

    Archives of General Psychiatry

    (1987)
  • J.M. Bland et al.

    Measurement error and correlation coefficients

    British Medical Journal

    (1996)
  • COMHWA, "Consumer Outcomes in Mental Health Western Australia " 2003 Retrieved November 4, 2009, from...
  • J. Dyrborg et al.

    The Children's Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice–substance and reliability as judged by intraclass correlations

    European Child and Adolescent Psychiatry

    (2000)
  • J. Endicott et al.

    The global assessment scale. A procedure for measuring overall severity of psychiatric disturbance

    Archives of General Psychiatry

    (1976)
  • T. Fernando et al.

    The reliability of axis V of DSM-III

    American Journal of Psychiatry

    (1986)
  • H.H. Goldman et al.

    Revising axis V for DSM-IV: a review of measures of social functioning

    American Journal of Psychiatry

    (1992)
  • K. Hanssen-Bauer et al.

    Cross-national reliability of clinician-rated outcome measures in child and adolescent mental health services

    Administration and Policy in Mental Health

    (2007)
  • Cited by (71)

    • Treatment Gains Are Sustainable in Pediatric Obsessive-Compulsive Disorder: Three-Year Follow-Up From the NordLOTS

      2020, Journal of the American Academy of Child and Adolescent Psychiatry
      Citation Excerpt :

      The scale has been shown to have reasonable to very good psychometric properties.42 Studies have shown moderate reliability with intraclass correlation coefficients of 0.73 in clinical settings.43 Furthermore, discriminant and concurrent validities have been demonstrated.41

    View all citing articles on Scopus
    View full text