Children's Global Assessment Scale (CGAS) in a naturalistic clinical setting: Inter-rater reliability and comparison with expert ratings
Introduction
Child and Adolescent Mental Health Services (CAMHS) have increasingly recognised the importance of assessing patients’ overall level of functioning with reliable and valid instruments. To this end, many multi- and unidimensional rating scales have been developed (COMHWA, 2003, KITH, 2007, MH-SMART, 2009, MH-SMART, 2009). In the UK, for instance, the CAMHS Outcome Research Consortium (CORC) recommends a core evaluation set consisting of three questionnaires including the Children's Global Assessment Scale (CGAS), the Strengths and Difficulties Questionnaire (SDQ) and the Commission for Health Improvement-Experience of Service Questionnaire (CHI-ESQ) (Wolpert et al., 2007).
Even though the rationale for the use of these instruments is based on the need for objective measures in the planning and evaluation of individual treatment regimens, they are also valuable for describing clinical and epidemiological populations (Bird et al., 1996, COMHWA, 2003, Shaffer et al., 1999, Winters et al., 2005). These tools might also be used by health-care providers to prioritise health-care resources and to compare different clinical units in quality assurance programmes (KITH, 2007, MH-SMART, 2009, MHNOCC, 2004, Wolpert et al., 2007).
Many clinics routinely use unidimensional rating scales of global functioning, with which the clinician assesses the patient's overall level of functioning during a particular time period, usually on a continuum on a 100-point scale (American Psychiatric Association, 1994, KITH, 2007, Wolpert et al., 2007). These instruments have been designed for easy use and they normally take only a few minutes to complete, if the clinician has adequate information available. One such unidimensional tool is the CGAS, which is derived from the Global Assessment Scale (GAS) and adapted for use in children and adolescents (Endicott et al., 1976, Shaffer et al., 1983). Another similar tool that is commonly used in adult psychiatry and in some child and adolescent psychiatric settings is the Global Assessment of Functioning (GAF), which is a hybrid of GAS and CGAS that was developed in conjunction with the Diagnostic and Statistical manual of Mental Disorders (DSM) III. The CGAS is currently well established and used in nationwide clinical settings. In 2008, the Stockholm County Council in Sweden launched a remunerative incentive scheme to increase the use of the CGAS. There were no quality requirements in terms of reliability measures, or formal training, coupled to the reimbursement programme.
Despite the widespread use of CGAS, it has been evaluated mainly in settings involving a relatively small number of homogenous raters (Bird et al., 1996, Bird et al., 1987, Green et al., 1994, Rey et al., 1995, Shaffer et al., 1983, Steinhausen, 1987, Weissman et al., 1990). The results from these studies are therefore not generalisable to the current large-scale clinical use of CGAS, which involves raters of different professions with different degree of experience and training – if any – in the use of CGAS. Hence, the reliability and validity of CGAS is unknown in the setting in which it is most commonly used.
Moreover, it is not obvious according to which standard the validity for CGAS should be assessed. In the studies cited above, the ‘true’ ratings are defined post hoc as the mean rating of the group. Using this definition, CGAS correlates fairly well with the Child Behaviour Check List (CBCL) in epidemiological and clinical settings (Bird et al., 1987, Green et al., 1994); significantly lower CGAS scores have been shown in a group of children referred to mental health services compared with a group of children that was not referred, as well as in a group of ‘cases’ compared with ‘non-cases’ (Bird et al., 1987). An alternative approach to assess validity is to compare the individual CGAS rating to a consensus rating defined by experienced clinicians. The latter method has been employed in a Norwegian study involving a total of 15 raters from five countries, where each national group made a consensus rating (Hanssen-Bauer et al., 2007a).
The aim of the present study was to investigate the inter-rater reliability of CGAS ratings in a large-scale naturalistic clinical setting, and to evaluate the agreement with expert ratings. To this end, we recruited 703 health-care workers who rated five case vignettes each. Their ratings were compared with the expert rating, which was defined by five clinicians with extensive training and experience in the use of psychiatric rating scales. The use of case vignettes instead of real-life patients offers the advantage of providing all raters with the same information, which makes them useful for assessing differences in clinical judgement (Peabody et al., 2000). To emulate the common practice when rating scales are introduced top-down on demand by health administrators, the raters had no or limited experience in the use of CGAS.
Section snippets
Methods
CGAS is a clinician-rated tool to assess the overall functioning of the child, taking into account all available information. The scoring ranges from 1 (the most impaired level) to 100 (superior level of functioning). The scale is separated into 10-point sections that are headed with a description of the level of functioning and followed by examples matching the interval.
CGAS was translated into Swedish in 2001 by Per Gustafsson and Madeleine Helgesson. The translation was revised and
Characteristics of the participants
In total, 703 raters, most of them working at outpatient units in Stockholm, participated in the study (Table 1). The raters were psychologists, social workers, medical doctors and other staff members (nurses, psychiatric technicians, special education teachers, occupational therapists, etc.). The vast majority had no experience using CGAS, but a majority had used GAF.
Comparisons with expert ratings
Table 2 shows that the untrained raters’ median and mean scoring were significantly higher than the expert rating for all five
Discussion
This is the first naturalistic study of the inter-rater reliability of CGAS in a large-scale clinical setting, involving more than 700 raters with different professional backgrounds. To emulate naturalistic conditions, the participants received no training prior to the ratings. The main finding was that the inter-rater reliability is moderate when CGAS is used in a naturalistic setting by professionals without prior training in the use of the instrument.
Acknowledgements
This study was made possible thanks to support from Child and Adolescent Mental Health Services, Stockholm County Council, Sweden, and through grants from the Capio Research Foundation in Sweden, and from the Swedish Medical Research Council (K2008-62x-14647-06-3). We wish to thank Prudence W. Fisher, Ph.D., Yanling Huo, M.S., David Shaffer, F.R.C.P. (Lond), F.R.C. (Lond) and J. Blake Turner, Ph.D. at the Division of Child and Adolescent Psychiatry, New York State Psychiatric Institute/Columbia
References (29)
- et al.
The Children's Global Assessment Scale in clinical practice: an empirical evaluation
Journal of the American Academy of Child and Adolescent Psychiatry
(1994) Global assessment of child psychopathology
Journal of the American Academy of Child and Adolescent Psychiatry
(1987)- et al.
Applying impairment criteria to children's psychiatric diagnosis
Journal of the American Academy of Child and Adolescent Psychiatry
(1990) Diagnostic and statistical manual of mental disorders : DSM-IV
(1994)- et al.
Effects of brief training on application of the Global Assessment of Functioning Scale
Psychological Reports
(2002) - et al.
Global measures of impairment for epidemiologic and clinical use with children and adolescents
International Journal of Methods in Psychiatric Research
(1996) - et al.
Further Measures Of The Psychometric Properties Of The Childrens' Global Assessment Scale
Archives of General Psychiatry
(1987) - et al.
Measurement error and correlation coefficients
British Medical Journal
(1996) - COMHWA, "Consumer Outcomes in Mental Health Western Australia " 2003 Retrieved November 4, 2009, from...
- et al.
The Children's Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice–substance and reliability as judged by intraclass correlations
European Child and Adolescent Psychiatry
(2000)
The global assessment scale. A procedure for measuring overall severity of psychiatric disturbance
Archives of General Psychiatry
The reliability of axis V of DSM-III
American Journal of Psychiatry
Revising axis V for DSM-IV: a review of measures of social functioning
American Journal of Psychiatry
Cross-national reliability of clinician-rated outcome measures in child and adolescent mental health services
Administration and Policy in Mental Health
Cited by (71)
Comparison of the Neurocognitive Profile of the Children of Parents with Bipolar Disorder and Controls: a Transnational Cross-Sectional Study
2023, Revista Colombiana de PsiquiatriaIntolerance of Uncertainty–Focused Treatment for Adolescents With Excessive Worry: A Pilot Feasibility Study
2020, Cognitive and Behavioral PracticeTreatment Gains Are Sustainable in Pediatric Obsessive-Compulsive Disorder: Three-Year Follow-Up From the NordLOTS
2020, Journal of the American Academy of Child and Adolescent PsychiatryCitation Excerpt :The scale has been shown to have reasonable to very good psychometric properties.42 Studies have shown moderate reliability with intraclass correlation coefficients of 0.73 in clinical settings.43 Furthermore, discriminant and concurrent validities have been demonstrated.41
Evaluation of a psychoeducation group for children presenting with functional tic-like behaviours
2024, Clinical Child Psychology and Psychiatry