Why the factorial structure of the SCL-90-R is unstable: Comparing patient groups with different levels of psychological distress using Mokken Scale Analysis

Abstract

Since its introduction, there has been a debate about the validity of the factorial structure of the SCL-90-R. In this study we investigate whether the lack of agreement with respect to the dimensionality can be partly explained by important variables that might differ between samples such as level of psychological distress, the variance of the SCL-90-R scores and sex. Three samples were included: a sample of severely psychiatrically disturbed patients (n=3078), a sample of persons with Gender Incongruence (GI; n=410) and a sample of depressed patients (n=223). A unidimensional pattern of findings were found for the GI sample. For the severely disturbed and depressed sample, a multidimensional pattern was found. In the depressed sample sex differences were found in dimensionality: we found a unidimensional pattern for the females, and a multidimensional one for the males. Our analyses suggest that previously reported conflicting findings with regard to the dimensional structure of the SCL-90-R may be due to at least two factors: (a) level of self-reported distress, and (b) sex. Subscale scores should be used with care in patient groups with low self-reported level of distress.

Introduction

The Symptom Checklist-90-Revised (SCL-90-R) (Derogatis, 1994) was designed to cover nine different dimensions of psychological distress; the mean item score across all 90 items with theoretical values ranging from 0 through 4 is referred to as the Global Severity Index (GSI), which is widely used as a global index for psychological distress. Since the introduction of the SCL-90(-R), there has been a debate about the validity of the factorial structure, which was aptly expressed in the title of the paper ‘Factor structure of the SCL-90-R: is there one?’ (Cyr et al., 1985). More than two decades have passed since the publication of that paper; however, the debate has still not abated, as recent publications have demonstrated (Olsen et al., 2004, Arrindell et al., 2006, Elliott et al., 2006, Hafkenscheid et al., 2007). On one hand, there is a group of researchers that firmly believe in the multidimensionality of the instrument (Arrindell et al., 2004b, Arrindell et al., 2004a, Arrindell et al., 2006), whereas another group has pointed out that alternative models with only one or at most a few factors show an equally good or better fit (Hafkenscheid, 2004, Hafkenscheid et al., 2007). In a recent paper, Paap et al. (2011b) proposed a new scale solution of 7 scales based on a study involving patients referred for a personality disorder (PD); scales were built on two start items that reflected the content of the disorder that corresponded with the specific scale. The new solution included 60 of the 90 items clustered in seven scales: Depression, Agoraphobia, Physical Complaints, Obsessive-Compulsive, Hostility (unchanged), Distrust and Psychoticism. The authors found that most of the new scales discriminated reliably between patients with moderately low scores to moderately high scores. The items forming the GSI showed low scalability, and the authors concluded that their research findings lent support for a multidimensional model of the SCL-90-R. The authors speculated that the lack of agreement between studies might be due to several factors, such as difference in variance, the existence of structure generating factors, differences in the interpretation of the fit indices, and, finally, the chosen analytic strategy (Paap et al., 2011b).

In the current study, we investigate whether the findings in the study by Paap et al. can be generalized to other patient groups by comparing the dimensionality of the PD sample to that of a sample of persons with Gender Incongruence (GI) and a sample of depressed outpatients. The term ‘GI’ signifies the incongruence between one's gender identity on one hand, and one's assigned gender and/or one's congenital primary and secondary sex characteristics on the other hand (Kreukels et al., 2010, Meyer-Bahlburg, 2010).¹ Following Kreukels et al., we use GI when referring to patients who have not yet been diagnosed with GID (APA, 1994) or transsexualism (WHO, 1992). We expect the reported level of psychological distress (estimated by the GSI) to be lower in the GI sample than in the depressed sample and PD sample. Haraldsen and Dahl (2000) showed that patients diagnosed with GID had slightly elevated GSI scores when compared to healthy adults, but did not reach the value of 1.0 which is the cut-off for clinically significant symptoms (GSI_GID=0.6, GSI_controls=0.4). In contrast, depressed outpatients have been found to exceed the cut-off (GSI_DEP=1.4) (Leinonen and Niemi, 2007), and so have the patients in the PD sample used in the study by Paap et al. (GSI_PD=1.5). Our main research questions are:

• Is the dimensionality of the SCL-90-R similar for patient groups that differ in level of reported psychological distress?

• Are the different factorial solutions found in the literature due to a difference in variance in reported psychological distress?

Following Paap et al. (2011b) and Meijer et al. (2011), Mokken Scale Analysis (MSA; Mokken, 1971) was used to analyze the data. MSA is a nonparametric Item Response Theory (IRT) approach that can be used to explore and test hypotheses about the dimensionality of a data-set, while at the same time resulting in scales adhering to a measurement model.