The online version of the Dutch Penn State Worry Questionnaire: Factor structure, predictive validity and reliability

Abstract

The Penn State Worry Questionnaire (PSWQ) is a widely used self report scale to assess pathological worry. Several studies have shown that the paper-and-pencil version of the PSWQ shows satisfactory psychometric properties. Yet, it is unknown whether the PSWQ is suited to be administered on Internet. As the amount of assessments conducted online is vastly increasing, we examined the factor structure, predictive validity and reliability of the online version of the PSWQ in a large community sample (N = 1025). Confirmatory factor analyses demonstrated that the best fit to the data was provided by a one factor model wherein the reverse-worded items also loaded on a separate method factor. In a second part of the study a self-selected subsample of the total sample (n = 189) kept a log of the frequency and duration of worry episodes for six consecutive days. Worry frequency and worry duration were both predicted by the PSWQ, thereby lending support to its predictive validity. These findings provide preliminary evidence that the online version of the PSWQ possesses psychometric properties that are similar to the paper-and-pencil version.

Introduction

Worry is a frequently observed symptom in a wide range of psychopathologies (Watkins, 2008). Worry predicts anxious and depressive feelings (Hong, 2007) and chronic worry is the central feature of generalized anxiety disorder (GAD; American Psychiatric Association, 2000). Furthermore, by worrying, people prolong the physiological stress response which increases the risk of somatic health problems (Brosschot et al., 2006, Verkuil et al., 2010). Given this pivotal impact of worry on mental and somatic health, its assessment warrants careful consideration.

In the present study we examined the psychometric aspects of the Penn State Worry Questionnaire (Meyer, Miller, Metzger, & Borkovec, 1990) as administered via Internet. The PSWQ is a commonly used scale to measure the tendency to engage in pathological worry. It is used in treatment outcome studies (e.g. Borkovec, Newman, Pincus, & Lytle, 2002), studies on worry and potentially pathogenic phenomena such as negative mood and ‘stop rules’ (Startup & Davey, 2001), intolerance of uncertainty (Dugas, Gagnon, Ladouceur, & Freeston, 1998) and meta-worry (Wells & Carter, 2001) and studies on the effects of worry on somatic health (Verkuil et al., 2010).

The PSWQ consists of 16 items directed at the pathological aspects of worrying, for example its uncontrollability (“Once I start worrying, I can’t stop). Five items are negatively worded (e.g. “When there is nothing more I can do about a concern, I don’t worry about it anymore”). Research in the past two decades has shown that the PSWQ possesses good psychometric qualities (Startup & Erickson, 2006). Concerning its reliability, the internal consistency and test–retest reliability of the PSWQ are high (Meyer et al., 1990, Startup and Erickson, 2006). Concerning its validity, the PSWQ is associated with other measures of worry such as the Worry Domains Questionnaire and the Anxious Thoughts Inventory (cf. Startup & Erickson, 2006). Furthermore, the PSWQ is able to distinguish GAD from other mental problems (Brown, Antony, & Barlow, 1992). One crucial aspect pertaining to its predictive validity is that it predicts the frequency and duration of worry episodes in daily life (Verkuil, Brosschot, & Thayer, 2007). In 432 students we found that worry in daily life was predicted by the PSWQ.

Several studies have focused on the latent structure underlying the PSWQ. Initially, exploratory factor analyses yielded two factors, one factor containing the eleven positively worded items (‘Worry’) and another factor containing the five negatively worded items (‘Absence of Worry’; Beck et al., 1995, Fresco et al., 2002, van Rijsoort et al., 1999). Yet, it is commonly observed in scales compromised of positively and negatively worded items that a two factor model provides a better fit than a one factor solution. The conceptual and practical meaning of a factor consisting of negatively worded items is however questionable (Marsh, 1996). Such a two factor solution might be better modeled as an artifact of response tendencies associated with the two kinds of items. More recent studies using confirmatory factor analyses have indeed provided evidence that a one factor model that includes the five negatively worded items as a so called ‘method factor’ provides a better fit than the two factor solution (Brown, 2003, Hazlett-Stevens et al., 2004, van der Heiden et al., 2010).

The present study's focus is on the reliability, predictive validity and factor structure of the online version of the PSWQ, because little is known about the similarity of the paper-and-pencil version of this scale and its administration via Internet. In general, studies suggest that online versions of scales perform similarly to their offline versions (Buchanan & Smith, 1999). Yet, several studies showed that Internet administration can also lead to different scoring distributions (Buchanan, 2003, Davis, 1999), and to differences in how the items load on the expected factors (Buchanan et al., 2005) compared to the paper-and-pencil versions. These differences might be due to the relative anonymity associated with test administration on the Internet which might increase self-disclosure of mental problems. With respect to worry, only one study previously examined the reliability (but not the validity) of the online PSWQ (Zlomke, 2009). This study, performed in a large sample of students, warranted some caution considering its Internet administration. Both the coefficients for internal consistency (Cronbach's α = .73) and test–retest reliability (r = .67, p < .001) were considerably lower than previously found (Cronbach's α: .88–.95; test–retest rs = .74–.92; Startup & Erickson, 2006). More importantly, the mean score (M = 62.60, SD = 12.31) was substantially higher than those found in previous studies. In unselected groups mean PSWQ scores usually range between 38 and 48 (Startup & Erickson, 2006). A score of 62 has even been suggested to be a cutoff for screening for cases of GAD in a group of unselected students (Behar, Alcaine, Zuellig, & Borkovec, 2003). Apart from this surprising magnitude, it raises the question whether the norms and cutoff points obtained with the paper-and-pencil can be used when the PSWQ is administered online.

The PSWQ is widely used and systems for online psychological assessments are increasingly being developed (e.g. Routine Outcome Monitoring; de Beurs et al., 2011). Hence, the aim of this study was to examine the psychometric properties of this scale when administered via Internet, in a community sample. We focused on its underlying factor structure, as well as on its reliability and predictive validity. Testing these features will eventually provide evidence whether the PSWQ can justifiably be used in online studies.