More reasons to be straightforward: Findings and norms for two scales relevant to social anxiety

https://doi.org/10.1016/j.janxdis.2011.02.002Get rights and content

Abstract

The validity of both the Social Interaction Anxiety Scale and Brief Fear of Negative Evaluation scale has been well-supported, yet the scales have a small number of reverse-scored items that may detract from the validity of their total scores. The current study investigates two characteristics of participants that may be associated with compromised validity of these items: higher age and lower levels of education. In community and clinical samples, the validity of each scale's reverse-scored items was moderated by age, years of education, or both. The straightforward items did not show this pattern. To encourage the use of the straightforward items of these scales, we provide normative data from the same samples as well as two large student samples. We contend that although response bias can be a substantial problem, the reverse-scored questions of these scales do not solve that problem and instead decrease overall validity.

Research highlights

► Two commonly used scales related to social anxiety have poorly performing reverse-scored questions. ► We provide further evidence of this finding by focusing on age and years of education. ► We provide stratified norms for both scales to encourage use of straightforward totals.

Introduction

Clinicians and researchers who assess social anxiety have recently been presented with evidence that two commonly used measures have a significant problem. The Brief Fear of Negative Evaluation scale (BFNE; Leary, 1983) and the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) both have a small number of reverse-scored items that have demonstrated problematic reliability and validity across multiple studies (Carleton et al., 2009, Duke et al., 2006, Rodebaugh et al., 2004, Rodebaugh et al., 2006, Rodebaugh et al., 2007, Weeks et al., 2005, Woods and Rodebaugh, 2005). All of these studies otherwise support the factorial, convergent, and discriminant validity of the straightforwardly worded items that make up the majority of each scale. None of the studies suggested discarding the measures, but the authors suggest either not using the reverse-scored items or replacing them with new straightforwardly worded items.

Our impression has been that few authors have heeded these suggestions. To test this impression, we conducted a search in August 2010 for articles published in 2009 and 2010 referring to the BFNE or SIAS. We retrieved all the articles (N = 27) that reported on use of the English-language version of the instruments in peer-reviewed, English-language journals that were available to us electronically. The majority of the articles (n = 25, 92.5%) used the original total(s) of the BFNE and/or SIAS, making this the modal practice in even the most recent literature. Below we consider three possible reasons for this practice; we have designed the current study to address the latter two of these. In contrast, we believe the first reason can be dismissed on the basis of available information.

Several researchers have suggested to us that reverse-scored questions help avoid response bias or careless responding. Training with well-known measures that include items to detect such problems, such as the Minnesota Multiphasic Personality Inventory-2 (MMPI-2; Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989) may have led to the general opinion that all measures should include similar items. Such items have indeed been used with some success to detect such problems in MMPI and MMPI-2 responses (Baer et al., 1992, Berry et al., 1992, Blanchard et al., 2003). The BFNE and SIAS reverse-scored items, however, are not plausible as validity indices. Validity indices with psychometric support generally rely upon participants finding it difficult to discern the purpose of the items. For example, the MMPI-2 has many validity scales, the items of which are interspersed throughout hundreds of items such that their validity-related purpose is difficult to discern (see, e.g., Greene, 1991). To the best of our knowledge, all supported validity indices involve at least ten items for which most participants will have difficulty discerning the intended purpose. In contrast, the number of items in question on the SIAS and BFNE is small (3 on the SIAS, 4 on the BFNE) on fairly short scales (20 items for the SIAS, 12 for the BFNE). Further, although the reverse-scored items may be confusing for some participants, it is hard to argue that their difference from the straightforwardly worded items is subtle. We find it unlikely that any researcher would consider the reverse-scored items on either scale as plausible candidates for validity indices in the vein of those derived from the MMPI-2, and we have not seen any argument or evidence suggesting such plausibility.

There are other reasons, however, to include reverse-scored items in a total. Some researchers appear to believe that the mere presence of reverse-scored items protects against response biases. More formal arguments have been made (e.g., Ray, 1983) that only completely balanced scales (i.e., with an equal number of straightforwardly worded and reverse-scored questions) can avoid certain response biases, because only such scales effectively counterbalance to avoid these biases. For example, in such scales, half of the items would be affected by biases toward acquiescence (tending to endorse items too easily) such that their total is too high, but the other half would be affected by the same bias such that their total was too low. One can therefore see the appeal of completely balanced scales, which have the promise of cancelling out many types of response bias over the entire scale.

The argument for balanced scales faces two problems in the SIAS and BFNE. First, neither the BFNE nor SIAS is completely balanced. At the risk of redundancy, we emphasize that the argument for completely balanced scales literally requires complete balance for the logic to apply. The second issue is that even if a scale were completely balanced, its special virtues depend upon both straightforwardly worded and reverse-scored items being at least roughly equivalently valid. That is, in the argument for completely balanced scales, all items are equally but oppositely affected by response biases; if that were not the case, balancing the scale would not have the effect of cancelling out validity problems over the length of the scale. Thus, arguments such as Ray's (1983) are challenged by findings that reverse-scored items may suffer disproportionately from validity problems.

Focusing on the BFNE and SIAS in particular, in the study conducted by Weeks et al. (2005), the straightforwardly worded items of the BFNE did not correlate with years of education in a group of clients with social anxiety disorder, whereas the reverse-scored items did. The most plausible explanation for this effect seems to be that less educated participants tended to provide less valid information on reverse-scored questions, possibly due to confusion over double-negatives. The same participants, however, had no apparent trouble responding to straightforward items. If years of education were simply acting as a proxy for acquiescence, then it should correlate with a higher straightforward total as well (but it did not). In a similar, but distinct finding, Rodebaugh et al. (2007) found that the reverse-scored items of the SIAS had a high correlation with extraversion, whereas the straightforwardly worded items did not display this problem. Such findings suggest that the argument for completely balanced scales is limited by the fact that, although all items may be affected by factors such as acquiescence and negativity, reverse-scored items may further suffer from additional challenges to validity (as also suggested by studies of other scales; e.g., Brown, 2003, Hazlett-Stevens et al., 2004).

The above arguments dispute the notion that the reverse-scored items have a special capacity to improve in the validity of the SIAS and BFNE. For at least the SIAS, it has been demonstrated that removing the reverse-scored item scores from the total generally improved overall convergent validity across a wide variety of measures across multiple samples (Rodebaugh et al., 2007). The available evidence might still be unconvincing because, although the available studies were conducted by different authors across many different samples, there remain only a few studies for each scale.

Most of the available evidence concerns problems with the reverse-scored items due to divergent factor loadings or lower internal consistency rather than demonstrations of specific causes of such problems. In general, our hypothesis is that any factor likely to increase participants’ vulnerability to confusion and error should adversely affect the reverse-scored items but not the straightforward items on the scales. The demonstration by Weeks et al. (2005) represents a single test of the hypothesis that reverse-scored questions (but not the straightforwardly worded items) display unwanted correlations with factors that might increase vulnerability to confusion. Additional demonstrations along the same lines are needed.

We hypothesized that age would represent a challenge to the reverse-scored items’ validity in the same vein as education. Increasing age is correlated with decreases in cognitive abilities such as explicit memory, reasoning, and processing speed (Salthouse, 1996, Salthouse, 2000, Schaie and Willis, 1993). Questions that require responding in the opposite direction vs. most of the items may be particularly problematic for older adults with such problems. With the number of older adults increasing in the USA and the high occurrence of anxiety disorders in adults over the age of 55 (Byers, Yaffe, Covinsky, Friedman, & Bruce, 2010), it is imperative to increase the field's understanding of the assessment of anxiety within older populations, particularly with regard to social anxiety disorder. Although there is a dearth of research on social anxiety disorder and older adults, the National Comorbidity Survey Replication showed that the 12-month prevalence rates for older adults with any anxiety disorder remained high (12%), and among these older adults social anxiety disorder was the second most prevalent anxiety disorder (3.5%). Moreover, even subthreshold mental health symptomatology can have negative effects on daily functioning and physical performance (Grabovich, Lu, Tang, Tu, & Lyness, 2010). Data on the utility of social anxiety measures in older populations is therefore sorely needed.

If researchers are relatively convinced by previous studies, there remains a final reason to continue to use the original totals, particularly for the SIAS, which is often used to describe samples in terms of their levels of social anxiety symptoms. The norms, means, and much of the other information in the literature concerning these scales have been presented based on the overall total score. More specifically, to the best of our knowledge there are no community norms available in the literature thus far for the straightforward totals of the BFNE or SIAS. Presentation of other norms is rare; Carleton, Collimore, and Asmundson (2007) provide some information on the straightforward items of the original BFNE for an undergraduate sample, and Rodebaugh et al. (2006) provide some information on the straightforward items of the SIAS for client and student samples. Most available norms are not stratified at all; none are stratified by age or ethnicity. In our informal review of the recent literature that employs the SIAS and BFNE, we noticed that some researchers who have recently used the original totals of the measures have done so only to use cut-offs for screening or to present means to characterize their groups. Presentation of more detailed norms would therefore reduce the need to use the original totals of the measures.

The current study was designed to address several of the issues raised above. First, we addressed whether reverse-scored questions demonstrate particular problems by testing the extent to which age and years of education explain lack of consistency between straightforward and reverse-scored questions. Given our hypothesis that the reverse-scored questions would demonstrate particular problems, we also planned to present norms for the straightforward BFNE and SIAS for community, client, and student samples (stratified by age, gender, and ethnicity, as appropriate).

Section snippets

Participants and procedure

All participants provided informed consent to the original study for which data were collected; all analyses for the current study were conducted on deidentified archival data. Participants included community, client, and student samples. In evaluating these samples, it is useful to have background information regarding recruitment sites. Washington University is a small, private, Midwestern metropolitan university located nearby downtown Saint Louis, Missouri. Temple University is a large,

Distributional properties

The skewness and kurtosis were examined along with histograms of the distributions of each subscale to be examined, in each sample. An overall pattern emerged that, for the community sample, the straightforwardly worded totals of both scales showed slight departures from normality (BFNE: skewness = 1.06, kurtosis = 0.64; SIAS: skewness = 1.28, kurtosis = 1.70) whereas the reverse-scored totals did not show this tendency (BFNE: skewness = −0.28, kurtosis = 0.04; skewness = −0.03, kurtosis = −0.54). Visual

Discussion

The current study provides further evidence that the reverse-scored items of the SIAS and BFNE have challenges to their validity, as has been found previously (e.g., Rodebaugh et al., 2004, Rodebaugh et al., 2006, Rodebaugh et al., 2007, Weeks et al., 2005). In the current study, we demonstrated that the reverse-scored items have problematic relationships with both age and level of education. Problems related to age were of most concern with people who were above the age of 60, whereas problems

Acknowledgments

This project was supported, in part, by the following Grants: 1 R01 MH064481-01A1, R01 MH064726, T32 AG00030, and T32 MH20004.

References (33)

  • L.S. Aiken et al.

    Multiple regression: testing and interpreting interactions

    (1991)
  • American Psychiatric Association

    Diagnostic and statistical manual of mental disorders

    (1994)
  • D.T.R. Berry et al.

    MMPI-2 random responding indices: validation using a self-report methodology

    Psychological Assessment

    (1992)
  • D.D. Blanchard et al.

    A comparison of the PAI and MMPI-2 as predictors of faking bad in college students

    Journal of Personality Assessment

    (2003)
  • E.J. Brown et al.

    Validation of the Social Interaction Anxiety Scale and the Social Phobia Scale across the anxiety disorders

    Psychological Assessment

    (1997)
  • J.N. Butcher et al.

    MMPI-2: manual for administration and scoring

    (1989)
  • Cited by (148)

    • The role of fear of evaluation in group perception

      2023, Journal of Anxiety Disorders
    View all citing articles on Scopus
    1

    Tel.: +1 215 204 1575.

    2

    Tel.: +1 212 543 5367.

    View full text