Improving social media measurement in surveys: Avoiding acquiescence bias in Facebook research

Highlights

• Agree–disagree questions about social media use are prone to acquiescence bias.

• Acquiescence inflates reliabilities and factor loadings, and alters correlations.

• Both better design and statistical tools can correct for these biases.

• There are benefits and limits for each corrective approach.

• Research would be improved by avoiding agree–disagree questions.

Abstract

Social media measurement relies heavily on self-report survey research. Hence, known biases in how individuals answer survey questions can introduce systematic errors into the social media literature. In particular, many common social media measures are prone to acquiescence response bias, an error that occurs due to individuals' tendency to agree with agree–disagree questions. The current study tests a series of techniques to both detect and overcome acquiescence bias in the context of Facebook measurement. Controlling for individuals' tendency to agree with agree–disagree questions, we find evidence that acquiescence has inflated the reliabilities and factor loadings of many Facebook use scales, and has altered correlations both among Facebook use measures and between those measures and related covariates. Further, when the individual-level tendency to agree with questions is controlled, Facebook measures demonstrate greater criterion validity in their relations to items that do not use agree–disagree scales. Having identified the presence of acquiescent responding, we test three methods for mitigating this response bias: the use of balanced scales, item-specific questions, and statistical correctives. All three methods appear to reduce the bias introduced by acquiescence. Thus, the results provide comparative evidence on strategies to alleviate the consistent impact of an important method bias in social media measurement and thereby contribute to improving the validity of social media research at large.

Introduction

Acquiescence bias is a pervasive problem in survey research that could translate to social media measurement as well. When questions are presented with agree–disagree (AD) or yes/no response options, some respondents select the “agree” (or “yes”) option disproportionately more frequently than the “disagree” (or “no”) option. Presumably, this tendency stems from the social norm to be agreeable (see Pasek & Krosnick, 2010). Acquiescence can introduce errors into data, as survey responses to acquiescence-prone measures conflate individuals' true attitudes and behaviors with agreeableness. At its most pernicious, this can lead survey researchers astray, inducing correlations between similarly worded items that may be designed to tap unrelated constructs and hence resulting in systematic errors (Bagozzi, 1984, MacKenzie and Podsakoff, 2012). These concerns have led some researchers to decry the use of AD Likert scales for social research (Fowler, 1995, Saris et al., 2010). Yet, despite these calls, social media measurement through surveys, and especially the literature on Facebook, has extensively relied on survey techniques that are prone to acquiescence bias.

Domains where AD scales are widely used might be particularly susceptible to inferential errors related to acquiescence; and research on the social network site Facebook appears to be one such area. With the explosive emergence of social media and the subsequent proliferation of scholarship on Facebook from diverse fields (e.g. Wilson, Gosling, & Graham, 2012), a number of strategies have been introduced for gauging users' behaviors on the site (Kuru & Pasek, in press). The vast majority of studies of Facebook use, however, have been dominated by acquiescence-prone measures. Further, these measures tend to conflate agreement with greater use of the s/ocial network site. Acquiescence bias could therefore be an important confound in studies of Facebook, potentially hindering researchers' attempts at understanding and situating the social experience and consequences of site use. Since the Facebook literature is yet a young and emerging field, researchers may still have a chance to adjust course, tracking use of the site in ways that are less susceptible to bias.

The current study assesses the extent to which acquiescence response bias may be influencing current studies of the correlates of Facebook use. To accomplish this, we look for the presence of method bias using an online survey experiment. Where methodological biases are observed, we test whether a variety of statistical correctives, balanced scales, and finally alternative question wording might mitigate these biases (cf. Saris et al., 2010). Structural equation modeling (SEM) and alternative question wording approaches are compared as potential ways to improve indexes of Facebook use as well as in their ability to predict a variety of theoretically related constructs.