Elsevier

Journal of Environmental Psychology

Volume 40, December 2014, Pages 359-371
Journal of Environmental Psychology

Review
The validity of self-report measures of proenvironmental behavior: A meta-analytic review

https://doi.org/10.1016/j.jenvp.2014.09.003Get rights and content

Highlights

  • Large effect size (r = .46) observed between self-reported and objective behavior.

  • 79% of variance in self-report measures unexplained by objective measures.

  • Significant heterogeneity detected, suggesting self-report validity varies widely.

  • Taken together, observed effect size is conventionally large but functionally small.

Abstract

Do self-reports match objective behavior? We performed a meta-analysis to quantify the association between self-reported and objective measures of proenvironmental behavior, and to evaluate the moderating influence of two socio-demographic and seven methodological moderators. Data from 6260 individuals or households, involving 19 measures of association in 15 studies, revealed a positive and nominally large (Cohen, 1988) effect size (r = .46). However, this means that 79% of the variance in the association between self-reported and objective behavior remains unexplained, which is especially troubling given the environmental context. We conclude that although this effect size is conventionally large, it is functionally small for testing theory and devising intervention campaigns, possibly leading researchers to draw misleading conclusions about the usefulness of theories that employ self-reports to predict objective behavior. These findings highlight a crucial need for research that strengthens the validity of self-reports for well-defined types of environmental behavior.

Introduction

As environmental concern mounts, interest in understanding the psychological determinants of proenvironmental behavior has increased. Environmental psychologists and others in related disciplines have endeavored to identify the most influential predictors of proenvironmental behavior and to develop models that represent the role of intrapersonal (e.g., attitudes, beliefs, and values), interpersonal (e.g., social norms), and external factors (e.g., rewards and punishments) (e.g., Gifford, 2006). Self-reports often are employed as measures of proenvironmental behavior (see Steg & Vlek, 2009 for review). They usually are obtained through surveys, questionnaires, or interviews, and they assess participants' subjective estimates of their own behavior. Of course, the use of self-reports rests on the assumption that they accurately reflect individuals' actual behavior – an assumption that has received mixed empirical support (e.g., ∗Hamilton, 1985, ∗Warriner et al., 1984). Some studies suggest that self-reports are adequate predictors of actual behavior (e.g., Corral-Verdugo & Figueredo, 1999), whereas others suggest the contrary (e.g., Fuj, Hennessy, & Mak, 1985).

Thus, construct validity is a key concern with the use of self-reports, given that researchers strive to employ the best measures of proenvironmental behavior. Construct validity refers to the degree to which a measure (e.g., self-reported water consumption) accurately assesses the intended construct (e.g., actual water consumption). The establishment of sound construct validity is an essential requirement for any rigorous line of research – in fact, without it the chosen variable simply cannot be said to represent the construct that it purports to measure.

The various appeals of self-report measures (primarily their ease of use) make it unlikely that their usage in the literature will decrease anytime soon, and therefore evaluating and understanding the nature and degree of error that they introduce is crucial. For this reason, this meta-analysis assesses the association between self-reported and objective measures of proenvironmental behavior in studies that have employed both types of measures to evaluate the same behavior. Additionally, it investigates several socio-demographic and methodological characteristics that may magnify or reduce the error in self-report measures.

Self-reports are the preferred method of data collection for the majority of researchers, owing to their low cost, relative ease of use, and flexibility. Indeed, simply asking participants to report, for example, how often they engage in a particular environmentally relevant behavior along a scale from ‘Never’ to ‘Always’ (e.g., De Young, 1990, Margai, 1997) is an easy way to obtain information about that behavior. Self-reports also allow researchers to investigate behaviors that may not otherwise be observable (see Tarrant & Cordell, 1997). Given these advantages, researchers often use self-report measures to ask participants to express the frequency with which they engage in a list of proenvironmental behaviors (e.g., Gatersleben et al., 2002, ∗Kaiser et al., 2001).

The broader survey methodology literature suggests that self-reports are only weakly associated with actual behavior (e.g., Peterson & Kerin, 1981). Inaccuracies may stem from a variety of sources. For example, self-report measures may be prone to exaggeration. Some evidence suggests that individuals tend to over-report their proenvironmental behavior (Barr, 2007, ∗Fuj et al., 1985, Geller, 1981, ∗Warriner et al., 1984), and social desirability bias has been suggested as a cause for this over-reporting and thus an important limitation of self-report measures of proenvironmental behavior as compared to objective measures (Stern & Oskamp, 1987; Tarrant & Cordell, 1997). Socially desirable responding is defined as “the tendency of subjects to attribute to themselves in self-description, personality statements with socially desirable scale values, and to reject those with socially undesirable scale values” (Edwards, 1957, p. vi). Recent research, however, has provided explicit evidence that social desirability appears to have a minimal and non-significant effect on self-reported proenvironmental behavior and no moderating effect on the relationship between self-reported environmental attitudes and proenvironmental behavior (Milfont, 2009).

Another disadvantage of self-report measures is that they are subjective by nature; descriptors such as ‘Often,’ may mean different things to different participants. In addition, survey instruments are best designed to assess attitudes and beliefs, and therefore even when respondents are explicitly asked to report their behavior, attitudes often seep into their responses. Some have also suggested that self-reports may largely reflect individuals' perceptions of their behavior (Olson, 1981), behavioral intentions (Lee, 1993), or other – sometimes false – beliefs and attitudes (Rathje, 1989), rather than objective behavior. Finally, limited memory or knowledge may also reduce the accuracy of self-reports (e.g., see Warriner, McDougal, & Claxton, 1984).

Given that the advantages of self-report measures make them common research tools in these other areas as well, these other disciplines must also grapple with similar issues pertaining to their use. As such, the validity of self-report data is a question which has received considerable research attention in a variety of other disciplines, including public opinion polling, consumer research, and – perhaps most notably – health research. This broader consideration of self-report validity warrants some discussion here.

In studying health-related behaviors, for example, self-reports are frequently used to privately assess what may be sensitive or personal issues. For example, health researchers have explored the validity of self-reports for assessing a variety of behaviors, including sexual behavior (e.g., Brener, Billy, & Grady, 2003). Research examining health-risk behaviors (e.g., unprotected sex and drug use) in adolescents often relies, by necessity, on self-report questionnaires. However, Brener et al. (2003) asserted that both the accuracy and truthfulness of such data might be compromised when these behaviors are measured through self-report, given that situational factors, such as a lack of privacy when responding to the questionnaire, may cause respondents to provide inaccurate information, or that respondents may inaccurately recall information, or may over- or under-report behaviors due to social desirability or the desire to avoid providing sensitive information. As an attempt to improve self-report accuracy, Denny et al. (2008) conducted a national survey of health and wellbeing among youth in which adolescents were provided with individual computer tablets. Students who preferred the tablet over a standard lab computer reported feeling more privacy and finding it easier to answer questions truthfully, thus potentially providing more valid self-report data (Denny et al., 2008).

For other health-related behaviors, such as physical activity, Affuso et al. (2011) found large discrepancies between self-reported physical activity levels and objectively measured activity via accelerometry. Thus, Affuso et al. concluded that self-reports are not a valid measure of exercise levels given individuals' tendency to under-report sedentary behaviors and that objective measures should be used instead. In the case of smoking behavior, however, Studts et al. (2006) demonstrated that self-reported smoking habits served as a valid measurement of actual smoking behavior, as evaluated via objective testing of urine samples, a finding which may surprise some given that self-reporting is often questioned due to social desirability and medical disapproval, which is thought to result in under-reported smoking behaviors. Thus, health researchers also struggle with issues related to whether subjective measures accurately capture their targets behaviors.

The validity of self-report measures is also of concern in the discipline of marketing and business research, where the accuracy of such measures can provide valuable information on consumers' brand preference, purchasing behavior, and product usage. Nencyz-Thiel, Beal, Ludwichowska, and Romaniuk (2013), for example, examined the accuracy of self-reported television viewing as compared to objectively measured minute-per-minute data on actual programs watched. Their research concluded that discrepancies exist between the objective measures and individuals' reporting of the amount of television viewing as well as the specific programs viewed.

Therefore, researchers across a variety of disciplines outside of environmental psychology must address similar issues pertaining to self-report validity and, as such, have undertaken efforts to validate their self-report measures and explored means to minimize the discrepancy between self-reports and objective measures of behavior. Within environmental psychology research outside of the domain of proenvironmental behavior, researchers have found that judgments of environmental features do not always correspond accurately with objective measures of the same features (McCormack, Cerin, Leslie, du Toit, & Owen, 2008). Those researchers suggest that future research should include measures of perceived and objective environmental attributes.

Therefore, similar to the fields of health and marketing, environmental psychology researchers rely heavily on self-reporting, which can be validated in different ways. Several types of unobtrusive measures have been used to compare self-reports with more objective measures and these can be broadly classified into three categories: device measurements, observations made by trained observers, and peer ratings.

Energy use and water consumption are two behavioral domains that lend themselves well to the evaluation of the accuracy of self-reports through device measurements – specifically, via meter readings. Many studies conducted in the 1980s used this method of data collection. For example, self-reported household energy consumption (for electricity, oil, and gas) was strongly correlated with utility company data (Warriner et al., 1984). However, Fuj et al. (1985) observed that participants tended to over-report their electricity conservation efforts, when compared to meter readings. Furthermore, the correlation between stated change to household electricity consumption during the last year and the observed change in electricity usage was quite low (r = .06). Consistent with this, self-reported water consumption has been shown to be only weakly correlated with the change in actual water consumption, χ²(6) = 1.9, p = .01 (Hamilton, 1985). Other such measurements may be obtained by a variety of devices, including GPS readings of transportation behavior (Bolderijk, Knockaert, Steg, & Verhoef, 2011) and satellite readings of deforested land (Vadez et al., 2003).

Other studies have obtained objective measures through direct observations made by trained individuals of participants' behaviors. For example, self-reported recycling and the observed frequency of this behavior was weakly associated (r = .14; Corral-Verdugo, 1997), as was self-reported recycling with the amount of material put out for recycling (r = .16; McGuire, 1984). On the other hand, others have found a much higher correlation between self-reported and in–home observations of the re-use of glass, clothing, and metal (mean r = .63; Corral-Verdugo & Figueredo, 1999).

More recently, some researchers have explored the accuracy of self-reports using ratings by people who are close to the participant (e.g., spouses and housemates). Peer ratings differ from the above in that they are made by an individual who is not affiliated with a research team. This approach is particularly well-suited to proenvironmental behavior given that many relevant behaviors are performed in private. For example, individuals' self-reports of four environmental behaviors were strongly associated in one study with a report by their spouses (r = .45; Lam & Cheng, 2002). However, the frequency of self-reported proenvironmental behavior can be considerably higher than peer-reported behavior (Chao & Lam, 2011).

The accuracy of self-reports as measures of proenvironmental behavior is especially important given that they are often used as the key criterion variable in research and theory development; for example, the theory of planned behavior (Ajzen, 1985, Ajzen, 2005), one of the most widely used models for predicting proenvironmental behavior, often uses self-reports as the dependent variable and assumes that behavioral intention is the closest and most direct psychological determinant of behavior, and that intention is, in turn, causally determined by three factors: attitude towards the behavior (determined by values and beliefs), social norms, and perceived behavioral control. The theory of planned behavior has been applied to predict a variety of environmental behaviors, such as recycling (e.g., Boldero, 1995) and public transportation use (e.g., Heath & Gifford, 2002).

In a meta-analysis that examined the efficacy of the theory of planned behavior (Armitage & Conner, 2001) – in which the majority of the 185 studies included were health-related but nine had proenvironmental behavior as the outcome measure – the theory of planned behavior explained 10% more variance in self-reported behavior than it explained in objective or observed behavior (R2 = .31 versus R2 = .21). Consistent with this finding, a recent study found that the theory of planned behavior predicted self-reported proenvironmental behavior considerably better than it predicted peer-reported behavior (R2 = .81 versus R2 = .10; Chao & Lam, 2011). Thus, self-report validity can have implications for theory development, given that prediction models may inadvertently exaggerate the usefulness of a theory for explaining objective proenvironmental behavior when the criterion variable is a self-report of proenvironmental behavior. Of course, methodological issues, such as content overlap (i.e., where the theory of planned behavior measures and self-report measures share similar wording) and similarities in response format across measures can also affect shared variance. For example, the mono-method bias, which is shared variance among constructs that are all measured in the same way, has been demonstrated in the environmental psychology literature (Evans, 1999). Factors such as these can methodologically inflate the correlations between measures and thus could account for the differences in the proportion of variance explained by the theory of planned behavior.

Given the possible limitations of self-reports, understanding the factors that may influence their validity is important. Two socio-demographic variables may moderate the association between self-reported and objective behavioral measures.1 First, participant gender may influence validity because previous studies have demonstrated gender differences in proenvironmental behavior, in that females typically report engaging in more proenvironmental behaviors than males (e.g., Lam & Cheng, 2002; Zelezny, Chua, & Aldrich, 2000). Furthermore, health research has demonstrated differences in self-report validity between male and female participants; for instance, when asked to provide bodily measurements, women tend to under-report their weight more often than men, and men tend to over-report their height more often than women (Elgar & Stewart, 2008). Second, participant age should be evaluated because several studies have shown that younger people tend to report being more environmentally concerned than older people (e.g., Van Liere & Dunlap, 1980), although some studies have found no actual age difference in certain types of proenvironmental behavior, such as household electricity consumption (e.g., Fuj et al., 1985). But, again, evidence from the domain of health research has shown that age influences self-report validity; for example, older individuals are more likely to over-report their height, as compared to younger individuals (Elgar & Stewart). Age-related discrepancies in self-reported versus objective measures are also observed in self-reported vaccination records as compared to electronic medical records, such that adults over age 50 are slightly less likely to underreport vaccinations than adults under age 50 (Rolnick et al., 2013).

In addition, seven methodological variables should be considered. First, the number of items in the self-report measure may moderate the association. For example, some studies that have used a large number of items, such as Kaiser, Frick, and Stoll-Kleemann’s (2001) 65-item measure, have reported strong correlations (i.e., r = .81), whereas other studies that used only one item have reported much weaker correlations (r = .06, Fuj et al., 1985; r = .18, Hamilton, 1985). Second, the number of response options may moderate the self-report-proenvironmental behavior relation. Self-reports have been particularly accurate at predicting proenvironmental behavior when they are dichotomized, such as ‘I do’ versus ‘I don't’ (e.g., Kaiser, Doka, Hofstetter, & Ranney, 2003), whereas some studies that use non-dichotomized scales, such as Likert-style scales, with a range of response options have yielded weaker correlations (e.g., Corral-Verdugo, 1995). Third, whether the unit of analysis is the individual or a household may moderate the association given that it may be more difficult for people to estimate others' behavior as opposed to their own.

Several additional exploratory moderators are important to consider. Fourth, details about the nature of the objective measure (i.e., whether it is obtained by a mechanical device, trained observers, or peer ratings) may explain some variance among observed effect sizes. Fifth, the type of proenvironmental behavior (e.g., recycling, energy usage, water usage, or transportation) may influence the degree of congruence between the two types of measures. For instance, self-reports have been shown to be either greater or lesser than peer ratings depending on the type of proenvironmental behavior examined (e.g., Lam & Cheng, 2002). Even within one behavioral domain (i.e., waste reduction), the association between self-reported and observed re-use and recycling behavior has differed (r = .08 and r = .25, respectively; Corral-Verdugo, Bernache, Encinas, & Garibaldi, 1995). Furthermore, inconsistencies in the literature also suggest that the type of behavior examined may moderate the degree of observed congruence between these two types of measures. Sixth, whether or not the validity of self-reports changed over the time period of the studies included in the sample is worth exploring. Thus, the year of study publication was included as an exploratory moderator to assess whether the validity of self-reports has changed over the time period of the studies included in the sample. Research in the health research domain suggests that date of study publication is an important variable to assess; for example, a study assessed progress in the reliability and validity of self-report measures of HIV-related sexual behavior since 1990, in an effort to make recommendations for research and practice (Weinhardt, Forsyth, Carey, Jaworski, & Durant, 1998). Last, study location (i.e., country), may also be useful to consider given the possibility that self-report validity may vary across countries.

This meta-analysis evaluated all studies that could be located which included a self-reported and objective measure of the same proenvironmental behavior – both assessed using the same participants – to examine the overall association between the two types of measures. In doing so, it investigated the degree to which self-reports reflect objective behavior, as well as whether or not a systematic tendency exists for self-reports to under- or over-estimate objective behavior. An additional objective was to assess the potential influence of two socio-demographic and seven methodological moderators.

Section snippets

Selection of studies

This meta-analysis included only studies that examined associations between a self-reported measure of an ecologically relevant behavior and an objective measure of the same behavior. For the purposes of this analysis, ecologically relevant behavior encompasses any action relevant to environmental and resource sustainability, either in terms of behaviors within the household or within the broader community. Self-report measures ranged, for example, from individuals' assessments of their water

Descriptive statistics

In total, 6260 individuals or households participated in the 15 studies included in the meta-analysis. Table 2 displays the descriptive statistics for each study. On average, participants were 38.2 years of age (range 24.2–45.2 years, k = 6 studies or cases7) and 31.49% of the participants were male (range 0–100%, k = 9). Year of study publication ranged from 1984 to 2011. The mean number of items in the self-report measure was

Discussion

Self-report measures of proenvironmental behavior are commonly used to reflect objective proenvironmental behavior, although to date, evidence for the validity of these measures has been mixed. This meta-analysis set out to investigate the degree of association between these two types of measures by combining the results of studies that contain both self-reports and objective measures of the same proenvironmental behavior(s). Across 6260 individuals and households, representing 19 association

Acknowledgments

The first author wishes to gratefully acknowledge funding support received from the Social Sciences and Humanities Research Council (Joseph-Armand Bombardier Canada Graduate Scholarships – Doctoral award (Award No. 767-2010-1141)), as well as the Pacific Institute for Climate Solutions. In addition, the authors would like to acknowledge the research assistance of Emma Fraser and Jas Johal.

References13 (77)

  • I. Ajzen

    From intentions to actions: A theory of planned behavior

  • I. Ajzen

    Attitudes, personality and behavior

    (2005)
  • C.J. Armitage et al.

    Efficacy of the theory of planned behavior: A meta-analytic review

    British Journal of Social Psychology

    (2001)
  • S. Barr

    Factors influencing environmental attitudes and behaviours: A U.K. case study of household waste management

    Environment and Behavior

    (2007)
  • J. Boldero

    The prediction of household recycling of newspapers: The role of attitudes, intentions, and situational factors

    Journal of Applied Social Psychology

    (1995)
  • M. Borenstein et al.

    Comprehensive meta-analysis version 2

    (2005)
  • Y.-L. *Chao et al.

    Measuring responsible environmental behavior: Self-reported and other-reported measures and their differences in testing a behavioral model

    Environment & Behavior

    (2011)
  • J. Cohen

    Statistical power analysis for the behavioral sciences

    (1988)
  • J. Cohen

    A power primer

    Psychological Bulletin

    (1992)
  • V. *Corral-Verdugo et al.

    A comparison of two measures of reuse and recycling behavior: Self-report and material culture

    Journal of Environmental Systems

    (1995)
  • V. *Corral-Verdugo et al.

    Convergent and divergent validity of three measures of conservation behavior: The multitrait-multimethod approach

    Environment & Behavior

    (1999)
  • J.A. *Cote

    Use of household refuse analysis to measure usual and period-specific food consumption

    American Behavioral Scientist

    (1984)
  • R. De Young

    Promoting conservation behavior in shared spaces: The role of energy monitors

    Journal of Environmental Systems

    (1990)
  • S.J. Denny et al.

    Hand-held internet tablets for school-based data collection

    BMC Research Notes

    (2008)
  • A.L. Edwards

    The social desirability variable in personality assessment and research

    (1957)
  • F.J. Elgar et al.

    Validity of self-report screening for overweight and obesity: Evidence from the Canadian community health survey

    Canadian Journal of Public Health

    (2008)
  • G.W. Evans

    Measurement of the physical environment as stressor

  • E.T. *Fuj et al.

    An evaluation of the validity and reliability ofsurvey response data on household electricity conservation

    Evaluation Review

    (1985)
  • B. Gatersleben et al.

    Measurement and determinants of environmentally significant consumer behavior

    Environment and Behavior

    (2002)
  • E.S. Geller

    Evaluating energy conservation programs: Is verbal report enough?

    Journal of Consumer Research

    (1981)
  • R. Gifford

    A general model of social dilemmas

    International Journal of Ecological Economics and Statistics

    (2006)
  • L.C. *Hamilton

    Self-reported and actual savings in a water conservation campaign

    Environment & Behavior

    (1985)
  • E. *Hirst et al.

    Accuracy of self-reports: Energy conservation surveys

    The Social Science Journal

    (1985)
  • Y. Heath et al.

    Extending the theory of planned behavior: Predicting the use of public transportation

    Journal of Applied Social Psychology

    (2002)
  • J.F. Hemphill

    Interpreting the magnitudes of correlation coefficients

    American Psychologist

    (2003)
  • F.G. Kaiser et al.

    Ecological behavior, environmental attitude, and feelings of responsibility for the environment

    European Psychologist

    (1999)
  • F.G. *Kaiser et al.

    Zur Angemessenheit selbstberichteten Verhaltens: Eine Validitätsuntersuchung der Skala Allgemeinen Ökologischen Verhaltens

    Diagnostica

    (2001)
  • S. Khemlani et al.

    Theories of the syllogism: A meta-analysis

    Psychological Bulletin

    (2012)
  • Cited by (0)

    13

    Note: References denoted with an asterisk (*) are those studies included in our meta-analysis.

    View full text