Skip to main content

Advertisement

SpringerLink
Log in

Incentivised Online Panel Recruitment and Subjective Wellbeing: Caveat Emptor

  • Original Research
  • Published:
Journal of Well-Being Assessment

Abstract

It is generally assumed that if a sample represents its broader population on key demographic variables, the data it yields will also be representative. Here we present evidence to suggest that this is not necessarily so when subjective wellbeing is measured from participants recruited through online panels. Using data from six countries: Australia, Germany, Sweden, the Netherlands, UK and USA, we reveal significant differences in subjective wellbeing between online panel data and nationally representative data, even though both are demographically comparable. These findings indicate that the online panels comprised an abnormally high proportion of people with low subjective wellbeing, thus rendering their data non-representative. Given the widespread use of online panels to collect data in the modern era, we issue a caveat emptor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Andrews, F. M., & Withey, S. B. (1976). Social indicators of well-being: American's perceptions of life quality. New York: Plenum Press.

    Book  Google Scholar 

  • Australian Bureau of Statistics (2016). Household use of information technology, Australia, 2014–2015, Category no. 8146.0. Canberra, Government of Australia.

  • Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C., Krosnick, J., Lavrakas, P. J., Lee, S., Link, M., Piekarski, L., Rao, K., Thomas, R. K., & Zahs, D. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781.

    Article  Google Scholar 

  • Barber, T., Chilvers, D., & Kaul, S. (2013). Moving an established survey online - or not? International Journal of Market Research, 55(2), 2–11.

    Article  Google Scholar 

  • Bethlehem, J., & Stoop, I. (2007). Online panels - a paradigm theft. In The challenges of a changing world. Proceedings of the fifth international conference of the Association for Survey Computing, September, 113–137.

  • Blanchflower, D. G., & Oswald, A. J. (2004). Well-being over time in Britain and the USA. Journal of Public Economics, 88, 1359–1386. https://doi.org/10.1016/S0047-2727(02)00168-8.

  • Capic, T., Li, N., & Cummins, R. A. (2017). Set-points for subjective wellbeing: a replication and extension. Social Indicators Research, (in press).

  • Couper, M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Noncoverage and nonresponse in an Internet survey. Social Science Research, 36, 131–148.

    Article  Google Scholar 

  • Cummins, R. A. (1995). On the trail of the gold standard for life satisfaction. Social Indicators Research, 35(2), 179–200. https://doi.org/10.1007/BF01079026.

    Article  Google Scholar 

  • Cummins, R. A. (1998). The second approximation to an international standard of life satisfaction. Social Indicators Research, 43(3), 307–334. https://doi.org/10.1023/A:1006831107052.

    Article  Google Scholar 

  • Cummins, R. A. (2010). Subjective wellbeing, homeostatically protected mood and depression: a synthesis. Journal of Happiness Studies, 11, 1–17. https://doi.org/10.1007/s10902-009-9167-0.

    Article  Google Scholar 

  • Cummins, R. A. (2016). The theory of subjective wellbeing homeostasis: A contribution to understanding life quality. In F. Maggino (Ed.), A life devoted to quality of life – Festschrift in honor of Alex C. Michalos (Vol. 60, pp. 61–79). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Cummins, R. A., & Weinberg, M. K. (2015). Multi-item measurement of subjective wellbeing: Subjective approaches. In W. Glatzer, L. Camfield, V. Møller, & M. Rojas (Eds.), Global handbook of quality of life: exploration of well-being of nations and continents (pp. 239–268). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Cummins, R. A., Woerner, J., Weinberg, M., Collard, J., Hartley-Clark, L., & Horfiniak, K. (2013). Australian Unity Wellbeing Index: Report 30.0 - The wellbeing of Australians: Social media, personal achievement, and work. Melbourne: Australian Centre on Quality of Life, School of Psychology, Deakin http://www.acqol.com.au/uploads/surveys/survey-030-report-part-a.pdf.

  • De Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255.

    Google Scholar 

  • De Leeuw, E. D., Mellenbergh, G. J., & Hox, J. J. (1996). The influence of data collection method on structural models: a comparison of a mail, a telephone, and a face-to-face survey. Sociology Methods Research, 24, 442–472.

    Article  Google Scholar 

  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.

    Article  Google Scholar 

  • Douglas, S. P., & Craig, C. S. (1983). International Marketing Research. Englewood Cliffs: Prentice-Hall.

    Google Scholar 

  • European Social Survey. (2012). ESS round 6 (2012) project instructions (PAPI). London: Centre for Comparative Social Surveys, City University London http://www.europeansocialsurvey.org/docs/round6/fieldwork/source/ESS6_source_project_instructions.pdf. Last accessed 13 April 2015.

    Google Scholar 

  • Gallup. (2009). Gallup World Poll. Washington, DC: Gallup.

    Google Scholar 

  • Helliwell, J., Layard, R., & Sachs, J. (2017). World Happiness Report 2017. New York: Sustainable Development Solutions Network.

    Google Scholar 

  • International Wellbeing Group. (2013). Personal Wellbeing Index Manual. Melbourne: Deakin University http://www.acqol.com.au/instruments.

    Google Scholar 

  • Maggino, F. (2016) Challenges, needs and risks in defining wellbeing indicators. In F. Maggino (Ed.), A life devoted to quality of life. Social indicators research series, vol 60. Cham: Springer. https://doi.org/10.1007/978-3-319-20568-7_13

  • Meade, A.W., & Craig, S.B. (2011). Identifying careless responses in survey data. Paper presented at the 26th Annual Meeting of Society for Industrial and Organizational Psychology, Chicago, IL.

  • OECD. (2013). OECD guidelines on measuring subjective well-being. Paris: OECD Publishing. https://doi.org/10.1787/9789264191655-en.

    Google Scholar 

  • OECD. (2017). How’s life? 2017: measuring well-being. Paris: OECD Publishing. https://doi.org/10.1787/how_life-2017-en.

    Book  Google Scholar 

  • Prince, K. R., Litovsky, A. R., & Friedman-Wheeler, D. G. (2012). Internet-mediated research: beware of bots. The Behavior Therapist, 35, 85–88.

    Google Scholar 

  • Pruchno, R. A., & Hayden, J. M. (2000). Interview modality: effects on costs and data quality in a sample of older women. Journal of Aging & Health, 12, 3–24.

    Article  Google Scholar 

  • Shawver, Z., Griffith, J. D., Adams, L. T., Evans, J. V., Benchoff, B., & Sargent, R. (2016). An examination of the WHOQOL-BREF using four popular data collection methods. Computers in Human Behavior, 55, 446–454.

    Article  Google Scholar 

  • Suh, E., Diener, E., & Fujita, F. (1996). Events and subjective well-being: only recent events matter. Journal of Personality and Social Psychology, 70, 1091–1102.

    Article  Google Scholar 

  • The World Bank (2016). Accessed online January 25 2017 via https://data.worldbank.org/indicator/it.net.user.zs

  • Tomyn, A. J., Weinberg, M. K., & Cummins, R. A. (2015). Intervention efficacy among 'at-risk' adolescents: a test of subjective wellbeing homeostasis theory. Social Indicators Research, 120, 883–895. https://doi.org/10.1007/s11205-014-0619-5

  • Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.

    Article  Google Scholar 

  • WHO Regional Office for Europe. (2012). Measurement of and target-setting for well-being Second meeting of the expert group, Paris, 25–26 June 2012. Copenhagen: WHO Regional Office for Europe.

    Google Scholar 

  • WHOQOL Group. (1998). Development of the World Health Organization WHOQOL-BREF Quality of Life Assessment. Psychological Medicine, 28, 551–558.

    Article  Google Scholar 

  • Williams, J. (2012). Survey methods in an age of austerity: driving value in survey design. International Journal of Market Research, 54(1), 35–48.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Psychology, Faculty of Health, Deakin University, Melbourne, Australia

    Melissa K. Weinberg & Robert A. Cummins

  2. University of Western Australia, Crawley, WA, 6009, Australia

    David A. Webb

  3. Copenhagen Business School, 2000, Frederiksberg, Denmark

    Wencke Gwozdz

  4. Justus-Liebig-Universität Gießen, Senckenbergstr. 3 (Zeughaus), 35390, Gießen, Germany

    Wencke Gwozdz

Authors
  1. Melissa K. Weinberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert A. Cummins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David A. Webb
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wencke Gwozdz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Melissa K. Weinberg or David A. Webb.

Ethics declarations

Conflict of Interest Statement

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Appendix 1: The General Life Satisfaction (GLS) and Personal Wellbeing Index (PWI) Items

Appendix 1: The General Life Satisfaction (GLS) and Personal Wellbeing Index (PWI) Items

1.1 English

GLS: Thinking about your own life and personal circumstances, how satisfied are you with your life as a whole? 0 = Not at all satisfied, 10 = Completely satisfied.

PWI: How satisfied are you with…?

  1. 1.

    your standard of living? [Standard of Living]

  2. 2.

    your health? [Personal Health]

  3. 3.

    what you are achieving in life? [Achieving in Life]

  4. 4.

    your personal relationships? [Personal Relationships]

  5. 5.

    how safe you feel? [Personal Safety]

  6. 6.

    feeling part of your community? [Community-Connectedness]

  7. 7.

    your future security? [Future Security]

1.2 German

GLS: Wie zufrieden sind Sie insgesamt mit Ihrem Leben? 0 = Ganz und gar nicht zufrieden; 10 = Völlig zufrieden.

PWI: Wie zufrieden sind Sie mit …?

  1. 8.

    Ihrem Lebensstandard?

  2. 9.

    Ihrer Gesundheit?

  3. 10.

    dem, was Sie in Ihrem Leben erreicht haben?

  4. 11.

    Ihren persönlichen Beziehungen?

  5. 12.

    damit, wie sicher Sie sich fühlen?

  6. 13.

    damit, wie Sie sich als Teil Ihrer sozialen Gemeinschaft fühlen?

  7. 14.

    Ihrer zukünftigen Sicherheit?

1.3 Dutch

GLS: Alles bij elkaar genomen, hoe tevreden bent u vandaag de dag met uw leven in het algemeen? 0 = Zeer ontevreden en 10 = Zeer tevreden.

PWI: Hoe tevreden bent u over...

  1. 15.

    uw levensstandaard?

  2. 16.

    uw gezondheid?

  3. 17.

    wat u hebt bereikt in het leven?

  4. 18.

    uw persoonlijke relaties?

  5. 19.

    uw gevoel van veiligheid?

  6. 20.

    uw gevoel deel uit te maken van een gemeenschap?

  7. 21.

    uw zekerheid van bestaan in de toekomst?

1.4 Swedish

GLS: Hur nöjd är du med livet i allmänhet? 0 = inte alls nöjd och 10 = helt nöjd.

PWI: Hur nöjd är du met…?

  1. 22.

    din levnadsstandard?

  2. 23.

    din hälsa?

  3. 24.

    det du uppnår i livet?

  4. 25.

    dina personliga relationer?

  5. 26.

    hur säker du känner dig?

  6. 27.

    hur mycket du känner dig som en del av. samhället?

  7. 28.

    din framtida säkerhet?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weinberg, M.K., Cummins, R.A., Webb, D.A. et al. Incentivised Online Panel Recruitment and Subjective Wellbeing: Caveat Emptor. J well-being assess 2, 41–55 (2018). https://doi.org/10.1007/s41543-018-0010-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41543-018-0010-4

Keywords

Search

Navigation