Skip to main content
SpringerLink
Log in

The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

  • Published:
Behavior Research Methods Aims and scope Submit manuscript

Abstract

In this study, we examined data quality among Amazon Mechanical Turk (MTurk) workers based in India, and the effect of monetary compensation on their data quality. Recent studies have shown that work quality is independent of compensation rates, and that compensation primarily affects the quantity but not the quality of work. However, the results of these studies were generally based on compensation rates below the minimum wage, and far below a level that was likely to play a practical role in the lives of workers. In this study, compensation rates were set around the minimum wage in India. To examine data quality, we developed the squared discrepancy procedure, which is a task-based quality assurance approach for survey tasks whose goal is to identify inattentive participants. We showed that data quality is directly affected by compensation rates for India-based participants. We also found that data were of a lesser quality among India-based than among US participants, even when optimal payment strategies were utilized. We additionally showed that the motivation of MTurk users has shifted, and that monetary compensation is now reported to be the primary reason for working on MTurk, among both US- and India-based workers. Overall, MTurk is a constantly evolving marketplace where multiple factors can contribute to data quality. High-quality survey data can be acquired on MTurk among India-based participants when an appropriate pay rate is provided and task-specific quality assurance procedures are utilized.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Antin, J., & Shaw, A. (2012). Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India. In CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2925–2934). New York, NY: ACM.

    Google Scholar 

  • Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351–368.

    Article  Google Scholar 

  • Bohannon, J. (2011). Human subject research: Social science for pennies. Science, 334, 307. doi:10.1126/science.334.6054.307

    Article  PubMed  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980

    Article  Google Scholar 

  • Bureau of Democracy, Human Rights and Labor. (2012). Country Reports on Human Rights Practices, India. Retrieved December 22, 2013, from U.S. Department of State website, www.state.gov/j/drl/rls/hrrpt/2012humanrightsreport/index

  • Chandler, J., Mueller, P., & Paolacci, G. (2013). Methodological concerns and advanced uses of crowdsourcing in psychological research. Manuscript submitted for publication. http://www.jessechandler.com/uploads/2/8/0/5/2805897/mturk_adv_methods.pdf

  • Chouliarakis, G., & Correa-López, M. (2014). A fair wage model of unemployment with inertia in fairness perceptions. Oxford Economic Papers, 66(1), 88–114.

    Article  Google Scholar 

  • DeScioli, P., Christner, J., & Kurzban, R. (2011). The omission strategy. Psychological Science, 22, 442–446.

    Article  PubMed  Google Scholar 

  • Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are your participants gaming the system? Screening Mechanical Turk Workers. In CHI ’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2399–2402). New York, NY: ACM.

    Google Scholar 

  • Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5, 159–163.

    Google Scholar 

  • Falk, A., Fehr, E., & Zehnder, C. (2006). Fairness perceptions and reservation wages—The behavioral effects of minimum wage laws. The Quarterly Journal of Economics, 121, 1347–1381.

    Google Scholar 

  • Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011). What’s the right price? Pricing tasks for finishing on time. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (pp. 26–31). Menlo Park, CA: AAAI.

    Google Scholar 

  • Gardner, R. M., Brown, D. L., & Boice, R. (2012). Using Amazon’s Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9, 532–534.

    Article  PubMed  Google Scholar 

  • Government of India, Ministry of Labor and Employment. (2011).Second annual report to the people on employment. Retrieved December 22, 2013, from the Government of India website, dget.gov.in/publications/annualreportemployment2011.pdf.

  • Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce (pp. 209–218). New York, NY: ACM. doi:10.1145/1807342.1807376

    Google Scholar 

  • Ipeirotis, P. (2010). Demographics of Mechanical Turk [Working Article]. New York University. http://hdl.handle.net/2451/29585, http://scholar.google.com/scholar?cluster=8873729056826117132&hl=en&as_sdt=0,33

  • Jenkins, G. D. J., Mitra, A., Gupta, N., & Shaw, J. D. (1998). Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology, 83, 777–787.

    Article  Google Scholar 

  • John, O. P., Robins, R. W., & Pervin, L. A. (2008). Handbook of personality, third edition: Theory and research. New York, NY: Guilford Press.

    Google Scholar 

  • Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon’s Mechanical Turk to increase student-faculty collaborative research. Teaching of Psychology, 39, 245–251.

    Article  Google Scholar 

  • Jones, D. N., & Paulhus, D. L. (2011). The role of impulsivity in the Dark Triad of personality. Personality and Individual Differences, 51, 679–682.

    Article  Google Scholar 

  • Kazai, G., Kamps, J., & Milic-Frayling, N. (2013). An analysis of human factors and label accuracy in crowdsourcing relevance judgements. Information Retrieval Journal, 16, 138–178. doi:10.1007/s10791-012-9205-0

    Article  Google Scholar 

  • Marge, M., Banerjee, S., & Rudnicky, A. I. (2010). Using the Amazon Mechanical Turk for transcription of spoken language. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (pp. 5270–5273). Los Alamitos, CA: IEEE Press. doi:10.1109/ICASSP.2010.5494979

    Google Scholar 

  • Mason, W., & Watts, D. J. (2010). Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2), 100–108.

  • Nyhan, B., & Reifler, J. (2011). Opening the political mind? The effects of self-affirmation and graphical information on factual misperceptions. Unpublished manuscript. Hanover, NH: Dartmouth College.

    Google Scholar 

  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.

    Article  Google Scholar 

  • Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.

    Google Scholar 

  • Paulhus, D. L., & Carey, J. M. (2011). The FAD-Plus: measuring lay beliefs regarding free will and related constructs. Journal of Personality Assessment, 93, 96–104.

    Article  PubMed  Google Scholar 

  • Rani, U., & Belser, P. (2012). The effectiveness of minimum wages in developing countries: The case of India. International Journal of Labour Research, 4, 45–66.

    Google Scholar 

  • Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Mechanical Turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). New York, NY: ACM.

    Google Scholar 

  • Schmitt, D. P., Allik, J., McCrae, R. R., & Benet-Martinez, V. (2007). The geographic distribution of Big Five Personality traits: Patterns and profiles of human self-description across 56 nations. Journal of Cross-Cultural Psychology, 38, 173–212.

    Article  Google Scholar 

  • Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.

    Article  Google Scholar 

  • Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology: General, 141, 423–428. doi:10.1037/a0025391

    Article  Google Scholar 

  • Silberman, M. S., Ross, J., Irani, L., & Tomlinson, B. (2010). Sellers’ problems in human computation markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 18–21). New York, NY: ACM.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Lander College, 75-31 150th Street, Flushing, NY, 11367, USA

    Leib Litman & Cheskie Rosenzweig

  2. Department of Computer Science, Lander College, 75-31 150th Street, Flushing, NY, 11367, USA

    Jonathan Robinson

Authors
  1. Leib Litman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cheskie Rosenzweig
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leib Litman.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Litman, L., Robinson, J. & Rosenzweig, C. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav Res 47, 519–528 (2015). https://doi.org/10.3758/s13428-014-0483-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3758/s13428-014-0483-x

Keywords

Search

Navigation