Lydia B. Chilton
John J. Horton
ABSTRACT
In order to understand how a labor market for human computation functions, it is important to know how workers search for tasks. This paper uses two complementary methods to gain insight into how workers search for tasks on Mechanical Turk. First, we perform a high frequency scrape of 36 pages of search results and analyze it by looking at the rate of disappearance of tasks across key ways Mechanical Turk allows workers to sort tasks. Second, we present the results of a survey in which we paid workers for self-reported information about how they search for tasks. Our main findings are that on a large scale, workers sort by which tasks are most recently posted and which have the largest number of tasks available. Furthermore, we find that workers look mostly at the first page of the most recently posted tasks and the first two pages of the tasks with the most available instances but in both categories the position on the result page is unimportant to workers. We observe that at least some employers try to manipulate the position of their task in the search results to exploit the tendency to search for recently posted tasks. On an individual level, we observed workers searching by almost all the possible categories and looking more than 10 pages deep. For a task we posted to Mechanical Turk, we confirmed that a favorable position in the search results do matter: our task with favorable positioning was completed 30 times faster and for less money than when its position was unfavorable.
- E. Adar, M. Dontcheva, J. Fogarty, and D. Weld. Zoetrope: interacting with the ephemeral web. In Proceedings of the 21st annual ACM symposium on User interface software and technology, pages 239--248. ACM, 2008. Google Scholar
Digital Library
- D. Autor. Wiring the labor market. Journal of Economic Perspectives, 15(1):25--40, 2001.Google ScholarCross Ref
- D. Bates and D. Sarkar. lme4: Linear mixed-effects models using S4 classes. URL http://CRAN.R-project.org/package=lme4, R package version 0.999375-28, 2008.Google Scholar
- S. Brin, L. Page, R. Motwami, and T. Winograd. The PageRank citation ranking: bringing order to the web. In Proceedings of ASIS'98, pages 161--172, 1998.Google Scholar
- B. Edelman, M. Ostrovsky, and M. Schwarz. Internet advertising and the generalized second-price auction: Selling billions of dollars worth of keywords. The American Economic Review, pages 242--259, 2007.Google Scholar
- A. Gelman and J. Hill. Data analysis using regression and multilevel/hierarchical models. Cambridge University Press Cambridge, 2007.Google Scholar
- D. Goldberg, D. Nichols, B. M. Oki, and D. Terry. Using collaborative filtering to weave an information tapestry. Commun. ACM, 35(12):61--70, 1992. Google ScholarDigital Library
- M. Hearst, A. Elliott, J. English, R. Sinha, K. Swearingen, and K. Yee. Finding the flow in web site search. Communications of the ACM, 45(9):49, 2002. Google ScholarDigital Library
- J. Horton and L. Chilton. The labor economics of paid crowdsourcing. Proceedings of the 11th ACM Conference on Electronic Commerce 2010 (forthcoming), 2010. Google ScholarDigital Library
- J. J. Horton, D. Rand, and R. J. Zeckhauser. The Online Laboratory: Conducting Experiments in a Real Labor Market. NBER Working Paper w15961, 2010.Google Scholar
- B. A. Huberman, D. Romero, and F. Wu. Crowdsourcing, attention and productivity. Journal of Information Science (in press), 2009. Google ScholarDigital Library
- P. Ipeirotis. Demographics of mechanical turk. New York University Working Paper, 2010.Google Scholar
- E. Law, L. Von Ahn, R. Dannenberg, and M. Crawford. Tagatune: A game for music and sound annotation. In International Conference on Music Information Retrieval (ISMIR'07), pages 361--364. Citeseer, 2003.Google Scholar
- W. Mason and D. J. Watts. Financial incentives and the 'performance of crowds'. In Proc. ACM SIGKDD Workshop on Human Computation (HCOMP), 2009. Google ScholarDigital Library
- M. Silberman, J. Ross, L. Irani, and B. Tomlinson. Sellers' problems in human computation markets. 2010.Google Scholar
- A. Spink and J. Xu. Selected results from a large study of Web searching: the Excite study. Inform. Resear.---Int. Electron. J, 6(1), 2000.Google Scholar
- L. Von Ahn and L. Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319--326. ACM, 2004. Google ScholarDigital Library
- H. Wickham. ggplot2: An implementation of the grammar of graphics. R package version 0.7, URL: http://CRAN.R-project.org/package-ggplot2, 2008.Google Scholar
Index Terms
- Task search in a human computation market
Recommendations
How many crowdsourced workers should a requester hire?
Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow
WebSci '13: Proceedings of the 5th Annual ACM Web Science ConferenceOntology evaluation has proven to be one of the more difficult problems in ontology engineering. Researchers proposed numerous methods to evaluate logical correctness of an ontology, its structure, or coverage of a domain represented by a corpus. ...
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsA growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Comments