M. Six Silberman
Bill Tomlinson
ABSTRACT
"Tools for human computers" is an underexplored design space in human computation research, which has focused on techniques for buyers of human computation rather than sellers. We characterize the sellers in one human computation market, Mechanical Turk, and describe some of the challenges they face. We list several projects developed to approach these problems, and conclude with a list of open questions relevant to sellers, buyers, and researchers.
- Zittrain, J. Ubiquitous human computing. Phil. Trans. R. Soc. A 366: 3813--3821, 2008.Google Scholar
Cross Ref
- Horton, J. J. and L. Chilton. The labor economics of paid crowdsourcing. arXiv:1001.0627v1 {cs.HC}, 2010. Google ScholarDigital Library
- Horton, J. J. The condition of the Turking class. arXiv:1001.1172v1 {cs.CY}, 2010.Google Scholar
- Snow, R., et al. Cheap and fast---but is it good? Proc. EMNLP '08.Google Scholar
- Feng, D., et al., Acquiring high-quality non-expert knowledge from on-demand workforce. Proc. ACL-IJCNLP '09. Google ScholarDigital Library
- Hsieh, G. et al. Can markets help? Proc. CSCW '08: 535--544. Google ScholarDigital Library
- Hsieh, G. and S. Counts. mimir. Proc. CHI '09: 769--778. Google ScholarDigital Library
- Hsieh, G., et al. Why pay? Proc. CHI '10: 305--314. Google ScholarDigital Library
- Burnham, T. and R. Sami. A reputation system for selling human computation. Proc. HCOMP '09. Google ScholarDigital Library
- Jain, S. and D. C. Parkes. The role of game theory in human computation systems. Proc. HCOMP '09. Google ScholarDigital Library
- Mason, W. and D. J. Watts. Financial incentives and the "performance of crowds." Proc. HCOMP '09. Google ScholarDigital Library
- Kittur, A., et al. Crowdsourcing user studies with Mechanical Turk. Proc. CHI '08: 453--456. Google ScholarDigital Library
- Heer, J. and M. Bostock. Crowdsourcing graphical perception. Proc. CHI '10: 203--212. Google ScholarDigital Library
- Jeon, G. Y., et al. Re-examining price as a predictor of answer quality. Proc. CHI '10: 325--328. Google ScholarDigital Library
- Seuken, S., et al. Hidden markets: UI design for a P2P backup application. Proc. CHI '10: 315--324. Google ScholarDigital Library
- Downs, J. S., et al. Are your participants gaming the system? Proc. CHI '10: 2399--2402. Google ScholarDigital Library
- Ross, J., et al. Who are the crowdworkers? Proc. CHI '10: 2863--2872. Google ScholarDigital Library
- Ross, J., et al. Who are the crowdworkers? Slides, alt. CHI, 14 April 2010, Atlanta, GA. www.ics.uci.edu/~jwross/pubs/RossEtAl-altCHI2010-slides.pdf.Google Scholar
- Ipeirotis, P. Mechanical Turk: The demographics. behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html.Google Scholar
- Ipeirotis, P. Turker demographics vs. internet demographics. behind-the-enemy-lines.blogspot. com/2009/03/turker-demographics-vs-internet.html.Google Scholar
- Ross, J. and B. Tomlinson. Worker motivations in Amazon Mechanical Turk. Unpub. manuscript, 2009.Google Scholar
- Ipeirotis, P. The new demographics of Mechanical Turk. behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html.Google Scholar
- spamgirl et al. HITs you should NOT Do! turkers.proboards.com/index.cgi?board= kgeneral&action=display&thread=5126.Google Scholar
- Irani, L. Turk Work. turkwork.differenceengines.com.Google Scholar
- Hartmann, B. Crowdsourcers respond to crowdsourcing. blogs.law.harvard.edu/difficultprobs/2010/01/07/crowdsourcers-respond-to-crowdsourcing.Google Scholar
- techlist. MTurk Requesters. techlist.com/mturk/requesters.Google Scholar
- techlist et al. Current requesters. mturkforum.com/showthread.php?t=1100.Google Scholar
- lightdark et al. Recording your history, would you like to? mturkforum.com/showthread.php?t=1109.Google Scholar
- lightdark et al. Scripts for user data. mturkforum.com/showthread.php?t=424.Google Scholar
- ergo et al. Block (hide) requester script for Firefox+Greasemonkey. mturkforum.com/showthread.php?t=1088.Google Scholar
- CloudCrowd. Intro from CloudCrowd, an alternative to Mechanical Turk. mturkforum.com/showthread.php?t=1169.Google Scholar
- Ba, S., et al. The dynamics of the electronic market. Inf. Sys. Frontiers 2(1): 31--40, 2000. Google ScholarDigital Library
- Hahn, J. The dynamics of mass online marketplaces. Proc. CHI '01: 317--324. Google ScholarDigital Library
- Resnick, P., et al. The value of reputation on eBay. Experimental Econ. 9(2): 79--101, 2006.Google ScholarCross Ref
- Steiglitz, K. Snipers, Shills, and Sharks: eBay and Human Behavior. Princeton, 2007. Google ScholarDigital Library
- Franklin, J., et al. An inquiry into the nature and causes of the wealth of internet miscreants. Proc. CCS '07: 375--388. Google ScholarDigital Library
- Li, G., et al. Botnet economics: uncertainty matters. In M. Johnson, ed., Managing Information Risk and the Economics of Security: 245--267. Springer, 2008.Google Scholar
- Namestnikov, Y. The economics of botnets. Kaspersky Lab Tech. Rep., 2009.Google Scholar
- Akerlof, G. A. and R. Shiller. Animal Spirits. Princeton, 2009.Google Scholar
- Zittrain, J. and E. Stark. Cyberlaw: Difficult Issues. cyber.law.harvard.edu/cyberlaw_winter10/Main_Page.Google Scholar
- andrewj and spamgirl. MTurk "App Store"? turkers.proboards.com/index.cgi?board=mturksuggest&action=display&thread=5543.Google Scholar
Index Terms
- Sellers' problems in human computation markets
Recommendations
Competing sellers in online markets: reserve prices, shill bidding, and auction fees
AAMAS '06: Proceedings of the fifth international joint conference on Autonomous agents and multiagent systemsIn this paper, we consider competition between sellers offering similar items in concurrent online auctions, where each seller must set its individual auction parameters (such as the reserve price) in such a way as to attract buyers. We show that there ...
Competing sellers in online markets: reserve prices, shill bidding, and auction fees
TADA/AMEC'06: Proceedings of the 2006 AAMAS workshop and TADA/AMEC 2006 conference on Agent-mediated electronic commerce: automated negotiation and strategy design for electronic marketsIn this paper, we consider competition between sellers offering similar items in concurrent online auctions, where each seller must set its individual auction parameters (such as the reserve price) in such a way as to attract buyers. We show that in the ...
Sellers competing for buyers in online markets: reserve prices, shill bids, and auction fees
IJCAI'07: Proceedings of the 20th international joint conference on Artifical intelligenceWe consider competition between sellers offering similar items in concurrent online auctions through a mediating auction institution, where each seller must set its individual auction parameters (such as the reserve price) in such a way as to attract ...
Comments