Most research in data mining and knowledge discovery relies heavily on the availability of datasets. With the rapid growth of user generated content on the internet, there is now an abundance of sources from which data can be drawn. Compared to the amount of work in the field on techniques for pattern discovery and knowledge extraction, there has been relatively little effort directed at the study of effective methods for collecting and evaluating the quality of data.
Human computation is a new research area that studies the process of channeling the vast internet population to perform tasks or provide data towards solving difficult problems that no known efficient computer algorithms can yet solve. There are various genres of human computation applications available today. Games with a purpose (e.g., the ESP Game) specifically target online gamers who, in the process of playing an enjoyable game, generate useful data (e.g., image tags). Crowdsourcing marketplaces (e.g. Amazon Mechanical Turk) are human computation applications that coordinate workers to perform tasks in exchange for monetary rewards. In identity verification tasks, users need to perform some computation in order to access some online content; a recent example of such a human computation application is reCAPTCHA, which leverages millions of users who solve CAPTCHAs every day to correct words in books that optical character recognition (OCR) programs fail to recognize with certainty.
While there has been active work in this area, there is no dedicated forum to discuss these research ideas. The Human Computation Workshop (HCOMP 2009), held in conjunction with the 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), is the first workshop bringing together academic and industry researchers to discuss existing human computation applications and future directions of this new subject area. An integral part of this workshop is a demo session where participants can showcase their human computation applications.
Our call for papers resulted in 33 high-quality submissions from a wide variety of perspectives. All papers were thoroughly reviewed by the program committee and external reviewers. Given the short, half-day duration of the workshop, only one-third of the submissions could be accommodated and appear in the proceedings. The accepted papers have been divided into four sessions: "Games", "Human Computation In Practice", "Game Theory" and "Labeling Cost and Efficiency".
Proceeding Downloads
User-centered design of a social game to tag music
We present "Herd It", a competitive, online, multi-player game that has the implicit benefit of collecting tags for music. We describe Herd It's user-centered design process and demonstrate that the game can collect both musical and social data. This ...
KissKissBan: a competitive human computation game for image annotation
In this paper, we propose a competitive human computation game, KissKissBan (KKB), for image annotation. KKB is different from other human computation games since it integrates both collaborative and competitive elements in the game design. In a KKB ...
Community-based game design: experiments on social games for commonsense data collection
Games with A Purpose have successfully harvested information from web users. However, designing games that encourage sustainable and quality data contribution remains a great challenge. Given that many online communities have enjoyed active ...
A demonstration of human computation using the Phrase Detectives annotation game
The goal of the ANAWIKI project is to experiment with Web collaboration and human computation to create largescale linguistically annotated corpora. We will present ongoing work and initial results of Phrase Detectives, a game designed to collect ...
Picture this: preferences for image search
We demonstrate a system designed to elicit relative relevance judgments from users to rank images with respect to an image query. The system has been deployed and in use publicly for approximately one year. Furthermore, preference data collected from ...
Page Hunt: using human computation games to improve web search
There has been a lot of work on evaluating and improving the relevance of web search engines, primarily using human relevance judgments or using clickthrough data. Both of these approaches look at the problem of learning the mapping from queries to web ...
TurKit: tools for iterative tasks on mechanical Turk
Mechanical Turk (MTurk) is an increasingly popular web service for paying people small rewards to do human computation tasks. Current uses of MTurk typically post independent parallel tasks. We are exploring an alternative iterative paradigm, in which ...
Search war: a game for improving web search
We present a competitive online game called Search War, which collects data that is useful for improving Web search. Specifically, as a by product of gameplay, players will provide, for a given web page, an evaluation of its relevance to a particular ...
Magic Bullet: a dual-purpose computer game
In this demo, we showcase Magic Bullet, an online game that is designed to streamline the robustness evaluation of CAPTCHAs. This game can also train people's typing skills, and be used to aid the development of better machine learning algorithms for ...
Seaweed: a web application for designing economic games
Seaweed is a web application for experimental economists with no programming background to design two-player symmetric games in a visual-oriented interface. Games are automatically published to the web where players can play against each other remotely ...
Thumbs-Up: a game for playing to rank search results
- Ali Dasdan,
- Chris Drome,
- Santanu Kolay,
- Micah Alpern,
- Alice Han,
- Tom Chi,
- Jamie Hoover,
- Ivan Davtchev,
- Sharad Verma
Human computation is an effective way to channel human effort spent playing games to solving computational problems that are easy for humans but difficult for computers to automate. We propose Thumbs-Up, a new game for human computation with the purpose ...
Games for games: manipulating contexts in human computation games
The present work and demonstration system aims at finding an efficient and cost-effective human computation method to expand the linguistic capabilities of interactive games that need it to respond appropriately to the language based input of their ...
From active towards InterActive learning: using consideration information to improve labeling correctness
Active learning methods have been proposed to reduce the labeling effort of human experts: based on the initially available labeled instances and information about the unlabeled data those algorithms choose only the most informative instances for ...
TagCaptcha: annotating images with CAPTCHAs
Image retrieval has long been plagued by limitations on automatic methods because they cannot reliably extract semantic data from low-level features. The result is that users must formulate awkward and inefficient queries in terms these systems can ...
CAPTCHA-based image labeling on the Soylent Grid
We introduce an open labeling platform for Computer Vision researchers based on Captchas, creating as a byproduct labeled image data sets while supporting web security. For the two different tasks of annotation and detection, we explore usability ...
Designing crowdsourcing community for the enterprise
In this paper, we describe the design principles used for implementing crowdsourcing within the enterprise. This is based on our distinction between two kinds of crowdsourcing: enterprise (inside a firewall) versus the public domain. Whereas public ...
A reputation system for selling human computation
We describe a reputation-driven market that motivates human computation sellers (workers) to produce optimal levels of quality when quality is not immediately measurable and contracts specifying the level of quality cannot be enforced. We consider the ...
The role of game theory in human computation systems
The paradigm of "human computation" seeks to harness human abilities to solve computational problems or otherwise perform distributed work that is beyond the scope of current AI technologies. One aspect of human computation has become known as "games ...
On formal models for social verification
The introduction of the ESP Game and other Games With A Purpose (GWAP) has demonstrated the potential of human computation in solving AI-hard problems. In such systems, users are normally required to input answers for questions proposed by the system, ...
Efficient human computation: the distributed labeling problem
Collecting large labeled data sets is a laborious and expensive task, whose scaling up requires division of the labeling workload between many teachers. When the number of classes is large, miscorrespondences between the labels given by the different ...
Financial incentives and the "performance of crowds"
The relationship between financial incentives and performance, long of interest to social scientists, has gained new relevance with the advent of web-based "crowd-sourcing" models of production. Here we investigate the effect of compensation on ...
Cited By
- Tetreault J, Chodorow M and Madnani N (2014). Bucking the trend, Language Resources and Evaluation, 48:1, (5-31), Online publication date: 1-Mar-2014.
-
Horton J Employer Expectations, Peer Effects and Productivity: Evidence from a Series of Field Experiments, SSRN Electronic Journal, 10.2139/ssrn.1652993
- Proceedings of the ACM SIGKDD Workshop on Human Computation