2010 Proceeding
Research in data mining and knowledge discovery relies heavily on the availability of datasets. However, compared to the amount of work in the field on techniques for pattern discovery and knowledge extraction, there has been relatively little effort directed at the study of effective methods for collecting and evaluating the quality of data.
Human computation is a relatively new research area that studies the process of channeling the vast internet population to perform tasks or provide data towards solving difficult problems that no known efficient computer algorithms can yet solve. There has been a lot of work on games with a purpose (e.g., the ESP Game) that specifically target online gamers who, in the process of playing an enjoyable game, generate useful data (e.g., image tags). There has been considerable interest and research on crowdsourcing marketplaces (e.g. Amazon Mechanical Turk), which are essentially human computation applications that coordinate workers to perform tasks in exchange for monetary rewards.
While there have been significant research challenges, increasing business interest and active work in human computation, till last year there was no dedicated forum to discuss these ideas. The first Human Computation Workshop (HComp2009) was held on June 28th, 2009, in Paris, France, collocated with KDD 2009. With HComp2010, we hope to continue to serve the needs of researchers and practitioners interested in this area, integrating work from a number of fields including KDD, information retrieval, gaming, machine learning, game theory, and Human Computer Interaction. Learning from HComp 2009, we expanded the topics of relevance to the workshop. We have also updated the organizing and program committees, bringing new people in, while keeping enough of the last year's team for continuity.
This year, our call for papers resulted in 35 submissions (18 long papers, 11 short papers and 6 demos) from a range of perspectives. All submissions were thoroughly reviewed, typically with 3 reviews each, by the organizing and program committees and external reviewers. Even though we had a number of submissions of high quality, given that we had only a half-day workshop, we could accept only 4 long papers, 4 short papers, 5 posters and 5 demos; these are the submissions that appear in the proceedings.
The overall themes that emerged from last year's workshop were very clear: on the one hand, there is the experimental side of human computation, with research on new incentives for users to participate, new types of actions, and new modes of interaction. On the more theoretic side, we have research modeling these actions and incentives to examine what theory predicts about these designs. Finally, last year's program focused on how to best handle noise, identify labeler expertise, and use the generated data for data mining purposes. This year's submissions demonstrated the continuation and evolution of these themes with the accepted papers divided into three sessions: "Market Design", "Human Computation in Practice", and "Task and Process Design". In the first session, the authors discuss the economic context of human computation and the practical lessons learned from building a human computation resource. In the second session we see many practical examples of human computation in fields as far-flung as virtual world construction to language translation and beyond. In the final session, the authors focus on how to get the best quality results out of the crowdsourced work. Given the strong submissions for demos both last year and this, we see -- and the community seems to recognize -- the poster and demo session as an integral part of the workshop, where participants can showcase their human computation applications.
Proceeding Downloads
Task search in a human computation market
In order to understand how a labor market for human computation functions, it is important to know how workers search for tasks. This paper uses two complementary methods to gain insight into how workers search for tasks on Mechanical Turk. First, we ...
The anatomy of a large-scale human computation engine
In this paper we describe Rabj, an engine designed to simplify collecting human input. We have used Rabj to collect over 2.3 million human judgments to augment data mining, data entry, and curation tasks at Freebase over the course of a year. We ...
Sellers' problems in human computation markets
"Tools for human computers" is an underexplored design space in human computation research, which has focused on techniques for buyers of human computation rather than sellers. We characterize the sellers in one human computation market, Mechanical Turk,...
Frontiers of a paradigm: exploring human computation with digital games
Each day millions of people play digital games with different motivations. These motivations rang from time beating to deep immersion into a narration or interacting with a community. To address all these different means, a range of game designs is ...
GiveALink tagging game: an incentive for social annotation
Social tagging systems lead to inferred relationships among resources, tags and users from shared annotations in support of applications such as search, recommendation, and navigation. However, users share annotations largely for their own individual ...
Crowdsourcing participation inequality: a SCOUT model for the enterprise domain
In large scale online multi-user communities, the phenomenon of 'participation inequality,' has been described as generally following a more or less 90-9-1 rule [9]. In this paper, we examine crowdsourcing participation levels inside the enterprise (...
Mutually reinforcing systems
This paper demonstrates strategies for designing mobile games with by-products in order to allow the acquisition of specific data. A mobile game with by-products called EyeSpy and a photo website called Realise will be used as examples to demonstrate ...
A note on human computation limits
In 2007, Penguin Books and De Montfort University used a wiki to crowd source a novel. The result was deemed a failure because of the many un-integrated elements; however, in the last year a model utilizing Twitter has achieved success: BBC Audiobooks ...
Reconstructing the world in 3D: bringing games with a purpose outdoors
We are interested in reconstructing real world locations as detailed 3D models, but to achieve this goal, we require a large quantity of photographic data. We designed a game to employ the efforts and digital cameras of everyday people to not only ...
Improving music emotion labeling using human computation
In the field of Music Information Retrieval, there are many tasks that are not only difficult for machines to solve, but that also lack well-defined answers. In pursuing the automatic recognition of emotions within music, this lack of objectivity makes ...
Webpardy: harvesting QA by HC
We present a human computation online game for enabling users to contribute to the creation of a corpus of question-resource pairs for harvesting web-based question answering. Our idea was motivated by the popular 'jeopardy' quiz.
Measuring utility of human-computer interaction
Human Computation, along with much of the Internet, only works when humans find tasks fun, enjoyable, or valuable enough to outweigh the time and effort they require to complete. The more value, or utility, that a task and interface provides, the more "...
Translation by iterative collaboration between monolingual users
In this paper we describe Monotrans, a new iterative translation process designed to leverage the massive number of online users who have minimal or no bilingual skill.
Sentence recall game: a novel tool for collecting data to discover language usage patterns
Recently we ran a simple memory test experiment, called sentence recall, in which participants were asked to recall sentences that they had just seen on the screen. Many participants, especially non-native English speakers, made various deviations in ...
Word sense disambiguation via human computation
One formidable problem in language technology is the word sense disambiguation (WSD) problem: disambiguating the true sense of a word as it occurs in a sentence (e.g., recognizing whether the word "bank" refers to a river bank or to a financial ...
Quality management on Amazon Mechanical Turk
Crowdsourcing services, such as Amazon Mechanical Turk, allow for easy distribution of small tasks to a large number of workers. Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take advantage ...
Exploring iterative and parallel human computation processes
Services like Amazon's Mechanical Turk have opened the door for exploration of processes that outsource computation to humans. These human computation processes hold tremendous potential to solve a variety of problems in novel and interesting ways. ...
Toward automatic task design: a progress report
A central challenge in human computation is in understanding how to design task environments that effectively attract participants and coordinate the problem solving process. In this paper, we consider a common problem that requesters face on Amazon ...
Cited By
-
Xue M, Xia S, Hu F, Yu H, Zhu Y and Chang V (2021). A method of adaptive filtering and learning based on label noise 2nd International Conference on Industrial IoT, Big Data, and Supply Chain, 10.1117/12.2624150, 9781510651326, (12)
-
Bentley A, Petcovic H and Cassidy D (2016). Development and validation of the anthropogenic climate change dissenter inventory, Environmental Education Research, 10.1080/13504622.2016.1250150, 25:6, (867-882), Online publication date: 3-Jun-2019.
- Proceedings of the ACM SIGKDD Workshop on Human Computation