Ran Gilad-Bachrach
ABSTRACT
Collecting large labeled data sets is a laborious and expensive task, whose scaling up requires division of the labeling workload between many teachers. When the number of classes is large, miscorrespondences between the labels given by the different teachers are likely to occur, which, in the extreme case, may reach total inconsistency. In this study we describe how globally consistent labels can be obtained, despite the absence of teacher coordination, and discuss the possible efficiency of this process in terms of human labor. We define a notion of label efficiency, measuring the ratio between the number of globally consistent labels obtained and the number of labels provided by distributed teachers. We show that the efficiency depends critically on the ratio α between the number of data instances seen by a single teacher, and the number of classes. We suggest several algorithms for the distributed labeling problem, and analyze their efficiency as a function of α. In addition, we provide an upper bound on label efficiency for the case of completely uncoordinated teachers, and show that efficiency approaches 0 as the ratio between the number of labels each teacher provides and the number of classes drops (i.e. α → 0).
- A. C. Atkinson and A. N. Donve. optimum experiment designs. Oxford University Press, 1992.Google Scholar
- A. Bar-Hillel, T. Hertz, N. Shental, and D. Weinshall. Learning a mahalanobis metric from equivalence constraints. Journal of Machine Learning Reseach (JMLR), 6(Jun):937--965, 2005. Google ScholarDigital Library
- A. Bar-Hillel and D. Weinshall. Learning with equivalence constraints, and the relation to multiclass classification. In Conference on Learning Theory (COLT), 2003.Google Scholar
- D. Cohn, L. Atlas, and R. Ladner. Training connectionist networks with queries and selective sampling. Advanced in Neural Information Processing Systems 2, 1990. Google ScholarDigital Library
- S. E. Decator. Efficient Learning from Faulty Data. PhD thesis, Harvard University, 1995. Google ScholarDigital Library
- M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html, 2007.Google Scholar
- Y. Freund, H. Seung, E. Shamir, and N. Tishby. Selective sampling using the query by committee algorithm. Machine Learning, 28:133--168, 1997. Google ScholarDigital Library
- G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, California Institute of Technology, 2007.Google Scholar
- B. Russell, A. Torralba, K. Murphy, and W. Freeman. Labelme: a database and web-based tool for image annotation. mit ai lab memo aim-2005-025, 2005.Google Scholar
- L. von Ahn. Games with a purpose. IEEE Computer, 39(6):92--94, 2006. Google ScholarDigital Library
Index Terms
- Efficient human computation: the distributed labeling problem
Recommendations
A comprehensive human computation framework: with application to image labeling
MM '08: Proceedings of the 16th ACM international conference on MultimediaImage and video labeling is important for computers to understand images and videos and for image and video search. Manual labeling is tedious and costly. Automatically image and video labeling is yet a dream. In this paper, we adopt a Web 2.0 approach ...
Efficient computation of entropy gradient for semi-supervised conditional random fields
NAACL-Short '07: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short PapersEntropy regularization is a straightforward and successful method of semi-supervised learning that augments the traditional conditional likelihood objective function with an additional term that aims to minimize the predicted label entropy on unlabeled ...
Selective Weakly Supervised Human Detection under Arbitrary Poses
AbstractIn this paper we study the problem of weakly supervised human detection under arbitrary poses within the framework of multi-instance learning (MIL). Our contributions are threefold: (1) we first show that in the context of weakly supervised ...
Highlights- We propose a novel Selective Weakly Supervised Detection method which outperforms the previous state-of-the-art methods.
- We annotate a new large-scale data set called LSP/MPII-MPHB (Multiple Poses Human Body) for human body detection.
Comments