Alan Wexelblat
Abstract
This article presents research—an experiment and the resulting prototype—on a method for treating gestural input so that it can be used for multimodal applications, such as interacting with virtual environments. This method involves the capture and use of natural , empty-hand gestures that are made during conventional descriptive utterances. Users are allowed to gesture in a normal continuous manner, rather than being restricted to a small set of discrete gestural commands as in most other systems. The gestures are captured and analyzed into a higher-level description. This description can be used by an application-specific interpreter to understand the gestural input in its proper context. Having a gesture analyzer of this sort enables natural gesture input to any appropriate application.
- BADLER, P. AND WEBBER, B.L. 1993. Simulating Humans: Computer Graphics Animatwn and Control. Oxford University Press, New York. Google Scholar
- Bos, E. 1992. Some virtues and limitations of action inferring interfaces. In Proceedings of UIST '92 (Monterey, Calif., Nov. 15 18). ACM Press, New York, 79-88. Google Scholar
- BOLT, R.A. 1984. The Human Interface. Van Nostrand Reinhold, New York.Google Scholar
- BOLT, R. A. ANn HER~NZ, E.J. 1992. Two-handed gesture with speech in multi-modal natural dialogue. In Proceedings of UIST '92. ACM Press, New York. Google Scholar
- CODELLA, C., JALILI, R., KOVED, L., LEWIS, B., LING, D. T., LIPSCOMB, J. S., RABENHORST, D. A., WANG, C. P., NORTON, h., SWEENEY, P., AND TURK, G. 1992. Interactive simulation in a multi-person virtual world. In Proceedings of CHI '92. ACM Press, New York, 329-334. Google Scholar
- DARRELL, T. AND PENTLANO, A. 1993. Space-time gestures. In the IEEE Conference on Vision and Pattern Recognition (June). IEEE, New York.Google Scholar
- FRENKEL, K.A. 1994. A conversation with Brenda Laurel. Interactions 1, i (Jan.), 45-52. Google Scholar
- KOONS~ D. B, SPARRELL~ C. J, AND THORISSON~ K.R. 1993. Integrating simultaneous input from speech, gaze, and hand gestures. In Intelligent Multi-Media Interfaces, M. Maybury, Ed. AAAI Press, Menlo Park, Calif., 252-276. Google Scholar
- KRUEGER, M. 1991. Artificial Reality. Addison-Wesley, Reading, Mass.Google Scholar
- LEWIS, J. B., KOVED, L., AND LING, D. 1991. Dialog structures for virtual worlds. In CHI'91 Proceedings. ACM Press, New York. Google Scholar
- MAES, P. 1993. ALIVE: An artificial interactive video environment. In Visual Proceedings of the SIGGRAPH '93 Conference: ACM SIGGRAPH. ACM Press, New York, 189-190.Google Scholar
- McNEILL, D. 1993. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, Ill.Google Scholar
- MURAKAMI, K. AND TAGUCH~, H. 1991. Gesture recognition using recurrent neural networks. In CHI'91 Conference Proceedings. ACM Press, New York. Google Scholar
- QUEK, F. 1994. Toward a vision-based hand gesture interface. In ACM Virtual Reality Systems and Technology ( VRST'94 ) Proceedings. ACM Press, New York. Google Scholar
- RIME, B. AND SCHIARATURA, L. 1991. Gesture and speech. In Fundamentals of Nonverbal Behavior, R. Feldman and B. Rime Eds. Press Syndicate of the University of Cambridge, New York, 239 281.Google Scholar
- SPARRELL, C.J. 1993. Coverbal iconic gestures in human-computer interaction. S. M. thesis, MTT Media Arts and Sciences Section, Cambridge, Mass.Google Scholar
- THORISSON, K., KOONS, D., AND BOLT, R. 1992. Multi-modal natural dialogue. In Proceedings of CHI '92. ACM Press, New York, 653-654. Google Scholar
- VAANANEN, K. AND BOHM, K. 1993. Gesture-driven interaction as a human factor in virtual environments--An approach with neural networks. In Virtual Reality Systems, M. A. Gigante and H. Jones, Eds. Academic Press, Ltd., London, U.K.Google Scholar
- WEIMER, D. AND GANAPATHY, S.K. 1989. A synthetic visual environment with hand gesturing and voice input. In CHI'89 Proceedings. ACM Press, New York. Google Scholar
- WEIMER, D. AND GANAPATHY, S. K. 1992. Interaction techniques using hand tracking and speech recognition. In Multimedia Interface Design, M. Blattner and R. B. Dannenberg, Eds. ACM Press, New York. Google Scholar
- WEXELBLAT, A. 1994a. A feature-based approach to continuous-gesture analysis. S. M. thesis, MIT Program in Media Arts and Sciences, Cambridge, Mass. Available as ftp://media.mit.edu/ wex/MS-Thesis.ps/gz.Google Scholar
- WEXELBLAT, A. 1994b. Natural gesture in virtual environments. In ACM Virtual Reality Systems and Technology (VRST'94 ) Proceedings. ACM Press, New York. Google Scholar
Index Terms
- An approach to natural gesture in virtual environments
Recommendations
Natural gesture in descriptive monologues
SIGGRAPH '06: ACM SIGGRAPH 2006 CoursesGesture plays a prominent role in human-human interaction, and it offers promise as a new modality for human-computer interaction. However, our understanding of gesture is still at an early stage. This study explores gesture in natural interaction and ...
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
Visual Gesture Interfaces for Virtual Environments
AUIC '00: Proceedings of the First Australasian User Interface ConferenceVirtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual ...
Reviews
Richard L. Upchurch
The movement toward the creation and use of virtual environments requires a consideration of what would characterize activity immersion. That is, how can we successfully incorporate users into virtual reality in a manner that allows them to act or behave as if they were truly in that situation__?__ Certainly an expectation for activity immersion would be the ability to use gestures to indicate activity (for <__?__Pub Fmt nolinebreak>example,<__?__Pub Fmt /nolinebreak><__?__Pub Caret> opening a door or pushing a button). This paper reports on efforts to understand gestures as they arise as part of normal human activity (in this case narrative descriptions) to “make gesticulation understandable to computers as an input mode.” The experiment reported was an exploration in gesture classification. The results indicate that gestures could serve as an intermediate language for input. The author claims, without providing evidence, that the descriptive language “worked well across subjects and across gestures within subjects.” The results of the experiment were used to design a feature-based gesture analyzer. A significant portion of the paper deals with issues related to the design of the system and its rationale (the separation of the analyzer from the interpreter). The paper does not include data on using the prototype with users or any quantitative assessment of the system in action.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments