Anuruddha Hettiarachchi
Daniel Wigdor
ABSTRACT
Advances in display and tracking technologies hold the promise of increasingly immersive augmented-reality experiences. Unfortunately, the on-demand generation of haptic experiences is lagging behind these advances in other feedback channels. We present Annexing Reality; a system that opportunistically annexes physical objects from a user's current physical environment to provide the best-available haptic sensation for virtual objects. It allows content creators to a priori specify haptic experiences that adapt to the user's current setting. The system continuously scans user's surrounding, selects physical objects that are similar to given virtual objects, and overlays the virtual models on to selected physical ones reducing the visual-haptic mismatch. We describe the developer's experience with the Annexing Reality system and the techniques utilized in realizing it. We also present results of a developer study that validates the usability and utility of our method of defining haptic experiences.
Supplemental Material
- Merwan Achibet, Maud Marchal, Ferran Argelaguet, and Anatole Lecuyer. 2014. The Virtual Mitten: A novel interaction paradigm for visuo-haptic manipulation of objects using grip force. 2014 IEEE Symposium on 3D User Interfaces (3DUI), IEEE, 59--66. http://doi.org/10.1109/3DUI.2014.6798843Google Scholar
Cross Ref
- Marco Agus, Andrea Giachetti, Enrico Gobbetti, Gianluigi Zanetti, and Antonio Zorcolo. 2014. A multiprocessor decoupled system for the simulation of temporal bone surgery. Computing and Visualization in Science 5, 1: 35--43. http://doi.org/10.1007/s00791002-0085--5Google ScholarDigital Library
- Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T. Silva. 2003. Computing and rendering point set surfaces. IEEE Transactions on Visualization and Computer Graphics 9, 1: 3--15. http://doi.org/10.1109/TVCG.2003.1175093 Google ScholarDigital Library
- Yuki Ban, Takashi Kajinami, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2012. Modifying an identified angle of edged shapes using pseudo-haptic effects. Haptics: Perception, Devices, Mobility, and Communication: 25--36. Retrieved September 23, 2015 from http://link.springer.com/chapter/10.1007/978-3-64231401-8_3 Google ScholarDigital Library
- Yuki Ban, Takashi Kajinami, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2012. Modifying an identified curved surface shape using pseudo-haptic effect. 2012 IEEE Haptics Symposium (HAPTICS), IEEE, 211--216. http://doi.org/10.1109/HAPTIC.2012.6183793 Google ScholarDigital Library
- Yuki Ban, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2012. Modifying an identified position of edged shapes using pseudo-haptic effects. Proceedings of the 18th ACM symposium on Virtual reality software and technology - VRST '12, ACM Press, 93--96. http://doi.org/10.1145/2407336.2407353 Google ScholarDigital Library
- Paul J. Besl and Neil D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2: 239--256. http://doi.org/10.1109/34.121791 Google ScholarDigital Library
- Anders G. Buch, Dirk Kraft, Joni-Kristian Kamarainen, Henrik G. Petersen, and Norbert Kruger. 2013. Pose estimation using local structure-specific shape and appearance context. 2013 IEEE International Conference on Robotics and Automation, IEEE, 2080--2087. http://doi.org/10.1109/ICRA.2013.6630856Google ScholarCross Ref
- Albert S. Carlin, Hunter G. Hoffman, and Suzanne Weghorst. 1997. Virtual reality and tactile augmentation in the treatment of spider phobia: a case report. Behaviour Research and Therapy 35, 2: 153--158. http://doi.org/10.1016/S0005-7967(96)00085-XGoogle ScholarCross Ref
- Kai-Yin Cheng, Rong-Hao Liang, Bing-Yu Chen, Rung-Huei Laing, and Sy-Yen Kuo. 2010. iCon: utilizing everyday objects as additional, auxiliary and instant tabletop controllers. Proceedings of the 28th international conference on Human factors in computing systems CHI '10, ACM Press, 1155--1164. http://doi.org/10.1145/1753326.1753499 Google ScholarDigital Library
- Christian Corsten, Ignacio Avellino, Max Möllers, and Jan Borchers. 2013. Instant user interfaces. Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces ITS '13, ACM Press, 71--80. http://doi.org/10.1145/2512349.2512799 Google ScholarDigital Library
- Michael Csongei, Liem Hoang, Ulrich Eck, and Christian Sandor. 2012. ClonAR: Rapid redesign of real-world objects. 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, 277--278. http://doi.org/10.1109/ISMAR.2012.6402572 Google ScholarDigital Library
- Lisa M. Di Diodato, Richard Mraz, Nicole S. Baker, and Simon J. Graham. 2007. A haptic force feedback device for virtual reality-fMRI experiments. IEEE transactions on neural systems and rehabilitation engineering?: a publication of the IEEE Engineering in Medicine and Biology Society 15, 4: 570--576. http://doi.org/10.1109/TNSRE.2007.906962Google Scholar
- Markus Funk, Oliver Korn, and Albrecht Schmidt. 2014. An augmented workplace for enabling userdefined tangibles. Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems CHI EA '14, ACM Press, 1285--1290. http://doi.org/10.1145/2559206.2581142 Google ScholarDigital Library
- Sidhant Gupta, Dan Morris, Shwetak N. Patel, and Desney Tan. 2013. AirWave: non-contact haptic feedback using air vortex rings. Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing - UbiComp '13, ACM Press, 419--428. http://doi.org/10.1145/2493432.2493463 Google ScholarDigital Library
- Steven J. Henderson and Steven Feiner. 2008. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. Proceedings of the 2008 ACM symposium on Virtual reality software and technology - VRST '08, ACM Press, 211--218. http://doi.org/10.1145/1450579.1450625 Google ScholarDigital Library
- Valentin Heun, Shunichi Kasahara, and Pattie Maes. 2013. Smarter objects: using AR technology to program physical objects and their interactions. CHI '13 Extended Abstracts on Human Factors in Computing Systems on CHI EA '13, ACM Press, 961--966. http://doi.org/10.1145/2468356.2468528 Google ScholarDigital Library
- Ken Hinckley, Randy Pausch, John C. Goble, and Neal F. Kassell. 1994. Passive real-world interface props for neurosurgical visualization. Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence CHI '94, ACM Press, 452--458. http://doi.org/10.1145/191666.191821 Google ScholarDigital Library
- Koichi Hirota and Michitaka Hirose. 1995. Providing force feedback in virtual environments. IEEE Computer Graphics and Applications 15, 5: 22--30. http://doi.org/10.1109/38.403824 Google ScholarDigital Library
- Hunter G. Hoffman. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180), IEEE Comput. Soc, 59--63. http://doi.org/10.1109/VRAIS.1998.658423 Google ScholarCross Ref
- Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits. Proceedings of the SIGCHI conference on Human factors in computing systems CHI '97, ACM Press, 234--241. http://doi.org/10.1145/258549.258715 Google ScholarDigital Library
- Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and Ryo Kawamura. 2001. Project FEELEX: adding haptic surface to graphics. Proceedings of the 28th annual conference on Computer graphics and interactive techniques SIGGRAPH '01, ACM Press, 469--476. http://doi.org/10.1145/383259.383314 Google ScholarDigital Library
- Benjamin Knoerlein, Gábor Székely, and Matthias Harders. 2007. Visuo-haptic collaborative augmented reality ping-pong. Proceedings of the international conference on Advances in computer entertainment technology - ACE '07, ACM Press, 91--94. http://doi.org/10.1145/1255047.1255065 Google ScholarDigital Library
- Aaron Kotranza and Benjamin Lok. 2008. Virtual Human + Tangible Interface = Mixed Reality Human An Initial Exploration with a Virtual Breast Exam Patient. 2008 IEEE Virtual Reality Conference, IEEE, 99--106. http://doi.org/10.1109/VR.2008.4480757Google ScholarCross Ref
- Harold W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2, 1--2: 83--97. http://doi.org/10.1002/nav.3800020109Google ScholarCross Ref
- Eun Kwon, Gerard J. Kim, and Sangyoon Lee. 2009. Effects of sizes and shapes of props in tangible augmented reality. 2009 8th IEEE International Symposium on Mixed and Augmented Reality, IEEE, 201--202. http://doi.org/10.1109/ISMAR.2009.5336463 Google ScholarDigital Library
- Daniel Leithinger, Sean Follmer, Alex Olwal, et al. 2013. Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13, ACM Press, 1441--1450. http://doi.org/10.1145/2470654.2466191 Google ScholarDigital Library
- Thomas H. Massie and Kenneth J. Salisbury. 1994. The phantom haptic interface: A device for probing virtual objects. Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems, 295-- 300.Google Scholar
- Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays. IEICE TRANSACTIONS on Information and Systems E77-D, 1321--1329.Google Scholar
- Yoichi Ochiai, Kota Kumagai, Takayuki Hoshi, Jun Rekimoto, Satoshi Hasegawa, and Yoshio Hayasaki. 2015. Fairy lights in femtoseconds. ACM SIGGRAPH 2015 Posters on SIGGRAPH '15, ACM Press, 1--1. http://doi.org/10.1145/2787626.2792630Google ScholarDigital Library
- Ohan Oda, Levi J. Lister, Sean White, and Steven Feiner. 2008. Developing an augmented reality racing game. 2. Retrieved September 21, 2015 from http://dl.acm.org/citation.cfm?id=1363200.1363203Google ScholarCross Ref
- Bernhard Pflesser, Andreas Petersik, Ulf Tiede, Karl Heinz Höhne, and Rudolf Leuwer. 2002. Volume cutting for virtual petrous bone surgery. Computer aided surgery?: official journal of the International Society for Computer Aided Surgery 7, 2: 74--83. http://doi.org/10.1002/igs.10036Google Scholar
- Jun Rekimoto. 2014. Traxion: a tactile interaction device with virtual force sensation. ACM SIGGRAPH 2014 Emerging Technologies on SIGGRAPH '14, ACM Press, 1--1. http://doi.org/10.1145/2614066.2614079 Google ScholarDigital Library
- Radu B. Rusu, Nico Blodow, and Michael Beetz. 2009. Fast Point Feature Histograms (FPFH) for 3D registration. 2009 IEEE International Conference on Robotics and Automation, IEEE, 3212--3217. http://doi.org/10.1109/ROBOT.2009.5152473 Google ScholarDigital Library
- Radu B. Rusu. 2010. Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. KI - Künstliche Intelligenz 24, 4: 345--348. http://doi.org/10.1007/s13218-010-0059-6Google ScholarCross Ref
- Satoshi Saga and Ramesh Raskar. 2012. Feel through window. SIGGRAPH Asia 2012 Emerging Technologies on SA '12, ACM Press, 1--3. http://doi.org/10.1145/2407707.2407715Google Scholar
- Makoto Sato. 2002. SPIDAR and virtual reality. Proceedings of the 5th Biannual World Automation Congress, TSI Press, 17--23. http://doi.org/10.1109/WAC.2002.1049515Google ScholarCross Ref
- Ruwen Schnabel, Roland Wahl, and Reinhard Klein. 2007. Efficient RANSAC for Point-Cloud Shape Detection. Computer Graphics Forum 26, 2: 214--226. http://doi.org/10.1111/j.1467-8659.2007.01016.xGoogle ScholarCross Ref
- Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional Reality. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems CHI '15, ACM Press, 3307--3316. http://doi.org/10.1145/2702123.2702389 Google ScholarDigital Library
- Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson, and Ali Israr. 2013. AIREAL: interactive tactile experiences in free air. ACM Transactions on Graphics 32, 4: 134. http://doi.org/10.1145/2461912.2462007 Google ScholarDigital Library
- Yuriko Suzuki and Minoru Kobayashi. 2005. Air jet driven force feedback in virtual reality. IEEE Computer Graphics and Applications 25, 1: 44--47. http://doi.org/10.1109/MCG.2005.1 Google ScholarDigital Library
- James Vallino and Christopher Brown. 1999. Haptics in augmented reality. Proceedings IEEE International Conference on Multimedia Computing and Systems, IEEE Comput. Soc, 195--200. http://doi.org/10.1109/MMCS.1999.779146 Google ScholarDigital Library
- Peter Weir, Christian Sandor, Matt Swoboda, Ulrich Eck, Gerhard Reitmayr, and Arindam Dey. 2012. BurnAR: Feel the heat. 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, 331--332. http://doi.org/10.1109/ISMAR.2012.6402599 Google ScholarDigital Library
- Microsoft HoloLens. Retrieved September 21, 2015 from https://www.microsoft.com/microsofthololens/en-usGoogle Scholar
- Kinect for Xbox One. Retrieved September 21, 2015 from http://www.xbox.com/en-ca/xboxone/accessories/kinect-for-xbox-oneGoogle Scholar
- ATOMIC Authoring Tool. Retrieved September 21, 2015 from http://www.sologicolibre.org/projects/atomic/en/Google Scholar
- buildAR. Retrieved September 21, 2015 from https://buildar.com/startGoogle Scholar
- Vuforia Augmented Reality for 3D Mobile Content. Retrieved September 21, 2015 from https://www.qualcomm.com/products/vuforiaGoogle Scholar
- metaio. Retrieved September 22, 2015 from https://www.metaio.com/Google Scholar
- dlib C++ Library - Optimization. Retrieved September 21, 2015 from http://dlib.net/optimization.htmlGoogle Scholar
- Meta Augmented Reality. Retrieved September 23, 2015 from https://www.getameta.com/Google Scholar
Index Terms
- Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality
Recommendations
Remixed Reality: Manipulating Space and Time in Augmented Reality
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsWe present Remixed Reality, a novel form of mixed reality. In contrast to classical mixed reality approaches where users see a direct view or video feed of their environment, with Remixed Reality they see a live 3D reconstruction, gathered from multiple ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical SystemsAs shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...
Comments