Benfang Xiao
Sharon Oviatt
ABSTRACT
Multimodal interfaces are designed with a focus on flexibility, although very few currently are capable of adapting to major sources of user, task, or environmental variation. The development of adaptive multimodal processing techniques will require empirical guidance from quantitative modeling on key aspects of individual differences, especially as users engage in different types of tasks in different usage contexts. In the present study, data were collected from fifteen 66- to 86-year-old healthy seniors as they interacted with a map-based flood management system using multimodal speech and pen input. A comprehensive analysis of multimodal integration patterns revealed that seniors were classifiable as either simultaneous or sequential integrators, like children and adults. Seniors also demonstrated early predictability and a high degree of consistency in their dominant integration pattern. However, greater individual differences in multimodal integration generally were evident in this population. Perhaps surprisingly, during sequential constructions seniors' intermodal lags were no longer in average and maximum duration than those of younger adults, although both of these groups had longer maximum lags than children. However, an analysis of seniors' performance did reveal lengthy latencies before initiating a task, and high rates of self talk and task-critical errors while completing spatial tasks. All of these behaviors were magnified as the task difficulty level increased. Results of this research have implications for the design of adaptive processing strategies appropriate for seniors' applications, especially for the development of temporal thresholds used during multimodal fusion. The long-term goal of this research is the design of high-performance multimodal systems that adapt to a full spectrum of diverse users, supporting tailored and robust future systems.
- Berk, L. E. Why children talk to themselves. Scientific American, 1994, 271(5), 78--83.Google Scholar
Cross Ref
- Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen & J. Clow. QuickSet: Multimodal interaction for distributed applications. In Proc. of Multimedia'97, 31--40. Google ScholarDigital Library
- Comblain, A. Working memory in Down's Syndrome: Training the rehearsal strategy. Down's Syndrome: Research and Practice, 1994, 2(3), 123--126.Google ScholarCross Ref
- Czaja, S. J. & C. C. Lee. Designing computer systems for older adults. In Handbook of Human-Computer Interaction (J. Jacko & A. Sears, eds.). LEA, NY, 2002, 413--427. Google ScholarDigital Library
- Heckmann, M., F. Berthommier & K. Kroschel. Noise adaptive stream weighting in audio-visual speech recognition. EURASIP JASP, 2002, 11, 1260--1273. Google ScholarDigital Library
- Huhns, M. & G. Weiss. Guest Editorial. Machine Learning (special issue on Multiagent Learning), 1998, 33(2-3), 123--128. Google ScholarDigital Library
- Illina, I. Tree-structured maximum a posteriori adaptation for a segment-based speech recognition system. In Proc. of ICSLP'02, 1405--1408.Google Scholar
- Iyengar, G. & C. Neti. A vision-based microphone switch for speech intent detection. In Proc. of RATFG-RTS'01, 101--105. Google ScholarDigital Library
- Jacko, J. A., I. U. Scott, F. Sainfort, K. P. Moloney, T. Kongnakorn, B. S. Zorich & V. K. Emery. Effects of multimodal feedback on the performance of older adults with normal and impaired vision. Lecture Notes in Computer Science (LNCS), 2003, 2615, 3-22. Google ScholarDigital Library
- Kart, C. S., E. K. Metress & S. P. Metress. Aging, Health and Society. Jones and Bartlett, Boston MA, 1988.Google Scholar
- Kemper, S. & T. L. Mitzner. Language, production and comprehension. In Handbook of the Psychology of Aging 5th Ed (J. E. Birren & K. W. Schaie, eds.). Academic Press, San Diego CA, 2001, 378--398.Google Scholar
- Luria, A. R. The Role of Speech in the Regulation of Normal and Abnormal Behavior. Liveright, NY, 1961.Google Scholar
- Oviatt, S. L. Ten myths of multimodal interaction. Communications of the ACM, 1999, 42(11), 74--81. Google ScholarDigital Library
- Oviatt, S. L., P. R. Cohen, M. W. Fong & M. P. Frank. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. In Proc. of ICSLP'92, 2, 1351--1354.Google Scholar
- Oviatt, S. L., P. R. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson & D. Ferro. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. Human Computer Interaction, 2000, 15(4), 263--322. Google ScholarDigital Library
- Oviatt, S. L., A. DeAngeli & K. Kuhn. Integration and synchronization of input modes during multimodal human-computer interaction. In Proc. of CHI'97, 415--422. Google ScholarDigital Library
- Salthouse, T. A. Theoretical Perspectives on Cognitive Aging. LEA, Hillsdale NJ, 1991.Google Scholar
- Swanson, H. L. What develops in working memory? A life span perspective. Dev. Psychology, 1999, 35(4), 986--1000.Google ScholarCross Ref
- Wingfield, A. & E. A. L. Stine-Morrow. Language and speech. In Handbook of Cognitive Aging 2nd Ed (F. I. M. Craik & T. A. Salthouse, eds.). LEA, Mahwah NJ, 2000. 359--416.Google Scholar
- Xiao, B., C. Girand & S. L. Oviatt. Multimodal integration patterns in children. In Proc. of ICSLP'02, 629--632.Google Scholar
Index Terms
- Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences
Recommendations
Individual differences in multimodal integration patterns: what are they and why do they exist?
CHI '05: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsTechniques for information fusion are at the heart of multimodal system design. To develop new user-adaptive approaches for multimodal fusion, the present research investigated the stability and underlying cause of major individual differences that have ...
Toward a theory of organized multimodal integration patterns during human-computer interaction
ICMI '03: Proceedings of the 5th international conference on Multimodal interfacesAs a new generation of multimodal systems begins to emerge, one dominant theme will be the integration and synchronization requirements for combining modalities into robust whole systems. In the present research, quantitative modeling is presented on ...
When do we interact multimodally?: cognitive load and multimodal communication patterns
ICMI '04: Proceedings of the 6th international conference on Multimodal interfacesMobile usage patterns often entail high and fluctuating levels of difficulty as well as dual tasking. One major theme explored in this research is whether a flexible multimodal interface supports users in managing cognitive load. Findings from this ...
Comments