Erin Treacy Solovey
Francine Lalooses
Audrey Girouard
Robert J.K. Jacob
ABSTRACT
Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.
- Bailey, B. P., et al. The Effects of Interruptions on Task Performance, Annoyance, and Anxiety in the User Interface. INTERACT'01, 2001, 593--601.Google Scholar
- Card, S., Moran, T. and Newell, A. The Psychology of Human Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ, 1983. Google ScholarDigital Library
- Chance, B., et al. A novel method for fast imaging of brain function, non-invasively, with light. Optics Express, 10 (2). 411--423.Google Scholar
- Chen, D., et al. Towards a Physiological Model of User Interruptability. INTERACT'07, 2008, 439--451. Google ScholarDigital Library
- Czerwinski, M., et al. Instant Messaging: Effects of Relevance and Timing. HCI, 2000, 71--76.Google Scholar
- Fairclough, S. H. Fundamentals of physiological computing. Interact. Comput., 21 (1-2). 133--145. Google ScholarDigital Library
- Fogarty, J., Hudson, S. E. and Lai, J. Examining the robustness of sensor-based statistical models of human interruptibility. CHI'04, ACM, 2004. Google ScholarDigital Library
- Girouard, A., Solovey, E. T. and Jacob, J. K. Designing a Passive Brain Computer Interface using Real Time Classification of Functional Near-Infrared Spectroscopy. Intl J of Auton. & Adaptive Comm. Sys. (2010). in press. Google ScholarDigital Library
- Grimes, D., et al. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. CHI'08, ACM, 2008. Google ScholarDigital Library
- Hall, M., et al. The WEKA data mining software: an update. SIGKDD Explor. Newsl., 11 (1). 10--18. Google ScholarDigital Library
- Hirshfield, L., et al. Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy. CHI'09, 2009. Google ScholarDigital Library
- Hornof, A. J.,et al. Knowing where and when to look in a time-critical multimodal dual task. CHI '10, 2010. Google ScholarDigital Library
- Hudson, S., et al. Predicting human interruptibility with sensors: a Wizard of Oz feasibility study. CHI '03, 2003. Google ScholarDigital Library
- Huppert, T., et al. A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans. Neuroimage, 2006:29(2). 368--382.Google Scholar
- Iqbal, S. T., et al. Towards an index of opportunity: understanding changes in mental workload during task execution. CHI '05, 2005. Google ScholarDigital Library
- Iqbal, S. T. and Bailey, B. P. Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. CHI extended abstracts, 2005. Google ScholarDigital Library
- Iqbal, S. T., Zheng, X. S. and Bailey, B. P. Task-evoked pupillary response to mental workload in human-computer interaction. CHI extended abstracts, 2004. Google ScholarDigital Library
- Jackson, M. M. and Mappus, R. Applications for Brain-Computer Interfaces. in Tan, D. and Nijholt, A. eds. Brain Computer Interfaces: Applying our Minds to Human-Computer Interaction, Springer, 2010, 89--104.Google ScholarCross Ref
- Koechlin, E., et al. The role of the anterior prefrontal cortex in human cognition. Nature, 1999, 148--151.Google Scholar
- Koechlin, E., et al. Dissociating the role of the medial and lateral anterior prefrontal cortex in human planning. PNAS, 97. 7651--7656.Google Scholar
- Kuikkaniemi, K., et al. The influence of implicit and explicit biofeedback in first-person shooter games. CHI'10, 2010. Google ScholarDigital Library
- Lee, J. C. and Tan, D. S. Using a low-cost electroencephalograph for task classification in HCI research UIST'06, ACM Press, 2006. Google ScholarDigital Library
- Mandryk, R. L. and Inkpen, K. M. Physiological indicators for the evaluation of co-located collaborative play CSCW '04, 2004. Google ScholarDigital Library
- McFarlane, D. Comparison of four primary methods for coordinating the interruption of people in human-computer interaction. Hum.-Comput. Interact., 17(1). 63--139. Google ScholarDigital Library
- Miyata, Y. and Norman, D. The Control of Multiple Activities. in Norman, D. and Draper, S. W. eds. User Centered System Design: New Perspectives on Human-Computer Interaction, Lawrence Erlbaum Associates, Hillsdale, NJ, 1986.Google Scholar
- Miyata, Y. and Norman, D. Psychological Issues in Support of Multiple Activities. in Norman, D. A. and Draper, S. W. eds. User Centered System Design: New Perspectives on Human-Computer Interaction, Lawrence Erlbaum Associates, Hillsdale, NJ, 1986, 265--284.Google ScholarCross Ref
- Monk, C., et al. The attentional costs of interrupting task performance at various stages. HFES Ann. Mtg, 2002.Google ScholarCross Ref
- Parasuraman, R., et al. A model for types and levels of human interaction with automation. IEEE Trans. on Systems, Man and Cybernetics, 30 (3). 286--297. Google ScholarDigital Library
- Picard, R. W., et al. Toward Machine Emotional Intelligence: Analysis of Affective Physiological State. IEEE Trans. Pattern Anal. Mach. Intell., 23 (10). 1175--1191. Google ScholarDigital Library
- Salvucci, D. D. and Bogunovich, P. Multitasking and monotasking: the effects of mental workload on deferred task interruptions. CHI'10, 2010. Google ScholarDigital Library
- Sasse, A., et al. Coordinating the Interruption of People in Human-Computer Interaction. INTERACT, 1999. Google ScholarDigital Library
- Schermerhorn, P. and Scheutz, M. Dynamic robot autonomy: investigating the effects of robot decision-making in a human-robot team task. ICMI, 2009. Google ScholarDigital Library
- Scheutz, M., et al. First steps toward natural human-like HRI. Auton. Robots, 22 (4). 411--423. Google ScholarDigital Library
- Solovey, E. T., et al. Using fNIRS Brain Sensing in Realistic HCI Settings: Experiments and Guidelines. UIST'09, 2009. Google ScholarDigital Library
- Starner, T., Schiele, B. and Pentland, A. Visual Contextual Awareness in Wearable Computing. ISWC, 1998. Google ScholarDigital Library
- Yuksel, B. F., et al. A novel brain-computer interface using a multi-touch surface. CHI'10, 2010. Google ScholarDigital Library
Index Terms
- Sensing cognitive multitasking for a brain-based adaptive user interface
Recommendations
Brainput: enhancing interactive systems with streaming fnirs brain input
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThis paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its ...
Toward a unified theory of the multitasking continuum: from concurrent performance to task switching, interruption, and resumption
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMultitasking in user behavior can be represented along a continuum in terms of the time spent on one task before switching to another. In this paper, we present a theory of behavior along the multitasking continuum, from concurrent tasks with rapid ...
Adaptive brain-computer interface
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing SystemsPassive brain-computer interfaces are designed to use brain activity as an additional input, allowing the adaptation of the interface in real time according to the user's mental state. While most current brain computer interface research (BCI) is ...
Comments