Abstract
Decoding movements from the human cortex has been a topic of great interest for controlling an artificial limb in non-human primates and severely paralyzed people. Here we investigate feasibility of decoding gestures from the sensorimotor cortex in humans, using 7 T fMRI. Twelve healthy volunteers performed four hand gestures from the American Sign Language Alphabet. These gestures were performed in a rapid event related design used to establish the classifier and a slow event-related design, used to test the classifier. Single trial patterns were classified using a pattern-correlation classifier. The four hand gestures could be classified with an average accuracy of 63 % (range 35–95 %), which was significantly above chance (25 %). The hand region was, as expected, the most active region, and the optimal volume for classification was on average about 200 voxels, although this varied considerably across individuals. Importantly, classification accuracy correlated significantly with consistency of gesture execution. The results of our study demonstrate that decoding gestures from the hand region of the sensorimotor cortex using 7 T fMRI can reach very high accuracy, provided that gestures are executed in a consistent manner. Our results further indicate that the neuronal representation of hand gestures is robust and highly reproducible. Given that the most active foci were located in the hand region, and that 7 T fMRI has been shown to agree with electrocorticography, our results suggest that this confined region could serve to decode sign language gestures for intracranial brain–computer interfacing using surface grids.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amunts K, Schlaug G, Jäncke L et al (1997) Motor cortex and hand motor skills: structural compliance in the human brain. Hum Brain Mapp 5:206–215. doi:10.1002/(SICI)1097-0193
Andersson P, Pluim JPW, Siero JCW et al (2011) Real-time decoding of brain responses to visuospatial attention using 7 T fMRI. PLoS One 6:e27638. doi:10.1371/journal.pone.0027638
Andersson P, Pluim JPW, Viergever MA, Ramsey NF (2012a) Navigation of a telepresence robot via covert visuospatial attention and real-time fMRI. Brain Topogr. doi:10.1007/s10548-012-0252-z
Andersson P, Ramsey NF, Raemaekers M et al (2012b) Real-time decoding of the direction of covert visuospatial attention. J Neural Eng 9:045004. doi:10.1088/1741-2560/9/4/045004
Andersson P, Ramsey NF, Viergever MA, Pluim JPW (2013) 7 T fMRI reveals feasibility of covert visual attention-based brain–computer interfacing with signals obtained solely from cortical grey matter accessible by subdural surface electrodes. Clin Neurophysiol. doi:10.1016/j.clinph.2013.05.009
Beisteiner R, Windischberger C, Lanzenberger R et al (2001) Finger somatotopy in human motor cortex. Neuroimage 13:1016–1026. doi:10.1006/nimg.2000.0737
Ben Hamed S, Schieber MH, Pouget A (2007) Decoding M1 neurons during multiple finger movements. J Neurophysiol 98:327–333. doi:10.1152/jn.0 0760.2006
Buracas GT, Boynton GM (2002) Efficient design of event-related fMRI experiments using M-sequences. Neuroimage 16:801–813
Chestek CA, Gilja V, Blabe CH et al (2013) Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas. J Neural Eng 10:026002. doi:10.1088/1741-2560/10/2/026002
Collinger JL, Wodlinger B, Downey JE et al (2012) High-performance neuroprosthetic control by an individual with tetraplegia. Lancet. doi:10.1016/S0140-6736(12)61816-9
Dechent P, Frahm J (2003) Functional somatotopy of finger representations in human primary motor cortex. Hum Brain Mapp 18:272–283. doi:10.1002/hbm.10084
Donoghue JP (2002) Connecting cortex to machines: recent advances in brain interfaces. Nat Neurosci 5(Suppl):1085–1088. doi:10.1038/nn947
Georgopoulos AP, Schwartz AB, Kettner RE (1986) Neuronal population coding of movement direction. Science 233:1416–1419. doi:10.1126/science.3749885
Glover GH (1999) Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage 9:416–429
Guenther FH, Brumberg JS, Wright EJ et al (2009) A wireless brain–machine interface for real-time speech synthesis. PLoS One 4:e8218. doi:10.1371/journal.pone.0008218
Hänggi J, Koeneke S, Bezzola L, Jäncke L (2010) Structural neuroplasticity in the sensorimotor network of professional female ballet dancers. Hum Brain Mapp 31:1196–1206. doi:10.1002/hbm.20928
Haxby JVJ, Gobbini MIM, Furey MLM et al (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430. doi:10.1126/science.1063736
Hermes D, Vansteensel MJ, Albers AM et al (2011) Functional MRI-based identification of brain areas involved in motor imagery for implantable brain–computer interfaces. J Neural Eng 8:025007. doi:10.1088/1741-2560/8/2/025007
Hermes D, Miller KJ, Vansteensel MJ et al (2012a) Neurophysiologic correlates of fMRI in human motor cortex. Hum Brain Mapp 33:1689–1699. doi:10.1002/hbm.21314
Hermes D, Siero JCW, Aarnoutse EJ et al (2012b) Dissociation between neuronal activity in sensorimotor cortex and hand movement revealed as a function of movement rate. J Neurosci 32:9736–9744. doi:10.1523/JNEUROSCI.0357-12.2012
Hochberg LR, Bacher D, Jarosiewicz B et al (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485:372–375. doi:10.1038/nature11076
Hotz-Boendermaker S, Funk M, Summers P et al (2008) Preservation of motor programs in paraplegics as demonstrated by attempted and imagined foot movements. Neuroimage 39:383–394. doi:10.1016/j.neuroimage.2007.07.065
Indovina I, Sanes JN (2001) Combined visual attention and finger movement effects on human brain representations. Exp Brain Res 140:265–279
Jäncke L, Shah NJ, Peters M (2000) Cortical activations in primary and secondary motor areas for complex bimanual movements in professional pianists. Brain Res Cogn Brain Res 10:177–183
Jansma JM, de Zwart JA, van Gelderen P et al (2013) In vivo evaluation of the effect of stimulus distribution on FIR statistical efficiency in event-related fMRI. J Neurosci Methods. doi:10.1016/j.jneumeth.2013.02.017
Jurkiewicz MT, Mikulis DJ, Fehlings MG, Verrier MC (2010) Sensorimotor cortical activation in patients with cervical spinal cord injury with persisting paralysis. Neurorehabil Neural Repair 24:136–140. doi:10.1177/1545968309347680
Karni A, Meyer G, Jezzard P et al (1995) Functional MRI evidence for adult motor cortex plasticity during motor skill learning. Nature 377:155–158. doi:10.1038/377155a0
Kubánek J, Miller KJ, Ojemann JG et al (2009) Decoding flexion of individual fingers using electrocorticographic signals in humans. J Neural Eng 6:066001. doi:10.1088/1741-2560/6/6/066001
Lachaux J-PJ, Fonlupt PP, Kahane PP et al (2007) Relationship between task-related gamma oscillations and BOLD signal: new insights from combined fMRI and intracranial EEG. Hum Brain Mapp 28:1368–1375. doi:10.1002/hbm.20352
Leuthardt EC, Schalk G, Wolpaw JR et al (2004) A brain–computer interface using electrocorticographic signals in humans. J Neural Eng 1:63–71. doi:10.1088/1741-2560/1/2/001
Miller KJ, Zanos S, Fetz EE et al (2009) Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans. J Neurosci 29:3132–3137. doi:10.1523/JNEUROSCI.5506-08.2009
Misaki M, Kim Y, Bandettini PA, Kriegeskorte N (2010) Comparison of multivariate classifiers and response normalizations for pattern-information fMRI. Neuroimage 53:16. doi:10.1016/j.neuroimage.2010.05.051
Mitchell TM, Hutchinson R, Niculescu RS (2004) Learning to decode cognitive states from brain images. Mach Learn 57:145–175
Monfils M-H, Plautz EJ, Kleim JA (2005) In search of the motor engram: motor map plasticity as a mechanism for encoding motor experience. Neuroscientist 11:471–483. doi:10.1177/1073858405278015
Ojemann GA, Ojemann J, Ramsey NF (2013) Relation between functional magnetic resonance imaging (fMRI) and single neuron, local field potential (LFP) and electrocorticography (ECoG) activity in human cortex. Front Hum Neurosci 7:34. doi:10.3389/fnhum.2013.00034
Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97–113
Pereira F, Mitchell T, Botvinick M (2009) Machine learning classifiers and fMRI: a tutorial overview. Neuroimage 45:S199–209. doi:10.1016/j.neuroimage.2008.11.007
Ramsey NF, Jansma JM, Jager G et al (2004) Neurophysiological factors in human information processing capacity. Brain 127:517–525. doi:10.1093/brain/awh060
Ramsey NF, van de Heuvel MP, Kho KH, Leijten FSS (2006) Towards human BCI applications based on cognitive brain systems: an investigation of neural signals recorded from the dorsolateral prefrontal cortex. IEEE Trans Neural Syst Rehabil Eng 14:214–217. doi:10.1109/TNSRE.2006.875582
Remy P, Zilbovicius M, Leroy-Willig A et al (1994) Movement- and task-related activations of motor cortical areas: a positron emission tomographic study. Ann Neurol 36:19–26. doi:10.1002/ana.410360107
Sanchez-Panchuelo RM, Francis S, Bowtell R, Schluppeck D (2010) Mapping human somatosensory cortex in individual subjects with 7 T functional MRI. J Neurophysiol 103:2544–2556. doi:10.1152/jn.0 1017.2009
Sanchez-Panchuelo RMR, Besle JJ, Beckett AA et al (2012) Within-digit functional parcellation of brodmann areas of the human primary somatosensory cortex using functional magnetic resonance imaging at 7 T. J Neurosci 32:15815–15822. doi:10.1523/JNEUROSCI.2501-12.2012
Sanes JN, Donoghue JP, Thangaraj V et al (1995) Shared neural substrates controlling hand movements in human motor cortex. Science 268:1775–1777
Schieber MH (2001) Constraints on somatotopic organization in the primary motor cortex. J Neurophysiol 86:2125–2143
Schieber MH (2002) Motor cortex and the distributed anatomy of finger movements. Adv Exp Med Biol 508:411–416
Siero JCW, Petridou N, Hoogduin H et al (2011) Cortical depth-dependent temporal dynamics of the BOLD response in the human brain. J cereb Blood Flow Metab 31:1999–2008. doi:10.1038/jcbfm.2011.57
Siero JC, Hermes D, Hoogduin H et al (2013) BOLD consistently matches electrophysiology in human sensorimotor cortex at increasing movement rates: a combined 7 T fMRI and ECoG study on neurovascular coupling. J cereb Blood Flow Metab. doi:10.1038/jcbfm.2013.97
Sorger B, Reithler J, Dahmen B, Goebel R (2012) A real-time fMRI-based spelling device immediately enabling robust motor-independent communication. Curr Biol 22:1333–1338. doi:10.1016/j.cub.2012.05.022
Turner RR (2002) How much cortex can a vein drain? Downstream dilution of activation-related cerebral blood oxygenation changes. Neuroimage 16:6. doi:10.1006/nimg 2002.1082
van der Zwaag W, Francis S, Head K et al (2009) fMRI at 1.5, 3 and 7 T: characterising BOLD signal changes. Neuroimage 47:10. doi:10.1016/j.neuroimage.2009.05.015
Vansteensel MJ, Hermes D, Aarnoutse EJ et al (2010) Brain–computer interfacing based on cognitive control. Ann Neurol 67:809–816. doi:10.1002/ana.21985
Wang W, Collinger JL, Degenhart AD et al (2013) An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 8:e55344. doi:10.1371/journal.pone.0055344.s010
Wennberg R, Quesney F, Olivier A, Rasmussen T (1998) Electrocorticography and outcome in frontal lobe epilepsy. Electroencephalogr Clin Neurophysiol 106:357–368
Yanagisawa T, Hirata M, Saitoh Y et al (2012) Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann Neurol 71:353–361. doi:10.1002/ana.22613
Zhang D, Song H, Xu R et al (2013) Toward a minimally invasive brain–computer interface using a single subdural channel: a visual speller study. Neuroimage 71C:30–41. doi:10.1016/j.neuroimage.2012.12.069
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bleichner, M.G., Jansma, J.M., Sellmeijer, J. et al. Give Me a Sign: Decoding Complex Coordinated Hand Movements Using High-Field fMRI. Brain Topogr 27, 248–257 (2014). https://doi.org/10.1007/s10548-013-0322-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10548-013-0322-x