Abstract
Although we live in a multisensory world, children’s memory has been usually studied concentrating on only one sensory modality at a time. In this study, we investigated how audiovisual encoding affects recognition memory. Children (n = 114) from three age groups (8, 10 and 12 years) memorized auditory or visual stimuli presented with a semantically congruent, incongruent or non-semantic stimulus in the other modality during encoding. Subsequent recognition memory performance was better for auditory or visual stimuli initially presented together with a semantically congruent stimulus in the other modality than for stimuli accompanied by a non-semantic stimulus in the other modality. This congruency effect was observed for pictures presented with sounds, for sounds presented with pictures, for spoken words presented with pictures and for written words presented with spoken words. The present results show that semantically congruent multisensory experiences during encoding can improve memory performance in school-aged children.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The sensitivity index (d′) was not used here since the results would remain exactly the same with this transformation and d′ would bring no additional information compared with the percentage correct. This is because there was a single recall phase consisting of the memorized items for all congruency categories and new items. Consequently, the false alarm rates (calculated on the basis of yes responses to new items) were equal for all congruency categories, and therefore, the d′ values would be in direct proportion to the percentage correct (hit) rates.
References
Bahrick L, Lickliter R (2012) The role of intersensory redundancy in early perceptual, cognitive and social development. In: Bremner A, Lewkowich D, Spence C (eds) Multisensory development. Oxford University Press, UK, pp 183–206
Brandwein A, Foxe J, Russo N, Altschuler T, Gomes H, Molholm S (2011) The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. Cereb Cortex 21:1042–1055
Calvert G, Spence C, Stein B (2004) The Handbook of Multisensory Processes. The MIT Press, London
Chen Y, Spence C (2010) When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked pictures. Cognition 114:389–404
Colavita F (1974) Human sensory dominance. Percept Psychophys 25:345–347
Constantidou F, Danos M, Nelson D, Baker S (2011) Effects of modality presentation of working memory in school-age children: evidence for the pictorial superiority hypothesis. Child Neuropsychol 17:173–196
Dekker TM, Mareschal D, Johnson MH, Sereno MI (2014) Picturing words? Sensorimotor cortex activation for printed words in child and adult readers. Brain Lang 139:58–67
Ernst M (2008) Multisensory integration: a late bloomer. Curr Biol 18:519–521
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169
Gori M, Del Viva M, Sandini G, Burr D (2008) Young children do not integrate visual and haptic information. Curr Biol 18:689–693
Heikkilä J, Alho K, Hyvönen H, Tiippana K (2015) Audiovisual semantic congruency during encoding enhances memory performance. Exp Psychol 62:123–130
Hillock A, Powers A, Wallace M (2011) Binding sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49:461–467
Jordan K, Baker J (2011) Multisensory information boosts numerical matching abilities in young children. Dev Sci 14:205–213
Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414
Lehmann S, Murray MM (2005) The role of multisensory memories in unisensory object discrimination. Cogn Brain Res 24:326–334
Lewkowicz DJ, Ghazanfar AA (2009) The emergence of multisensory systems through perceptual narrowing. Trends Cogn Sci 13:470–478
Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465
Moran Z, Bachman P, Pham P, Cho S, Cannon T, Shams L (2013) Multisensory encoding improves auditory recognition. Multisens Res 26:581–592
Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Gonzalez Andino S, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135
Nava E, Pavani F (2013) Changes in sensory dominance during childhood: converging evidence from the Colavita effect and the sound-induced flash illusion. Child Dev 84:604–616
Ngo M, Sinnet S, Soto-Faraco S, Spence C (2010) Repetition blindness and the Colavita effect. Neurosci Lett 480:186–190
Potter MC (1993) Very short-term conceptual memory. Mem Cogn 21:156–161
Potter MC (2012) Conceptual short term memory in perception and thought. Front Psychol 3:1–11
Schneider TR, Engel AK, Debener S (2008) Multisensory identification of natural objects in a two-way crossmodal priming paradigm. Exp Psychol 55:121–132
Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12:411–417
Thelen A, Talsma D, Murray M (2015) Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138:148–160
Torppa M, Lyytinen P, Erskine J, Eklund K, Lyytinen H (2010) Language development, literacy skills and predictive connections to reading in Finnish children with and without familiar risk for dyslexia. J Learn Disabil 43:308–321
Acknowledgments
This research was funded by a grant from the University of Helsinki. It is part of the research activities of CICERO Learning Network, Finland, www.cicero.fi. We are grateful to the pupils, guardians and teachers in Iivisniemi elementary school and Tähtiniitty elementary school, Espoo, Finland, where the research was conducted. We thank Professor Kimmo Alho for comments on the manuscript. The Multimodal Stimulus Set was developed by T.R. Schneider, S. Debener and A.K. Engel at the Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Germany.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Heikkilä, J., Tiippana, K. School-aged children can benefit from audiovisual semantic congruency during memory encoding. Exp Brain Res 234, 1199–1207 (2016). https://doi.org/10.1007/s00221-015-4341-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-015-4341-6