Modality effects in verbal working memory: differential prefrontal and parietal responses to auditory and visual stimuli
Introduction
Working memory (WM), the ability to maintain and manipulate information during a short interval, has been widely used to investigate the basic operations underlying higher brain function Baddeley, 1997, Goldman-Rakic, 1999. The component processes involved in WM—encoding, rehearsal, storage and executive processes on the contents of stored memory—characterize key cognitive operations of the human brain. The highly influential Baddeley–Hitch model Baddeley, 1986, Baddeley and Hitch, 1974 has proposed a system that consists of a central executive, an attentional controller aided by two subsidiary systems—the phonological loop, capable of holding verbal information, and the visuospatial sketchpad, which performs a similar function for spatial information. This model has been recently extended to include a fourth component—an episodic buffer that provides temporary storage of information held in a multimodal code, which is capable of binding information from the subsidiary systems, and from long-term memory, into a unitary episodic representation (Baddeley, 2000). The two subsidiary systems are thought to form active storage buffers that combine information from sensory input and from the central executive. Although the Baddeley–Hitch model has been influential in our thinking about cognitive processes involved in WM, its neural basis remains poorly specified. In particular, very little information is currently available about the extent to which verbal working memory (VWM) processes are dependent on the modality of sensory input.
The aim of the present study is to examine similarities and differences in the processing of VWM for aurally and visually presented stimuli. Most of our understanding of the neural networks underlying VWM has been based on studies using visually presented stimuli (vis-VWM; see Smith and Jonides, 1998 for a review). In contrast, few studies have examined the neural basis of auditory VWM (aud-VWM), and fewer still have directly examined modality differences using similar tasks in a within-subjects design.
The distinction between aud-VWM and vis-VWM is an important one, with implications for both theoretical and experimental research into the neural processes underlying working memory. In particular, an examination of similarities and differences in processing of different types of stimuli can provide insights into the internal representations of stimuli in working memory. Lesion studies have provided some evidence for modality-specific deficits Basso et al., 1982, Shallice and Warrington, 1977, Warrington and Shallice, 1969, Warrington and Shallice, 1972, leading to the suggestion that short-term memory processing in the two modalities is mediated by separate streams that have different properties and capabilities (Penney, 1989). Behavioral research has pointed to differences in auditory and visual short-term memory processing Allport et al., 1972, Kroll et al., 1970, Penney, 1989 but the task conditions in which modality differences can be elicited are complex. While some researchers have suggested an auditory bias (i.e. better performance in auditory, compared to visual, VWM) (Penney, 1989), others have shown that an auditory bias can be easily created with simple perturbations to the task (Beaman, 2002). Although the precise nature of modality-specific differences is still being resolved in behavioral studies, these findings, taken together, nevertheless suggest that there are important differences in the internal representations of auditory and visual stimuli in VWM.
Despite its potential significance, to date only two brain imaging studies have examined differences between aud-VWM and vis-VWM. A PET study by Schumacher et al. (1996) showed common activation of bilateral SMA [Broadmann Area (BA) 6], bilateral superior (BA 7) and posterior parietal cortex (BA 40), and right cerebellum during 2-back aud-VWM and vis-VWM tasks. A direct comparison of the two modalities revealed few differences: auditory, compared to visual, VWM showed greater activation in Broca's area, whereas visual, compared to auditory, VWM did not reveal greater activation in any brain region. Schumacher et al. (1996) interpreted these findings as providing evidence for relatively minor quantitative differences in encoding, and further suggested that VWM processing is amodal. However, Ruchkin et al. (1997a) found significant differences in amplitude and timing of event-related potentials (ERPs) elicited during aud-VWM and vis-VWM. Left frontal negative waves indexing retention and rehearsal operations were triggered earlier and were more long lasting during aud-VWM. On the other hand, posterior regions showed larger evoked potentials for visual stimuli (see also Lang et al., 1992). Thus, although the PET study by Schumacher et al. (1996) suggests that VWM processing is primarily amodal, behavioral and ERP studies have provided evidence for modality-specific differences in VWM processing in both prefrontal and posterior brain regions. A third related study used PET to measure regional cerebral blood flow (rCBF) during performance of 1-back auditory and visual WM tasks, although the primary focus of the study was the examination of multimodal interference Klingberg, 1998, Klingberg et al., 1996. Compared to the two other studies Ruchkin et al., 1997a, Schumacher et al., 1996, Klingberg et al.'s studies focused on nonverbalizable, pitch and luminance based, stimuli in the two modalities, a design we explicitly avoided here. Interestingly, none of these studies has conducted a direct statistical comparison between effects in the two modalities, so that amodal effects, if any, were never rigorously examined.
To help resolve these discrepancies in the literature, we used functional magnetic resonance imaging (fMRI) to examine VWM processing, with identical experimental paradigms and similar auditory and visual stimuli. We examined the extent to which the working memory networks in the two modalities overlap, whether there are any gross (qualitative) modality differences and whether there are quantitative differences within brain areas that are activated by both modalities. One focus of our work was to resolve divergent findings between the two published PET (Schumacher et al., 1996) and ERP (Ruchkin et al., 1997a) studies. Towards this end, a close correspondence was maintained between auditory and visual stimuli, analogous to these PET and ERP studies so that differences between modalities could be attributable only to differences in the way auditory and visual codes are used in the context of VWM.
We hypothesized that the phonological loop would be engaged by both auditory and visual stimuli. Correspondingly, we predicted activation of similar regions of the posterior parietal cortex. We further hypothesized that it would be necessary for visual stimuli to be transformed to a phonological format, and then entered into a phonological store through articulatory rehearsal. Auditory stimuli, however, do not have to undergo such a transformation as they are more automatically encoded into a phonological format (Shallice and Vallar, 1990). We therefore predicted that brain regions involved in making the transformation to the phonological store would show greater activation during vis-VWM, compared to aud-VWM. Alternatively, it might also be possible that visual and auditory stimuli are handled by entirely distinct processing streams in the association cortex, in which case we would expect to find distinct, neuroanatomically segregated patterns of activation.
Section snippets
Subjects
Fourteen healthy right-handed subjects (7 females, 7 males; ages: 18–32 years) participated in this study after giving written informed consent. Handedness was assessed using the Edinburgh test (Oldfield, 1971). The human subjects committee at Stanford University School of Medicine approved all protocols used in this study.
Experimental procedure
Subjects performed two identical VWM experiments in the scanner: (i) an aud-VWM task, in which stimuli were presented aurally and (ii) a vis-VWM task, in which stimuli were
Accuracy
Performance levels were high in both aud-VWM and vis-VWM experiments. In the aud-VWM experiment, accuracy in the VWM and control conditions was 95.44 ± 1.23% (mean ± SEM) and 97.62 ± 0.76%, respectively. In the vis-VWM experiment, accuracy in the VWM and control conditions was 94.44 ± 1.01% and 97.32 ± 0.72%, respectively. ANOVA of accuracy data revealed that the interaction between Modality (Auditory, Visual) and Task (VWM, control) was not statistically significant [F(1, 13) = 0.33; P =
Discussion
The aim of our study was to examine similarities and differences in brain activation during auditory and visual verbal working memory using quantitative comparison of activations during the two tasks. We observed a significant overlap of activation during aud-VWM and vis-VWM in several brain regions including the dorsolateral and ventrolateral prefrontal cortex, intraparietal sulcus, supramarginal gyrus and the basal ganglia, in both hemispheres (Table 1; Fig. 1). However, a direct statistical
Acknowledgements
It is a pleasure to thank Prof. D.S. Ruchkin for helpful discussion, and the two anonymous reviewers for helpful suggestions. This research was supported by a grant from the Norris Foundation, the Swiss National Science Foundation and NIH grants HD40761 and MH62430.
References (66)
The episodic buffer: a new component of working memory
Trends Cogn. Sci.
(2000)- et al.
Left hemisphere damage and selective impairment of auditory verbal short-term memory. A case study
Neuropsychologia
(1982) - et al.
A parametric study of prefrontal cortex involvement in human working memory
NeuroImage
(1997) - et al.
Working memory in another dimension: functional imaging of human olfactory working memory
NeuroImage
(2001) - et al.
Multisubject fMRI studies and conjunction analyses
NeuroImage
(1999) Circuitry of the frontal association cortex and its relevance to dementia
Arch. Gerontol. Geriatr.
(1987)The physiological approach: functional architecture of working memory and disordered cognition in schizophrenia
Biol. Psychiatry
(1999)- et al.
Domain-specific distribution of working memory processes along human prefrontal and parietal cortices: a functional magnetic resonance imaging study
Neurosci. Lett.
(2001) - et al.
Recoding, storage, rehearsal and grouping in verbal short-term memory: an fMRI study
Neuropsychologia
(2000) - et al.
Cortical DC potential shifts accompanying auditory and visual short-term memory
Electroencephalogr. Clin. Neurophysiol.
(1992)
Event-related potentials reveal topographical and temporal distinct neuronal activation patterns for spatial and object working memory
Brain Res. Cogn. Brain Res.
Dissociating prefrontal and parietal cortex activation during arithmetic processing
NeuroImage
The assessment and analysis of handedness: the Edinburgh inventory
Neuropsychologia
Combining spatial extent and peak intensity to test for activations in functional imaging
NeuroImage
Modality-specific processing streams in verbal working memory: evidence from spatio-temporal patterns of brain activity
Brain Res. Cogn. Brain Res.
Multiple visuospatial working memory buffers: evidence from spatiotemporal patterns of brain activity
Neuropsychologia
PET evidence for an amodal verbal working memory system
NeuroImage
Auditory-verbal short-term memory impairment and conduction aphasia
Brain Lang.
Event-related fMRI of auditory and visual oddball tasks
Magn. Reson. Imaging
Electrophysiological studies of human face perception. I: potentials generated in occipitotemporal cortex by face and non-face stimuli
Cereb. Cortex
On the division of attention: a disproof of the single channel hypothesis
Q. J. Exp. Psychol.
Dissociation of storage and rehearsal in verbal working memory
Psychol. Sci.
Working Memory
Human Memory: Theory and Practice
Working memory
Inverting the modality effect in serial recall
Q. J. Exp. Psychol., A
Modality independence of word comprehension
Hum. Brain Mapp.
Modality differences in relation to grouping in immediate recall
Br. J. Psychol.
Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming
Brain
Auditory and visual word processing studied with fMRI
Hum. Brain Mapp.
Conduction Aphasia
Mechanisms of auditory backward masking in the stimulus suffix effect
Psychol. Rev.
Cited by (202)
The role of visual working memory in capacity-limited cross-modal ensemble coding
2024, NeuropsychologiaWorking memory performance in disordered gambling and gaming: A systematic review
2023, Comprehensive PsychiatrySimilarities and differences in working memory and neurometabolism of obsessive-compulsive disorder and major depressive disorder
2022, Journal of Affective DisordersDoes adding video and subtitles to an audio lesson facilitate its comprehension?
2022, Learning and Instruction