Elsevier

NeuroImage

Volume 21, Issue 1, January 2004, Pages 340-351
NeuroImage

Modality effects in verbal working memory: differential prefrontal and parietal responses to auditory and visual stimuli

https://doi.org/10.1016/j.neuroimage.2003.09.019Get rights and content

Abstract

The neural bases of verbal (nonspatial) working memory (VWM) have been primarily examined using visual stimuli. Few studies have investigated the neural bases of VWM using auditory stimuli, and fewer have explored modality differences in VWM. In this study, we used functional magnetic resonance imaging (fMRI) to examine similarities and differences between visual VWM (vis-VWM) and auditory VWM (aud-VWM) utilizing identical stimuli and a within-subjects design. Performance levels were similar in the two modalities and there was extensive overlap of activation bilaterally in the dorsolateral and ventrolateral prefrontal cortex (DLPFC and VLPFC), intraparietal sulcus, supramarginal gyrus and the basal ganglia. However, a direct statistical comparison revealed significant modality differences: the left posterior parietal cortex, primarily along the intraparietal sulcus, showed greater responses during vis-VWM whereas the left dorsolateral prefrontal cortex showed greater responses during aud-VWM. No such differences were observed in the right hemisphere. Other modality differences in VWM were also observed, but they were associated with relative decreases in activation. In particular, we detected bilateral suppression of the superior and middle temporal (auditory) cortex during vis-VWM, and of the occipital (visual) cortex during aud-VWM, thus suggesting that cross-modal inhibitory processes may help to provide preferential access to high-order heteromodal association areas. Taken together, our findings suggest that although similar prefrontal and parietal regions are involved in aud-VWM and vis-VWM, there are important modality differences in the way neural signals are generated, processed and routed during VWM.

Introduction

Working memory (WM), the ability to maintain and manipulate information during a short interval, has been widely used to investigate the basic operations underlying higher brain function Baddeley, 1997, Goldman-Rakic, 1999. The component processes involved in WM—encoding, rehearsal, storage and executive processes on the contents of stored memory—characterize key cognitive operations of the human brain. The highly influential Baddeley–Hitch model Baddeley, 1986, Baddeley and Hitch, 1974 has proposed a system that consists of a central executive, an attentional controller aided by two subsidiary systems—the phonological loop, capable of holding verbal information, and the visuospatial sketchpad, which performs a similar function for spatial information. This model has been recently extended to include a fourth component—an episodic buffer that provides temporary storage of information held in a multimodal code, which is capable of binding information from the subsidiary systems, and from long-term memory, into a unitary episodic representation (Baddeley, 2000). The two subsidiary systems are thought to form active storage buffers that combine information from sensory input and from the central executive. Although the Baddeley–Hitch model has been influential in our thinking about cognitive processes involved in WM, its neural basis remains poorly specified. In particular, very little information is currently available about the extent to which verbal working memory (VWM) processes are dependent on the modality of sensory input.

The aim of the present study is to examine similarities and differences in the processing of VWM for aurally and visually presented stimuli. Most of our understanding of the neural networks underlying VWM has been based on studies using visually presented stimuli (vis-VWM; see Smith and Jonides, 1998 for a review). In contrast, few studies have examined the neural basis of auditory VWM (aud-VWM), and fewer still have directly examined modality differences using similar tasks in a within-subjects design.

The distinction between aud-VWM and vis-VWM is an important one, with implications for both theoretical and experimental research into the neural processes underlying working memory. In particular, an examination of similarities and differences in processing of different types of stimuli can provide insights into the internal representations of stimuli in working memory. Lesion studies have provided some evidence for modality-specific deficits Basso et al., 1982, Shallice and Warrington, 1977, Warrington and Shallice, 1969, Warrington and Shallice, 1972, leading to the suggestion that short-term memory processing in the two modalities is mediated by separate streams that have different properties and capabilities (Penney, 1989). Behavioral research has pointed to differences in auditory and visual short-term memory processing Allport et al., 1972, Kroll et al., 1970, Penney, 1989 but the task conditions in which modality differences can be elicited are complex. While some researchers have suggested an auditory bias (i.e. better performance in auditory, compared to visual, VWM) (Penney, 1989), others have shown that an auditory bias can be easily created with simple perturbations to the task (Beaman, 2002). Although the precise nature of modality-specific differences is still being resolved in behavioral studies, these findings, taken together, nevertheless suggest that there are important differences in the internal representations of auditory and visual stimuli in VWM.

Despite its potential significance, to date only two brain imaging studies have examined differences between aud-VWM and vis-VWM. A PET study by Schumacher et al. (1996) showed common activation of bilateral SMA [Broadmann Area (BA) 6], bilateral superior (BA 7) and posterior parietal cortex (BA 40), and right cerebellum during 2-back aud-VWM and vis-VWM tasks. A direct comparison of the two modalities revealed few differences: auditory, compared to visual, VWM showed greater activation in Broca's area, whereas visual, compared to auditory, VWM did not reveal greater activation in any brain region. Schumacher et al. (1996) interpreted these findings as providing evidence for relatively minor quantitative differences in encoding, and further suggested that VWM processing is amodal. However, Ruchkin et al. (1997a) found significant differences in amplitude and timing of event-related potentials (ERPs) elicited during aud-VWM and vis-VWM. Left frontal negative waves indexing retention and rehearsal operations were triggered earlier and were more long lasting during aud-VWM. On the other hand, posterior regions showed larger evoked potentials for visual stimuli (see also Lang et al., 1992). Thus, although the PET study by Schumacher et al. (1996) suggests that VWM processing is primarily amodal, behavioral and ERP studies have provided evidence for modality-specific differences in VWM processing in both prefrontal and posterior brain regions. A third related study used PET to measure regional cerebral blood flow (rCBF) during performance of 1-back auditory and visual WM tasks, although the primary focus of the study was the examination of multimodal interference Klingberg, 1998, Klingberg et al., 1996. Compared to the two other studies Ruchkin et al., 1997a, Schumacher et al., 1996, Klingberg et al.'s studies focused on nonverbalizable, pitch and luminance based, stimuli in the two modalities, a design we explicitly avoided here. Interestingly, none of these studies has conducted a direct statistical comparison between effects in the two modalities, so that amodal effects, if any, were never rigorously examined.

To help resolve these discrepancies in the literature, we used functional magnetic resonance imaging (fMRI) to examine VWM processing, with identical experimental paradigms and similar auditory and visual stimuli. We examined the extent to which the working memory networks in the two modalities overlap, whether there are any gross (qualitative) modality differences and whether there are quantitative differences within brain areas that are activated by both modalities. One focus of our work was to resolve divergent findings between the two published PET (Schumacher et al., 1996) and ERP (Ruchkin et al., 1997a) studies. Towards this end, a close correspondence was maintained between auditory and visual stimuli, analogous to these PET and ERP studies so that differences between modalities could be attributable only to differences in the way auditory and visual codes are used in the context of VWM.

We hypothesized that the phonological loop would be engaged by both auditory and visual stimuli. Correspondingly, we predicted activation of similar regions of the posterior parietal cortex. We further hypothesized that it would be necessary for visual stimuli to be transformed to a phonological format, and then entered into a phonological store through articulatory rehearsal. Auditory stimuli, however, do not have to undergo such a transformation as they are more automatically encoded into a phonological format (Shallice and Vallar, 1990). We therefore predicted that brain regions involved in making the transformation to the phonological store would show greater activation during vis-VWM, compared to aud-VWM. Alternatively, it might also be possible that visual and auditory stimuli are handled by entirely distinct processing streams in the association cortex, in which case we would expect to find distinct, neuroanatomically segregated patterns of activation.

Section snippets

Subjects

Fourteen healthy right-handed subjects (7 females, 7 males; ages: 18–32 years) participated in this study after giving written informed consent. Handedness was assessed using the Edinburgh test (Oldfield, 1971). The human subjects committee at Stanford University School of Medicine approved all protocols used in this study.

Experimental procedure

Subjects performed two identical VWM experiments in the scanner: (i) an aud-VWM task, in which stimuli were presented aurally and (ii) a vis-VWM task, in which stimuli were

Accuracy

Performance levels were high in both aud-VWM and vis-VWM experiments. In the aud-VWM experiment, accuracy in the VWM and control conditions was 95.44 ± 1.23% (mean ± SEM) and 97.62 ± 0.76%, respectively. In the vis-VWM experiment, accuracy in the VWM and control conditions was 94.44 ± 1.01% and 97.32 ± 0.72%, respectively. ANOVA of accuracy data revealed that the interaction between Modality (Auditory, Visual) and Task (VWM, control) was not statistically significant [F(1, 13) = 0.33; P =

Discussion

The aim of our study was to examine similarities and differences in brain activation during auditory and visual verbal working memory using quantitative comparison of activations during the two tasks. We observed a significant overlap of activation during aud-VWM and vis-VWM in several brain regions including the dorsolateral and ventrolateral prefrontal cortex, intraparietal sulcus, supramarginal gyrus and the basal ganglia, in both hemispheres (Table 1; Fig. 1). However, a direct statistical

Acknowledgements

It is a pleasure to thank Prof. D.S. Ruchkin for helpful discussion, and the two anonymous reviewers for helpful suggestions. This research was supported by a grant from the Norris Foundation, the Swiss National Science Foundation and NIH grants HD40761 and MH62430.

References (66)

  • A. Mecklinger et al.

    Event-related potentials reveal topographical and temporal distinct neuronal activation patterns for spatial and object working memory

    Brain Res. Cogn. Brain Res.

    (1996)
  • V. Menon et al.

    Dissociating prefrontal and parietal cortex activation during arithmetic processing

    NeuroImage

    (2000)
  • R.C. Oldfield

    The assessment and analysis of handedness: the Edinburgh inventory

    Neuropsychologia

    (1971)
  • J.B. Poline et al.

    Combining spatial extent and peak intensity to test for activations in functional imaging

    NeuroImage

    (1997)
  • D.S. Ruchkin et al.

    Modality-specific processing streams in verbal working memory: evidence from spatio-temporal patterns of brain activity

    Brain Res. Cogn. Brain Res.

    (1997)
  • D.S. Ruchkin et al.

    Multiple visuospatial working memory buffers: evidence from spatiotemporal patterns of brain activity

    Neuropsychologia

    (1997)
  • E.H. Schumacher et al.

    PET evidence for an amodal verbal working memory system

    NeuroImage

    (1996)
  • T. Shallice et al.

    Auditory-verbal short-term memory impairment and conduction aphasia

    Brain Lang.

    (1977)
  • A.A. Stevens et al.

    Event-related fMRI of auditory and visual oddball tasks

    Magn. Reson. Imaging

    (2000)
  • T. Allison et al.

    Electrophysiological studies of human face perception. I: potentials generated in occipitotemporal cortex by face and non-face stimuli

    Cereb. Cortex

    (1999)
  • D.A. Allport et al.

    On the division of attention: a disproof of the single channel hypothesis

    Q. J. Exp. Psychol.

    (1972)
  • E. Awh et al.

    Dissociation of storage and rehearsal in verbal working memory

    Psychol. Sci.

    (1996)
  • A.D. Baddeley

    Working Memory

    (1986)
  • A. Baddeley

    Human Memory: Theory and Practice

    (1997)
  • A.D. Baddeley et al.

    Working memory

  • C.P. Beaman

    Inverting the modality effect in serial recall

    Q. J. Exp. Psychol., A

    (2002)
  • J.R. Booth et al.

    Modality independence of word comprehension

    Hum. Brain Mapp.

    (2002)
  • Brett, M., 2000. The MNI brain and the Talairach atlas....
  • D.E. Broadbent et al.

    Modality differences in relation to grouping in immediate recall

    Br. J. Psychol.

    (1980)
  • R.L. Buckner et al.

    Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming

    Brain

    (2000)
  • M.W. Chee et al.

    Auditory and visual word processing studied with fMRI

    Hum. Brain Mapp.

    (1999)
  • S. Cohen

    Conduction Aphasia

    (1992)
  • R.G. Crowder

    Mechanisms of auditory backward masking in the stimulus suffix effect

    Psychol. Rev.

    (1978)
  • Cited by (202)

    View all citing articles on Scopus
    View full text