fMRI-adaptation evidence of overlapping neural representations for objects related in function or manipulation

Abstract

Sensorimotor-based theories of semantic memory contend that semantic information about an object is represented in the neural substrate invoked when we perceive or interact with it. We used fMRI adaptation to test this prediction, measuring brain activation as participants read pairs of words. Pairs shared function (flashlight–lantern), shape (marble–grape), both (pencil–pen), were unrelated (saucer–needle), or were identical (drill–drill). We observed adaptation for pairs with both function and shape similarity in left premotor cortex. Further, degree of function similarity was correlated with adaptation in three regions: two in the left temporal lobe (left medial temporal lobe, left middle temporal gyrus), which has been hypothesized to play a role in mutimodal integration, and one in left superior frontal gyrus. We also found that degree of manipulation (i.e., action) and function similarity were both correlated with adaptation in two regions: left premotor cortex and left intraparietal sulcus (involved in guiding actions). Additional considerations suggest that the adaptation in these two regions was driven by manipulation similarity alone; thus, these results imply that manipulation information about objects is encoded in brain regions involved in performing or guiding actions. Unexpectedly, these same two regions showed increased activation (rather than adaptation) for objects similar in shape. Overall, we found evidence (in the form of adaptation) that objects that share semantic features have overlapping representations. Further, the particular regions of overlap provide support for the existence of both sensorimotor and amodal/multimodal representations.

Introduction

Many theories of semantic memory posit that the meanings of concepts can be described as patterns of activation that are distributed over semantic features (e.g., Allport, 1985, Barsalou, 1999, Tyler et al., 2000). A benefit of this kind of architecture is that relationships between concepts can be captured via overlapping representations (McRae et al., 1997, Masson, 1995). Some distributed theories of semantic memory further suggest that semantic information about an object is represented in the neural substrate that is invoked when we perceive and/or interact with that object. Specifically, these sensorimotor-based theories suggest that the meaning of a (concrete) concept is not distributed in an amodal, unitary semantic system, but instead different aspects of meaning are stored in physically distal networks, according to the modality in which the information was acquired (e.g., Warrington and McCarthy, 1987, Glenberg, 1997, Barsalou, 1999). Amodal theories (e.g., Caramazza and Shelton, 1998), on the other hand, do not posit that representations are sensorimotor-based and so need not predict that concepts are situated in modality specific cortices.

Regardless of whether it is posited that meanings are distributed over features in an amodal system or across multiple, modality specific systems, all distributed models of semantic memory assume that the representations of concepts that share features overlap. This means that activating a particular concept should also partially activate other concepts that share its features. The semantic priming effect, wherein identifying a target word is facilitated when it is preceded by a (conceptually) related prime word (e.g., Meyer and Schvaneveldt, 1971) can therefore be interpreted as support for distributed models. However, semantic priming studies typically use primes and targets from the same semantic category, and as many have pointed out (e.g., Kellenbach et al., 2000), category co-exemplars are usually related in multiple ways (e.g., crayon and pencil are both thin, oblong, used for marking paper, and grasped with the thumb and the second and third fingers). Therefore, although the results of these studies show that semantically related words partially activate each other, they cannot identify which of these features are responsible for the facilitation effect. Identifying the responsible features would help distinguish between sensorimotor and amodal distributed models of semantic memory; if concepts that are related via sensorimotor features partially activate each other, this would suggest that their meanings are represented (at least in part) as patterns of activation over sensorimotor-based attributes.

A handful of studies have explicitly manipulated the semantic relationship between primes and targets (e.g., Schreuder et al., 1984). Most of these studies explored whether semantic priming would be obtained when primes and targets have the same shape or function (function is defined here as the purpose for which an object is used, e.g., flashlight and lantern have the same function), and both shape and function priming have been observed¹ (Schreuder et al., 1984, Flores d'Arcais et al., 1985, Taylor and Heindel, 2001). Semantic priming has also been observed for objects that are manipulated (i.e., interacted with) similarly (e.g., piano and typewriter, which are both tapped with the fingers [Myung et al., 2006]). Results from visual world eyetracking studies, in which preferential fixations were observed on objects related (in function, shape, or manipulation) to a named object, are also consistent with the hypothesis that related objects have overlapping representations (Yee, et al., under review; Myung et al., 2006). Behavioral studies therefore suggest that objects with similar functions, manipulations or shapes do in fact have partially overlapping representations. If this is true, then this overlap should be instantiated at the neural level. The experiment reported here examines whether feature overlap is instantiated at the neural level by using an fMRI adaptation paradigm that permits identification of the particular brain regions in which the overlap is located.

The assumption underlying the fMRI adaptation paradigm is that repeated presentation of the same visual or verbal stimulus results in reduced fMRI signal levels in brain regions that process that stimulus, either because of neuronal “fatigue” (e.g., firing-rate adaptation) or because the initial activation of a stimulus' representation is less neurally efficient than subsequent activation (see Grill-Spector et al., 2006 for a review). In a typical fMRI adaptation experiment, stimuli are presented which are either identical (which produces an adaptation/reduced hemodynamic response) or completely different (producing a recovery response). However, it is possible to use stimuli pairs that are related, rather than identical, and several recent fMRI studies of semantically related word pairs have found less activation for related than unrelated word pairs, predominantly in temporal and/or inferior frontal cortices (Kotz et al., 2002, Rossell et al., 2003, Rissman et al., 2003, Copland et al., 2003, Giesbrecht et al., 2004, Matsumoto et al., 2005, Gold et al., 2005, Tivarus et al., 2006, Kuperberg et al., 2007, Bedny et al., 2008).

The adaptation paradigm can also be employed while varying the level of stimulus similarity (e.g., Kourtzi and Kanwisher, 2001, Epstein et al., 2003, Fang et al., 2005) in order to obtain a measure of neurally perceived difference: the greater the similarity, the greater the expected adaptation. Because of this sensitivity to similarity, the adaptation paradigm is a natural fit for examining whether, as predicted by distributed models, semantically related objects have overlapping representations; for if they do, when one object's representation is activated, semantically related objects should also be partially activated. Thus the presentation of semantically related objects should produce adaptation, with the greater the relatedness, the greater the adaptation. Two recent fMRI studies found just such an ordered effect of relatedness (but c.f. Raposo et al., 2006). In Wheatley et al. (2005) subjects silently read pairs of words which were either unrelated in meaning, semantically related, or identical. In left ventral temporal cortex and in anterior left inferior frontal gyrus, the greatest activity was found for unrelated word pairs, less for semantically related pairs, and least of all for identical pairs. Similarly, in Wible et al. (2006), subjects heard pairs of words that were either strongly connected (in that they shared many associates), weakly connected (sharing fewer associates), or unrelated. In bilateral posterior superior and middle temporal cortex, activation was greatest for unrelated pairs, less for weakly connected, and the least of all for strongly connected pairs.

The fact that many of these semantic adaptation studies found adaptation in the inferior frontal and middle temporal gyri in particular could be interpreted as support for amodal representations, as it has been suggested that these areas underlie amodal semantic processing (e.g., Postler et al., 2003). Crucially, however, none of these prior adaptation studies attempted to test specific sensorimotor features. Therefore, they do not address whether concepts' representations may also be comprised of sensorimotor features that are represented in sensorimotor areas.

Brain imaging studies that have addressed particular semantic features have demonstrated that when retrieving information about a specific attribute of an object (e.g., its color or shape), brain areas in, or just anterior to, those implicated in perceiving that attribute become active (at least for color, shape, and manipulation; for a review, see Thompson-Schill et al., 2006). While it is possible that these sensorimotor regions were activated as a consequence of tasks that directed attention to their corresponding features, there is some evidence to the contrary: even under conditions that do not require attending to manipulation information (i.e., under passive viewing) pictures of tools produce greater activation in left ventral premotor and left posterior parietal cortex than non-manipulable objects (Chao and Martin, 2000). The fact that motor regions were automatically activated when tools were viewed implies that these motor regions are part of tools' conceptual representations. This is consistent with the notion that concepts are represented in or near perceptual cortices. However, these studies do not address whether objects that share semantic features have overlapping (rather than nearby) representations.

If as sensorimotor-based theories suggest, the meaning of a (concrete) concept is distributed over the multiple brain regions involved in perceiving and/or interacting with the object, then conceiving of an object should automatically activate these sensorimotor regions. An object's shape and the way it is manipulated can both be directly linked to sensorimotor information. On the other hand, function, in the way we define it (the purpose for which an object is used, e.g., flashlight and lantern share the same function) does not by itself have a direct sensorimotor correlate. It is therefore possible that shape and manipulation similarity will be reflected in adaptation in regions devoted to processing object form and to motor programming respectively, but that function similarity will be instantiated in regions that have been hypothesized to represent more abstract, higher order relationships.

We use an adaptation paradigm to focus on two features in particular: function and shape, asking specifically whether pairs of objects with similar functions and/or shapes have overlapping neural representations (as indicated by eliciting a smaller neural response than unrelated pairs), and also whether the extent of any such overlap can be predicted by the degree of similarity. Because there is considerable variability across such pairs in manipulation similarity (e.g., things that are similar in both function and shape tend to be manipulated more similarly, while things similar in shape but not function tend not to be), we also examine whether objects that are manipulated similarly have overlapping representations.