Recollection- and familiarity-based decisions reflect memory strength

Functional brain imaging studies have shown that medial temporal, parietal and prefrontal cortices are involved in recognition memory of prior episodes (Rugg and Wilding, 2000 ; Rugg and Yonelinas, 2003 ). The functional role that these regions play in memory retrieval, however, is still debated. Specifically, it is unclear whether recollection, the retrieval of specific information about a past experience, and familiarity, a sense that an event has been previously experienced (Tulving, 1985 ), are mediated by dissociated neural systems or separate strong memories from weak memories. Some studies suggest that separate cortical networks (Yonelinas et al., 2005 ) and differential activation in parietal cortex (Vilberg and Rugg, 2007 ) mediate these two distinct memory processes, whereas other studies suggest that recollection and familiarity reflect differences in the strength of a common memory trace (Donaldson, 1996 ; Dunn, 2004 ; Gonsalves et al., 2005 ; Squire et al., 2007 ; Wixted, 2007 ).

In most episodic retrieval studies written words have been used as stimuli (e.g., Cabeza et al., 2001 ; Henson et al., 1999 ) and the neural correlates of retrieving pictures from memory are not fully understood. We have recently conducted a series of event-related fMRI studies to investigate the neural mechanisms of retrieving pictorial information from long-term memory and the effects of visual similarity between old and new pictures. Subjects memorized paintings (Yago and Ishai, 2006 ) or unfamiliar Asian faces (Ishai and Yago, 2006 ) and 4 days later performed an old-new recognition memory task in the MR scanner. The old pictures were presented with new ones that were visually similar, somewhat similar or visually different from the old paintings. Consistent with our hypothesis, subjects were slower and less accurate to reject new, visually similar paintings. We found activation in a distributed cortical network that included face- and object-selective regions in the visual cortex, as well as prefrontal areas where the old paintings evoked stronger activation than the new ones, regardless of their visual similarity. Moreover, activation elicited by new paintings in parietal cortex was reduced with decreased similarity to the old items, whereas in the hippocampus and precuneus, stronger responses were evoked by the new, visually different paintings. We concluded that recognition memory is mediated by classification of new items as a match or a mismatch, based on their degree of visual similarity to old items (Ishai and Yago, 2006 ; Yago and Ishai, 2006 ). Our old-new task, however, did not address the issue of memory processes and the extent to which the observed behavioral and neural responses were due to recollection- or familiarity-based memory decisions.

To test whether recollection and familiarity judgments were influenced by the degree of visual similarity between old and new pictures, we used event-related fMRI with portraits, landscapes and abstract compositions by painters with a unique style (Figure 1 ). Based on our previous study, we predicted that subjects would correctly reject more new paintings that were visually different from the old ones, and would make more false alarms to new, visually similar paintings. Furthermore, we hypothesized that if recollection and familiarity judgments about complex pictures reflect strong memories and weak memories, respectively, correctly remembered items would be associated with stronger neural activation than known items.

Twenty-one normal, right-handed subjects (11 females, mean age 25 years) with normal or corrected to normal vision participated in the study. All subjects gave informed written consent for the procedure in accordance with protocols approved by the University Hospital of Zurich. The subjects, students from the University of Zurich, had no formal art education and reported visiting art museums once a year or less. Post-scan questionnaires revealed that all subjects were unfamiliar with the paintings and had not seen them prior to the experiment.

Stimuli were displayed using Presentation (www.neurobs.com, version 9.13) and were projected with a magnetically shielded LCD video projector onto a translucent screen placed at the feet of the subject. During the study phase, subjects performed a flower detection task on 20 portraits by Modigliani and Renoir, 20 landscapes by Pissarro and Van Gogh and 20 abstract compositions by Kandinsky and Miro (Figure 1 ). Each painting was presented for 3 s and subjects had to indicate whether it includes any flowers by pressing one of two buttons. In each of the three time series collected during the study phase, paintings (4 epochs of 15 s each) alternated with 15-s fixation epochs.

We then collected the anatomical images and after this 10-min scan, subjects performed a surprise memory test, in which the 60 old paintings were randomly presented with 60 new paintings. The new paintings were either visually similar or visually different from the old paintings. The degree of visual similarity between the new paintings and the old ones was assessed in a separate behavioral pilot (for details see Yago and Ishai, 2006 ). Each painting was presented for 3 s and subjects had to make a Remember/Know/New decision by pressing one of three buttons. Subjects were instructed to respond “Remember” if they were absolutely convinced that the picture was presented during the flower detection task; “Know” if the picture seemed familiar; and “New” if the picture was not presented in the flower detection task. In each of the three time series collected in the test phase, paintings (8 epochs of 15 s each) alternated with 15-s fixation epochs. In both study and test, each run included one category of paintings (i.e., portraits, landscapes or abstract compositions).

Data were collected using a 3T Philips Intera whole body MR scanner (Philips Medical Systems, Best, The Netherlands). Changes in blood-oxygenation level-dependent MRI signal were measured by using sensitivity encoded gradient-echo echoplanar sequence (SENSE, Pruessmann et al., 1999 ) with 35 axial slices, TR = 3 s, TE = 35 ms, flip angle = 82°, field of view = 220 mm, acquisition matrix = 80 × 80, reconstructed voxel size = 1.72 mm × 1.72 mm × 4 mm and SENSE acceleration factor R = 2.

High-resolution spoiled gradient recalled echo structural images were collected in the same session for all the subjects (180 axial slices, TR = 20 ms, TE = 2.3 ms, field of view = 220 mm, acquisition matrix = 224 × 224, reconstructed voxel size = 0.9 mm × 0.9 mm × 0.75 mm). These high-resolution anatomical images were collected after the study phase and provided detailed anatomical information for the region-of-interest (ROI) analysis and for 3D normalization to the Talairach and Tournoux (1998) atlas.

Accuracies and reaction times were computed for each subject, category of paintings (portraits, landscapes, abstract paintings) and response type (Yes/No during the flower detection task; Remember/Know/New during the memory retrieval test). ANOVA was used to compare the various conditions.

Functional MRI data were analyzed in BrainVoyager QX Version 1.8 (Brain Innovation, Maastricht, The Netherlands). All volumes were realigned to the first volume, corrected for motion artefacts and spatially smoothed using a 5-mm FWHM Gaussian filter. The main effects during the study and test were analyzed using multiple regression (Friston et al., 1995 ). Based on the contrast of paintings vs. fixation, a set of ROIs was anatomically defined for each subject with clusters that showed a significant effect (p < 0.0001, uncorrected). These regions included the inferior occipital gyrus (IOG), fusiform gyrus (FG), dorsal occipital cortex (DOC), intraparietal sulcus (IPS), inferior frontal gyrus (IFG), insula and the anterior cingulate cortex (ACC). The contrasts of Remember vs. Know and Remember vs. New further revealed significant activation in the precuneus and in two medial temporal lobe structures, the parahippocampal cortex (PHC) and the hippocampus. In each subject and each ROI, the mean parameter estimates were calculated separately for each response type (Yes/No during flower detection task; Remember/Know/New during memory test) and were used for between-subjects random-effects analyses.

Finally, we tested whether reaction times and fMRI activation during the study phase could predict subsequent behavioral and neural responses to the old paintings during the test phase. Thus, responses during the flower detection task were sorted based on subsequent Remember and Know judgments subjects made during the retrieval test.

Behavioral data. In this phase subjects were presented with portraits, landscapes and abstract paintings and indicated whether each painting contained any flowers. The behavioral data collected while subjects performed the task in the scanner are shown in Figure S1 . As most paintings did not include flowers, the proportion of No responses was higher than the proportion of Yes responses [t(124) = 10.9, p < 0.00001 for portraits, landscapes and abstract paintings]. Furthermore, subjects made significantly faster Yes responses than No responses [t(120) = 3.3, p < 0.001 for portraits, landscapes and abstract paintings]. Interestingly, subjects reported seeing flowers in more than 20% of the abstract paintings, which do not depict natural objects, but rather use purely visual forms of line, color and shape. Moreover, it took subjects significantly longer to decide whether these abstract compositions contained flowers. Thus, the differences in response latencies between Yes abstract responses and Yes responses to portraits and landscapes were significant [t(36) = 4.2, p < 0.001 and t(36) = 2.9, p < 0.01, respectively], as well as the difference in response latencies between No responses to abstract paintings and No responses to portraits [t(40) = 4.4, p < 0.001].

Imaging data. The main effect, namely responses evoked by all paintings as compared with the fixation baseline, revealed activation within a distributed cortical network that included multiple visual, parietal and prefrontal regions (Figure S2 ). Significant activation was found in the DOC (mean Talairach coordinates: −28, −82, 12; 33, −82, 12), IOG (−24, −80, −3; 21, −80, −5), FG (−28, −53, −12; 28, −53, −10) IPS (−32, −51, 47; 29, −51, 44), IFG (−44, 3, 33; 43, 4, 34) and ACC (8, 7, 53). Within all regions, activation evoked by Yes and No responses during the flower detection task was virtually identical, ruling out differential effects of attention during task performance.

We then tested whether behavioral and neural responses during study could predict subsequent memory performance during test. We found that shorter response latencies for paintings with flowers predicted subsequent Remember judgments during the memory test. In terms of the neural response, we found that Yes responses during the flower detection task resulted in similar activation in the FG during subsequent Remember and Know judgments (Figure S3 ).

Behavioral data. Ten minutes after performing the flower detection task, subjects performed a surprise memory task. In this test phase the paintings from the study phase were randomly presented with new paintings that were either visually similar or visually different from the old ones (see Figure 1 ) and subjects made Remember/Know/New decisions. The behavioral data collected while subjects performed the task in the scanner are shown in Figure 2 . The proportion of remember responses to the old items was significantly higher than both know and new responses [t(61) = 5.8, p < 0.000001 in both comparisons]. Consistent with our prediction, the proportion of false alarms, namely remember and know responses to new items, significantly decreased with decreased visual similarity between the old and the new items [remember new similar vs. remember new different, t(61) = 5.7, p < 0.000001; know new similar vs. know new different, t(61) = 10.5, p < 0.000001]. In terms of response latencies, know decisions took significantly longer than both remember and new responses [know vs. remember, t(187) = 9.0, p < 0.000001; know vs new, t(187) = 8.2, p < 0.000001 for all paintings]. Finally, consistent with our prediction, subjects responded significantly faster to the new, visually different than to the new, visually similar paintings [t(61) = 4.9, p < 0.00001].

Imaging data. The main effect, namely responses evoked by all paintings as compared with the fixation baseline, revealed activation within a distributed cortical network that included multiple visual, parietal and prefrontal regions (Figure 3 ). Significant activation was found in the DOC (mean Talairach coordinates: −27, −81, 11; 32, −81, 11), IOG (−29, −77, −1; 28, −77, −6), FG (−28, −55, −15; 33, −55, −14) IPS (−23, −54, 38; 27, −54, 39), IFG (−44, 3, 33; 43, 4, 34), insula (−42, 8, 5; 39, 8, 2) and ACC (2, 15, 40). Within the DOC, IOG, FG, IPS and IFG activation elicited by correctly remembered paintings was significantly higher than activation evoked by correctly know and new judgments (p < 0.000001 in both comparisons).

We then analyzed the activation evoked by correct and incorrect responses in all ROIs (Figure 4 ). Within the IPS, hits, namely correct responses to old items, evoked stronger activation than misses (p < 0.05), correct rejection of new paintings (p < 0.01); and false alarms (p < 0.001). To test our hypothesis about visual similarity, we compared correct responses to old paintings with correct responses to new, visually similar and new, visually different items (Figure 6 ). Consistent with our previous study (Yago and Ishai, 2006 ), we found that within the IPS, new, visually different paintings evoked less activation than new, visually similar ones (p < 0.01). To further understand the effect of visual similarity on memory decisions, we compared remember responses to old items (correct responses) with remember responses to new, visually similar items (false alarms). Within the FG, the difference between these responses was not significant, however in the IPS, remember responses to old items evoked significantly stronger activation than remember responses to new, visually similar items (p < 0.02), consistent with previous reports about the role of the IPS in mediating the old/new effect.

We then contrasted the memory responses. Interestingly, comparing correct remember judgments with correct know judgments revealed activation predominantly in the precuneus (Figure 5 ). Analysis of the BOLD response within this region indicated not only stronger responses to remembered items, but also a BOLD latency shift, with a significantly earlier peak for recollection (5.6 s) than familiarity [10.3 s, t(40) = 3.1, p < 0.004]. Contrasting correctly remembered pictures with correct rejections of new pictures further revealed a similar pattern of activation, albeit with a lower threshold. Activation in the precuneus was stronger and peaked significantly earlier during remember (5.7 s) than new [10.2 s, t(40) = 3.37, p < 0.002] judgments. Finally, contrasting correct know responses with correct rejection of new items revealed similar hemodynamic response profiles and the peak of the BOLD response (11.1 and 10.2 s, respectively) was not statistically significant [t(40) = 0.54, p = 0.59].

Comparing correct remember judgments with correct know judgments also revealed activation in the medial temporal lobe, namely in the PHC and the hippocampus (Figure 6 ). In both regions, correctly remembered paintings evoked significantly stronger activation than correctly known items [t(40) = 2.75, p < 0.009 in the PHC and t(40) = 2.47, p < 0.018 in the hippocampus]. The difference between remembered and new paintings, however, was not statistically significant [t(40) = 1.55, p = 0.13 in the PHC and t(40) = 1.16, p = 0.25 in the hippocampus].

We investigated the neural correlates that mediate recognition memory of portraits, landscapes and abstract paintings. During the study phase, subjects performed an attention demanding flower detection task. The behavioral data showed that response latencies were significantly shorter for the representational paintings, i.e., portraits and landscapes, in which familiar objects were clearly depicted. In contrast, the visual search for flowers in abstract compositions took much longer, in line with our previous studies of object indeterminacy in art paintings (Ishai et al., 2007 ). The flower detection task evoked activation in a distributed cortical network, in which paintings with and without flowers evoked virtually identical responses, ruling out differential effects of attention during encoding. Interestingly, shorter response latencies for paintings with flowers predicted subsequent Remember judgments during the memory test, suggesting that fast identification and detection of the flower targets facilitated later recollection of information about these paintings. Stronger activation in the FG for paintings with flowers predicted subsequent Remember and Know judgments, whereas reduced activation for paintings without flowers predicted subsequent Know judgments.

The surprise Remember/Know/New memory retrieval test revealed that most of the old paintings were correctly recognized and that responses to the new items depended on their visual similarity to the old ones. Consistent with our hypothesis, the proportion of false alarms, namely remember and know responses to new paintings, was significantly reduced with decreased visual similarity. Consistent with previous findings (e.g., Dewhurst and Conway, 1994 ), correct and incorrect know responses were associated with significantly longer latencies, suggesting that subjects hesitated before deciding that a picture looked familiar but they could not recollect additional information about its prior experience.

The recognition memory task evoked activation within a distributed cortical network that included similar visual, parietal and prefrontal regions to those activated during the study phase. Within DOC, IOG, FG, IPS and IFG, correctly remembered old items elicited stronger activation than both correctly known and new items. In some of these regions, greater activation for remembered than known words has been previously found (Henson et al., 1999 ), suggesting that recollection elicits enhanced activation than familiarity regardless of stimulus format. When remember judgments were directly contrasted with know judgments, a significant cluster of activation was found in the precuneus, a region implicated in many episodic memory retrieval (e.g., Fletcher et al., 1995 ; Shannon and Buckner, 2004 ; Yago and Ishai, 2006 ; Yonelinas et al., 2005 ) and visual imagery (Ishai et al., 2000a , 2002 ; Mechelli et al., 2004 ) studies. Within the precuneus, correctly remembered paintings not only elicited stronger activation than known items, but a latency shift of the BOLD response was observed, with a significantly earlier peak during recollection than familiarity-based judgments. Within the temporal lobe, activation in the hippocampus and PHC was stronger during recollection than during familiarity-based memory decisions. Our findings provide empirical evidence in support of a recent perspective according to which the Remember/Know procedure separates strong memories from weak memories (Squire et al., 2007 ; Wixted, 2007 ). It is of interest that within the PHC and the hippocampus, activation evoked by remembered paintings was not statistically significant from activation evoked by new items. Our fMRI findings are consistent with a previous report, in which activation in the hippocampus was recorded in epileptic patients using depth electrodes. Some hippocampal neurons increased their firing rate in response to old pictures, whereas other neurons signaled novelty by increased firing rate in response to new pictures (Rutishauser et al., 2006 ). Taken collectively, our findings that recollection evoked stronger activation than familiarity within multiple regions suggest that the memory decisions reflect memory strength and not independent memory processes.

Activation in parietal cortex during the memory test revealed stronger responses to the old, correctly remembered paintings than to the new items, consistent with previous ERP and fMRI studies (e.g., Curran and Cleary, 2003 ; Kahn et al., 2004 ; Wilding, 2000 ). Furthermore, activation within the IPS was reduced with decreased similarity between the new paintings and the old ones. The IPS, traditionally considered a region of the dorsal frontoparietal attention network, was implicated in many cognitive studies of attention, particularly in target detection tasks (Corbetta et al., 2000 ; Kincade et al., 2005 ; Shulman et al., 2001 ) and the segmentation of old from new items (Pollmann et al., 2003 ). Numerous recognition memory studies have further shown that posterior parietal cortex does not merely detect old items but, rather, mediates higher order cognitive processes associated with memory retrieval (Konishi et al., 2000 ; Shannon and Buckner, 2004 ; Wheeler and Buckner, 2003 ). Taken collectively, our previous (Ishai and Yago 2006 ; Yago and Ishai, 2006 ) and current findings suggest not only that the parietal cortex mediates the old/new effect, but also processes the degree of visual similarity between old and new items. These findings support the “mnemonic accumulator” hypothesis, according to which recognition memory decisions are based on the integration of sensory signals (Wagner et al., 2005 ).

Models of recognition memory assume that recollection and familiarity are independent processes during retrieval (Yonelinas, 2002 ). Evidence for such neuroanatomical dissociation came from studies in which subjects were instructed to rate their memory confidence (Yonelinas et al., 2005 ) or to indicate the amount of recollected information (Vilberg and Rugg, 2007 ). Future studies will determine the extent to which the dissociation of the neural correlates of recollection and familiarity could be generalized across various experimental paradigms. Although remember and know responses are exclusive, recollected items are also familiar ones. It is therefore highly likely that retrieval of mnemonic information about complex pictures is modulated by activation within a distributed neural system, where memory strength modulates the neural response. The redundant relationship between recollection and familiarity has been corroborated by neuropsychological (Knowlton, 1998 ) and electrophysiological (Yovel and Paller, 2004 ) studies. Our current findings suggest that the same cortical structures are activated during recollection- and familiarity-based judgments and are consistent with models of a continuum of mnemonic information on which the subject establishes a criterion (Donaldson, 1996 ; Dunn, 2004 ; Gonsalves et al., 2005 ). This criterion, as shown in our current and previous studies (Ishai and Yago, 2006 ; Yago and Ishai, 2006 ), depends on the degree of visual similarity between old and new items.

A distributed neural system for recognition memory is consistent with recent neuroanatomical findings in amnesic patients, indicating that the ability to recollect remote memories depends not only on the medial temporal lobe but on widely distributed neocortical areas in the occipital, parietal and prefrontal lobes (Bayley et al., 2005 ; Squire and Bayley, 2007 ). Furthermore, a distributed memory network is not only physiologically and ecologically plausible, but also confirms with previous fMRI studies, showing that the representation of objects and faces in the human brain is not modular, but rather distributed across a wide expanse of cortex (Haxby et al., 2001 ; Ishai et al., 1999 , 2000b ). Within this distributed neural system, the memory strength is manifested by differential BOLD responses during recollection and familiarity judgments. Future studies will determine the extent to which these two memory processes are manifested by differential patterns of effective connectivity among regions.

In summary, our results show that recollection- and familiarity-based memory decisions depend on the degree of visual similarity between old and new items. Furthermore, recognition memory of complex pictures is mediated by activation within a distributed cortical network, where remembered and known items evoke differential BOLD responses that reflect their memory strength.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

We thank Edward Wilding and Andrew Yonelinas for reading the manuscript. This study was supported by the Swiss National Science Foundation grant 3200B0-105278 and by the Swiss National Center for Competence in Research: Neural Plasticity and Repair.

See Figures S1 –S3 .