It is less than you expected: The feedback-related negativity reflects violations of reward magnitude expectations
Research highlights
ā¶ The FRN codes deviations of reward magnitude expectations. ā¶ The FRN in response to no reward is larger for higher potential rewards. ā¶ This effect does not depend on insight into reward probability. ā¶ In learners the effect correlates with performance accuracy/reward expectation.
Introduction
Learning the association between behavioural responses and their consequences is a prerequisite for successful adaptation to varying environmental demands. In monkeys, single dopaminergic neurons in the midbrain have been shown to code positive and negative errors in reward prediction (Schultz et al., 1997, Schultz and Dickinson, 2000, Tobler et al., 2005). In estimating the overall reward associated with specific responses or events, the expected value is relevant which takes into account valence, magnitude and probability of an outcome. In humans, it has been shown that these different components of expected value are coded in a distributed set of brain regions along the mesolimbic pathway (Knutson, Taylor, Kaufman, Peterson, & Glover, 2005). Apart from the basal ganglia, which are critically involved in learning from monetary feedback (Bellebaum, Koch, Schwarz, & Daum, 2008), a key role in this context is assigned to the anterior cingulate cortex (ACC) which integrates information about outcome valence, probability and magnitude (Knutson & Cooper, 2005).
Evidence obtained from event-related potentials in human subjects strongly suggests that the ACC is implicated in performance monitoring and action selection. Performance errors in human subjects are accompanied by a negative deflection in the event-related potential (ERP), the error related negativity (ERN) or Ne, which peaks at about 100Ā ms after the motor response and is generated in the ACC (Dehaene et al., 1994, Falkenstein et al., 1990, Gehring et al., 1993). A similar component is elicited by performance feedback in situations when stimulus response associations are unknown to the subject. This component is termed feedback-related negativity (FRN) or feedback ERN which is also generated in the ACC, as revealed by source localization techniques (Bellebaum and Daum, 2008, Gehring and Willoughby, 2002). The FRN is typically more pronounced for negative compared to positive feedback (Miltner et al., 1997, Nieuwenhuis et al., 2004) and for active learning compared to learning by observation (Bellebaum, Kobza, Thiele, & Daum, 2010). Evidence from functional neuroimaging supports the results of ERP source localisation. The dorsal ACC is activated when subjects commit performance errors and when external signals give error feedback (Holroyd, Nieuwenhuis, et al., 2004).
Holroyd and Coles (2002) proposed a reinforcement-learning (RL) theory of the FRN suggesting that ACC activity as reflected by the FRN mirrors the activity of the midbrain dopamine (DA) system. According to this account, reward prediction errors should be coded in the amplitude of the FRN. Despite the finding that a significant FRN is only observed in response to unexpected negative outcomes (Holroyd, Nieuwenhuis, Yeung, & Cohen, 2003), early studies did not find evidence for prediction error coding in the ACC. Similarly sized FRN amplitudes for different outcome probabilities were reported, suggesting that the FRN reflects a binary system of performance monitoring distinguishing between outcomes which are better or worse than expected, without coding the magnitude of the prediction error (Hajcak, Holroyd, Moser, & Simons, 2005). One shortcoming of these early studies, however, relates to the fact that reward expectations were not taken into account. As the prediction error is conceptualized as the deviation of the actual from the expected outcome, assessing reward expectations of the subjects is of central importance. When subjectsā expectations were accounted for in a follow-up study, an increase in FRN amplitude with increasing prediction error magnitude was indeed observed (Hajcak, Moser, Holroyd, & Simons, 2007). Again, prediction errors were manipulated by varying reward probability. Several other studies corroborated this finding (Bellebaum and Daum, 2008, Eppinger et al., 2008, Holroyd et al., 2009), supporting the RL theory of the FRN.
As outlined above, the expected reward value is a function of reward probability and magnitude. If the ACC coded the deviation of actually obtained from expected outcomes, the FRN would be expected to be sensitive to deviations from expected reward magnitude. There is as yet no clear evidence for such a modulation. When subjects could choose between risky and non-risky, i.e. high or low magnitude, outcomes without knowing whether their choices resulted in a gain or loss, the pattern of outcome-related FRN amplitudes reflected binary processing. Losses were associated with larger amplitude FRNs, but no differences between large and small losses (or gains) were observed (Yeung & Sanfey, 2004). The authors concluded that the monitoring system located in the ACC is involved in a fast and coarse evaluation of ongoing events leading to a simple distinction between good and bad outcomes. Similar results emerged in other studies, corroborating the notion that the ACC evaluates outcomes in a binary manner (Hajcak et al., 2006, Holroyd et al., 2004a, Holroyd et al., 2006, Sato et al., 2005). More recent studies provided first hints that reward magnitude information might be processed in the FRN time window (Goyer et al., 2008, Wu and Zhou, 2009), but these findings could not clearly be assigned to the processing of reward magnitude per se or to prediction errors. Moreover, FRN has been reported to show sensitivity to magnitude in correspondence to risk-avoidant behavior (Polezzi, Sartori, Rumiati, Vidotto, & Daum, 2009).
The present study was motivated by the assumption that any violation of reward expectations is reflected in ACC activity and thus in FRN amplitude, especially for negative outcomes. We hypothesized that higher expected monetary rewards should yield larger FRN amplitudes in response to non-rewarding feedback. A task previously applied to assess the effect of reward probabilities on feedback processing was modified to induce expectations on reward magnitude (Bellebaum and Daum, 2008, Bellebaum et al., 2010). In contrast to previous studies examining the effect of reward magnitude, subjects could maximize the amount of money they earned by learning an explicit rule determining reward probability. Recent evidence suggests that FRN amplitude modulations are stronger, if action outcome contingencies can be learned (Holroyd et al., 2009). A further aim of the study was to add new evidence on the coding of outcome magnitude and valence in the P300 ERP component. Previous work on this issue has provided inconsistent results. Some studies suggested that the P300 codes reward magnitude information without being sensitive to outcome valence (Sato et al., 2005, Yeung and Sanfey, 2004). Other studies reported larger P300 amplitudes for gains than for losses (Hajcak et al., 2007) and very recent work suggests that the P300 codes both outcome valence and magnitude (Wu & Zhou, 2009).
Section snippets
Subjects
Twenty healthy subjects with a mean age of 23.6 years (SDĀ =Ā 4.2) participated in this study (12 women). All subjects were students at the Ruhr University of Bochum, Germany. They were all right-handed and had normal or corrected-to-normal vision. The study was approved by the Ethics Committee of the Faculty of Medicine of the Ruhr-University of Bochum and all participants gave written informed consent.
The learning task
A variant of a previously administered learning task was used (Bellebaum and Daum, 2008,
Accuracy
Fig. 2A shows the average number of correct responses for all 20 subjects in six blocks of trials for the three reward magnitudes. Repeated-measures ANOVA with the factors BLOCK (1ā6) and RM (5 cent, 20 cent and 50 cent) yielded a significant linear increase in the number of correct responses over the six blocks (linear trend: F(1,19)Ā =Ā 5.740; pĀ =Ā .027). Neither the main effect of RM nor the RM x BLOCK interaction reached significance (both pĀ >Ā .709).
Reaction times
Mean reaction times for the three potential reward
Discussion
According to the RL theory, the FRN reflects the activity of a reinforcement-learning system and codes negative errors in reward prediction (Holroyd & Coles, 2002), mirroring the activity of single dopaminergic neurons in the midbrain (Schultz et al., 1997). The results of earlier studies suggested that the FRN codes outcomes in a binary manner, i.e. a distinction between good and bad or better and worse than expected outcomes, without reflecting the degree of deviation from expectancy (Hajcak
Acknowledgement
We thank the Ministry of Innovation, Science, Research and Technology of the federal state of Nordrhein-Westfalen, Germany, for supporting this research within the young researcher programme (Ministerium fĆ¼r Innovation, Wissenschaft, Forschung und Technologie (MIWFT) des Landes Nordrhein-Westfalen).
References (45)
- et al.
Reward expectation modulates feedback-related negativity and EEG spectra
Neuroimage
(2007) - et al.
A core system for the implementation of task sets
Neuron
(2006) - et al.
Better or worse than expected? Aging, learning, and the ERN
Neuropsychologia
(2008) - et al.
A new method for off-line removal of ocular artifact
Electroencephalography and Clinical Neurophysiology
(1983) - et al.
The feedback-related negativity reflects the binary evaluation of good versus bad outcomes
Biological Psychology
(2006) - et al.
The good, the bad and the neutral: Electrophysiological responses to feedback stimuli
Brain Research
(2006) - et al.
Neural correlates of rapid reversal learning in a simple model of human social interaction
Neuroimage
(2003) - et al.
Reinforcement-related brain potentials from medial frontal cortex: Origins and functional significance
Neuroscience and Biobehavioral Reviews
(2004) - et al.
The P300 and reward valence, magnitude, and expectancy in outcome evaluation
Brain Research
(2009) - et al.
Anterior cingulate error-related activity is modulated by predicted reward
European Journal of Neuroscience
(2005)
Learning-related changes in reward expectancy are reflected in the feedback-related negativity
European Journal of Neuroscience
Focal basal ganglia lesions are associated with impairments in reward-based reversal learning
Brain
It was not MY fault: Event related brain potentials in active and observational learning from feedback
Cerebral Cortex
Localization of a neural system for error detection and compensation
Psychological Science
Distinct brain networks for adaptive and stable task control in humans
Proceedings of the National Academy of Sciences U.S.A.
Effects of errors in choice reaction tasks on the ERP under focused and divided attention
A neural system for error detection and compensation
Psychological Science
The medial frontal cortex and the rapid processing of monetary gains and losses
Science
Rapid electrophysiological brain responses are influenced by both valence and magnitude of monetary rewards
Journal of Cognitive Neuroscience
Brain potentials associated with expected and unexpected good and bad outcomes
Psychophysiology
To err is autonomic: Error-related brain potentials, ANS activity, and post-error compensatory behavior
Psychophysiology
It's worse than you thought: The feedback negativity and violations of reward prediction in gambling tasks
Psychophysiology
Cited by (157)
Neural correlates of evaluations of non-binary social feedback: An EEG study
2023, International Journal of PsychophysiologyAffective imagery boosts the reward related delta power in hazardous drinkers
2023, Psychiatry Research - NeuroimagingEvent-related potentials in response to early terminated and completed sequential decision-making
2023, International Journal of PsychophysiologyIs approaching sedentary behaviour or physical activity a reward? An EEG study
2023, Behavioural Brain ResearchAge-related differences in ERP correlates of value-based decision making
2023, Neurobiology of AgingCitation Excerpt :Findings on the FB-P3 have been less consistent than for the FRN (San Martin, 2012). While some studies focused on enlarged FB-P3 in response to larger sized outcomes regardless of valence (Bellebaum, Polezzi, & Daum, 2010; Yeung & Sanfey, 2004), some others found that the FB-P3 is enlarged more in response to unexpected outcomes rather than outcome size per se and thus reflects expectancy violation (Donaldson et al., 2016; Stewardson and Sambrook, 2020; Wu & Zhou, 2009). Distinct from the FB-P3, the following LPP is a more sustained response that shows sensitivity to both outcome valence (e.g., positive vs. negative feedback) as well as expectancy (e.g., positive RPE and negative RPE), but not to their interaction (Donaldson et al., 2016).
Sex differences on the response to others' gains and losses under cooperation and competition
2022, International Journal of Psychophysiology