Elsevier

Consciousness and Cognition

Volume 67, January 2019, Pages 44-55
Consciousness and Cognition

Full Length Article
Action co-representation and the sense of agency during a joint Simon task: Comparing human and machine co-agents

https://doi.org/10.1016/j.concog.2018.11.008Get rights and content

Highlights

  • Participants performed a joint Simon task with a machine or a human partner.

  • The joint Simon task was coupled with an intentional binding task.

  • A joint Simon effect was found only when the partner was a human (and not a machine).

  • Intentional binding increased in human-human compared to human-machine interactions.

  • Interacting with humans boosts both the sense of agency and action co-representation.

Abstract

Recent studies have suggested that individuals are not able to develop a sense of joint agency during joint actions with artificial systems. We sought to examine whether this lack of joint agency is linked to individuals’ inability to co-represent the machine-generated actions. Fifteen participants observed or performed a Simon response time task either individually, or jointly with another human or a computer. Participants reported the time interval between their response (or the co-actor response) and a subsequent auditory stimulus, which served as an implicit measure of participants’ sense of agency. Participants’ reaction times showed a classical Simon effect when they were partnered with another human, but not when they collaborated with a computer. Furthermore, participants showed a vicarious sense of agency when co-acting with another human agent but not with a computer. This absence of vicarious sense of agency during human-computer interactions and the relation with action co-representation are discussed.

Introduction

The sense of agency refers to the experience of controlling one’s own actions, and through them, events in the outside world (Haggard & Chambon, 2012). This form of self-awareness is the basis of the legal and ethical structure of modern societies and forms the basis on which humans experience responsibility (Caspar et al., 2016, Frith, 2014, Moretto et al., 2011). Interestingly, studies have shown the emergence of a form of shared control when engaged in a cooperative task: we develop an experience of agency for actions and outcomes generated by our partner, a phenomenon called we-agency. This experience of agency could support our engagement in the cooperative task. Given that engagement with automated systems is essential in the context of human-machine cooperative activities such as piloting an airplane, understanding the challenges involved in developing a sense of shared agency with automata becomes a critical concern. The present study aims at addressing this issue and in particular at investigating the factors contributing to our experiences of vicarious agency during joint tasks with humans and artificial agents.

During the last decades, the sense of agency has largely been investigated using both explicit and implicit measures (for a recent review see Haggard, 2017). For instance, one of the implicit measures most widely used to quantify people’s sense of agency is the Intentional Binding phenomenon (IB; see Moore & Obhi, 2012): when a voluntary movement generates a sensory outcome, people experience an illusory temporal compression between the movement and the outcome. Importantly, this temporal binding seems to reliably occur in situations in which the participant is the agent, but not with passive movements (Haggard, Clark, & Kalogeras, 2002).

A large body of studies have used IB to shed light on how the sense of agency emerges during joint actions (Capozzi et al., 2016, Obhi and Hall, 2011a, Strother et al., 2010). On the one hand, it has been shown that people can exhibit IB for actions and outcomes generated by another human agent when they are engaged in a cooperative task (Obhi and Hall, 2011a, Strother et al., 2010). For instance, in Strother and colleagues’ (2010) study, pairs of participants were seated in front of a monitor and each member of the pair had his/her right finger on one side of the spacebar. The fingers of the participants were hidden so that no visual information about who was pressing the key was available. Participants alternately pressed the spacebar. During the operant trials, the key press triggered a tone 200 ms later. Participants had to estimate the perceived moment of the key press or the tone occurrence. By computing the action estimate and the tone estimate into one interval, the authors observed intentional binding for the other-generated shared action in a nearly similar fashion as for the self-generated action (Strother et al., 2010). In addition, in Obhi and Hall (2011a) study, participants were divided in pairs and were asked to press a spacebar. In a first experiment, the two participants could initiate the press at the time of their own choosing, but whenever one participant pressed the spacebar first, the other was to join in and press the spacebar as soon as possible. In a second experiment, the roles of the participants were pre-defined: one of the participants was to press the spacebar first while the second was always following her/his move. In both experiments, participants had to report their feeling of causal responsibility (i.e., their experience of agency) using a percentage scale and also to judge the onset time of the first key press or the onset of the tone. Interestingly, in both experiments, although only the initiator reported a reliable feeling of causal responsibility, both individuals showed IB (Obhi & Hall, 2011a). Notably, participants experienced IB for their own actions but also for the actions executed by the partner, suggesting that during joint tasks individuals can develop a vicarious sense of agency. It must be emphasized that it is the mere belief that another human agent caused an outcome (and not the fact of seeing or not the human-generated action) that allows individuals to develop a vicarious sense of agency.

On the other hand, there are studies that showed that intentional binding was not socially shared during cooperative tasks (Capozzi et al., 2016, Pfister et al., 2014). For example, Capozzi and colleagues (2016) asked participants to perform a key press that triggered after 259 ms (T1) an auditory tone that served as a start signal for a (non-naïve) co-agent key press. This co-agent key press triggered a different auditory tone which occurred at a fixed delay of 629 ms after T1 (T2). In the cooperative condition, they were told to coordinate their actions as if they wanted to create a melody with the two tones. In the competitive condition, the participants were told that the co-agent had to perform his key press as fast as possible as if he wanted to “wipe out” T1. Participants had to verbally report their perceived duration of either T1 or T2. The authors found that during sequential joint actions with a co-agent, individuals demonstrated IB for their self-generated actions but not for their partner’s action, independently of the context of the performed action (Capozzi et al., 2016). In addition, in Pfister et al. (2014)'s study, participants were paired. One assumed a leader role and the other a follower role. The leader had to press a key at the moment of their own choice and the key press triggered a tone after a certain delay (interval 1). This tone served as a go-signal for the follower to press his or her own key (interval 2). The follower's key press could trigger a tone after a random delay (interval 3) or no tone at all. Both the leader and the follower had to verbally judge the interval lengths. The results showed that the leader's interval estimations were always shorter than the follower's interval estimations, meaning that the leaders always made more IB than the followers. In addition, as the initiator of the action, the leader made IB while the follower, as the observer, did not (interval 1). For the interval 2 estimation, the leader experienced IB for the follower's action but the follower did not. Hence, leaders' sense of agency does not only concern their own actions and their adjacent effect but also predictable actions of other agents. Finally, when the follower's key press generated a sensory consequence, the temporal interval estimations (interval 3) were similar for the leader and the follower: none of them exhibited IB. That is, the follower never experienced a sense of agency whether it be over their own action and its effect or over those of the leader (Pfister et al., 2014).

Interestingly, with respect to artificial systems, there are studies that have consistently found that individuals’ sense of self-agency and vicarious agency were degraded when interacting with a non-human-like artificial agent (e.g. a computer). For example, Obhi and Hall (2011b) showed that in a joint task participants exhibited IB for their own actions and those generated by another human co-agent, whereas, when acting jointly with a machine, IB for their own actions and those generated by the machine disappeared. In a similar vein, Berberian and colleagues (Berberian, Sarrazin, Le Blaye, & Haggard, 2012) showed that human operators experienced a very strong decrease of their sense of agency when interacting with highly automated autopilot interfaces. Moreover, they reported a lack of control over the outcomes generated by the automated system. Conversely, it has been shown that observing an action performed by a human-like automaton (e.g. an anthropomorphic hand with servo-actuated tendons rotating each finger joint) induced a temporal binding effect in a similar way as during the observation of another person performing the same action (Khalighinejad, Bahrami, Caspar, & Haggard, 2016).

Taken together these studies suggest that the ability to build a sense of vicarious agency during a joint task with another agent is highly influenced by the nature of the agent one interacts with. During joint tasks with non-human-like automated machines people fail not only to experience the agency of the artificial co-actor but also their own agency. This decrease of agency during human-machine interactions might lead in turn to a reduction of involvement with the artificial system since a felt lack of control would directly impact the operator’s engagement and responsibility in the task (Caspar et al., 2016, Frith, 2014, Moretto et al., 2011).

This study aims at exploring this decrease in individuals’ sense of agency during their interactions with automated agents compared to human-human interactions. Past research suggested that the experience of agency for the actions performed by another individual is based on the ability to simulate or co-represent that individual’s motor plans and intentions (Obhi and Hall, 2011b, Wohlschläger et al., 2003; see also Kunde, Weller, & Pfister, 2017). Based on this hypothesis, people would not experience a sense of vicarious agency when interacting with computers because they fail to simulate computer-generated actions (Obhi & Hall, 2011b). In line with this assumption, the present study investigated the link between action co-representation and individuals’ experience of agency during actions performed by another human versus an automated artificial agent in a joint task. The ability to co-represent the actions of the other agent was investigated with the Social Simon task (Sebanz, Knoblich, & Prinz, 2003)

Action co-representation of human-generated actions has been classically demonstrated by the Social Simon Effect (SSE; Sebanz et al., 2003). In the classical Simon effect people’s reaction time in response to the presentation of a target stimulus decreases when the target is presented in the same relative spatial location as the response. For instance, in one version of this paradigm participants are required to execute a left hand action as soon as a red target is presented and a right hand action as soon as a green target it presented. Red/green targets are displayed in either the same relative location as the participant’s response (e.g., the target is presented to the left of a central fixation and participants have to perform a left key press) or in the opposite location (e.g., participants have to press the right key but the target is presented on the left). It has been shown that people are faster and more accurate when the target is on the same side as the response compared to when it is presented on the opposite side. In the go-nogo version of the task, that is to say when participants had to respond to only one target (e.g., respond only to the green target with the right hand), the interference disappeared. This result suggested that the Stimulus-Response congruency effect observed in the classical Simon task derives from the cognitive interference caused by two different action representations that are concurrently activated (Simon & Wolf, 1963). However, interestingly, Sebanz et al. (2003) showed that when the participant performed the same go-nogo task with another agent (joint Simon task), and each of them was required to respond to only one target (i.e., the participant responded only to the green target, and the partner only to the red target), the interference effect for the incongruent Stimulus-Response key mapping reappeared. Hence, it has been proposed that during a joint task, individuals integrate the behavior of their partner into their own motor scheme even though it is not task relevant (Sebanz et al., 2003), but see the reference coding hypothesis (Dolk et al., 2011, Dolk et al., 2013) or the task representation hypothesis (Yamaguchi, Wall, & Hommel, 2018). Consistently, neuroimaging data showed that when someone observes or imagines a conspecific performing an action, their premotor cortex is activated – though to a lesser extent than during action execution – allowing action understanding and prediction with the help of the observer’s own motor expertise (Calvo-Merino et al., 2004, Kilner et al., 2007, Pineda et al., 2000). Moreover, it has been shown that during a joint Simon task, stimuli that referred to the partner’s action evoked a similar frontal cerebral signal as the stimuli that were addressed to the participants themselves (Sebanz, Knoblich, Prinz, & Wascher, 2006).

Some recent studies have started investigating whether action co-representation also occurs during cooperative tasks with artificial automated systems. For example, a study showed a congruency effect during a joint Simon task when participants thought they were interacting with an unseen computer (Wen & Hsieh, 2015). However, other studies that used a similar paradigm found no congruency effect, whether at the behavioral or at the cerebral level, during a joint Simon task with an unseen computer (Tsai, Kuo, Hung, & Tzeng, 2008) or with a robot with high level of human-like appearance (i.e. a full humanoid robot with a torso, an anthropomorphic head, eyes, and two arms with fingers) when it was described as an unintentional robot (Stenzel, Chinellato, Tirado Bou, & del Pobil, 2012). Similarly, cerebral motor resonance has been shown not to arise during the observation of non-human-like machine-generated actions (Mann et al., 2007, Perani et al., 2001; but see Gazzola et al., 2007, Kuz et al., 2015) while action recognition mechanisms has been shown to be sensitive to artificial humanoid avatars’ actions (Ferstl, Bülthoff, & de la Rosa, 2017).

In sum, these studies suggest that during joint actions people tend to spontaneously simulate the action of another human partner but not the actions of an artificial agent (excepted when people believe that the artificial partner is an intentional human-like agent). Moreover as we mentioned above, a separate line of evidence indicates that people can experience agency for other human’s actions but fail to experience it for actions performed by artificial systems. In line with these elements of evidence, the present study sought to investigate whether the ability to experience agency for the actions performed by another agent is linked to the ability to co-represent the agent’s actions. To address this issue we ran a behavioral study combining a Social Simon reaction time task and an intentional binding task. The reaction times observed in the Simon task served as an index of action co-representation while IB was used as an implicit measure of the sense of agency. More specifically, participants performed the Simon task alone (standard Simon condition), or with another agent (joint go-nogo conditions) that could be another human or a computer (we used a desktop computer considering that it is the artificial system that people most commonly work with). Participants were required to estimate the time interval between their action or the action of the co-agent and a subsequent auditory outcome. This allowed us to investigate two issues. Firstly, it permitted us to assess whether participants co-represent the actions performed by a human and an automated artificial co-actor. For instance, if people do not co-represent the action performed by an automated system, we should observe the classical stimulus-response congruency effect (see description of the Simon task above) when participants perform a joint task with another human partner but not when they perform it with an artificial agent. Secondly, it allowed us to evaluate whether participants’ experience of self-agency and vicarious agency during joint actions is linked to their ability to co-represent the action of their partner.

Section snippets

Ethic statement

This study was approved by the CERES (recommendation n°201726, institutional ethical research committee of the Paris Descartes University, France). The investigation was carried out in accordance with the Declaration of Helsinki and each participant provided his or her written informed consent before starting the experiment. All participants were assigned a number in order to ensure the anonymity of the data.

Participants

Sixteen healthy adults volunteered to take part in the experiment (6 women, mean age

Data analyses

Our dependent measures were the mean target detection Response Times (RTs) and the mean perceived action-tone temporal interval. Statistical analyses were performed with R software (3.3.1 version). Extreme values (the values that were below or above 2 standard deviations from the mean) of the participants’ response times and perceived intervals were excluded from further analyses in order to eliminate outliers and allow for robust statistical analyses. The significance level was set at α = .05.

Social Simon effect

We first examined the SSE during joint-actions with another human and with a desktop computer (Fig. 3). We assessed the normality of the RTs distributions of the differences between the congruent trials and the incongruent trials in the standard Simon, individual go-nogo, joint HH self and joint HM self conditions using the Shapiro-Wilk test. The analyses showed that none of the RTs distribution deviated from normality (all W > 0.90 and all p > .10). We then computed a within-subjects 4 × 2

Discussion

In this study, our aim was to investigate the sense of agency in joint action with another human versus a desktop computer. To assess action co-representation, we used a typical Social Simon task where participants had to detect a target that could appear on the same side as their response key or on the opposite side. Accurate target detection triggered an auditory tone after a randomized delay. Participants had to estimate the temporal delay between the target detection and the onset of the

Conclusion

We raised the question of the relationship between action co-representation and the sense of we-agency. Given the convergence of the SSE and the vicarious sense of agency, we hypothesized that as soon as individuals are able to use their own motor system to simulate their partner’s action into their own motor system, they can understand their partner’s intention with the help of their previous experiences, supporting the construction of a vicarious sense of agency. During human-machine

Author contribution

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding source

This work was funded by a French National Research Agency grant (Young researcher program – ANR-15-CE26-0010-01). Elisabeth Pacherie’s work was supported by ANR-10-LABX-0087 IEC and ANR-10-IDEX-0001-02 PSL*.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References (47)

  • D. Perani et al.

    Different brain correlates for watching real and virtual hand actions

    Neuroimage

    (2001)
  • N. Sebanz et al.

    Representing others' actions: Just like one's own?

    Cognition

    (2003)
  • N. Sidarus et al.

    Difficult action decisions reduce the sense of agency: A study using the eriksen flanker task

    Acta Psychologica

    (2016)
  • L. Strother et al.

    Subjective agency and awareness of shared actions

    Consciousness and Cognition

    (2010)
  • A. Wohlschläger et al.

    Intentionality as a constituting condition for the own self-and other selves

    Consciousness and Cognition

    (2003)
  • B. Berberian et al.

    Automation technology and sense of control: A window on human agency

    PLoS One

    (2012)
  • F. Beyer et al.

    Beyond self-serving bias: Diffusion of responsibility reduces sense of agency and outcome monitoring

    Social Cognitive and Affective Neuroscience

    (2017)
  • B. Calvo-Merino et al.

    Action observation and acquired motor skills: An fMRI study with expert dancers

    Cerebral Cortex

    (2004)
  • J.A. Dewey et al.

    Do implicit and explicit measures of the sense of agency measure the same thing?

    PLoS One

    (2014)
  • T. Dolk et al.

    How “social” is the social Simon effect?

    Frontiers in Psychology

    (2011)
  • T. Dolk et al.

    The Joint Simon Effekt: A review and theoretical integration

    Frontiers in Psychology

    (2014)
  • T. Dolk et al.

    The (not so) social simon effect: A referential coding account

    Journal of Experimental Psychology: Human Perception and Performance

    (2013)
  • P. Haggard

    Sense of agency in the human brain

    Nature Reviews Neuroscience

    (2017)
  • Cited by (38)

    • Agency and social affordance shape visual perception

      2023, Cognition
      Citation Excerpt :

      Different research teams investigated the sense of agency during coordinated actions between two partners. They revealed that participants experienced IB for their own actions as well as for their partners' actions (Bolt, Poncelet, Schultz, and Loehr, 2016; Bolt and Loehr, 2017, 2021; Jenkins, Esemezie, & Lee, 2021; Le Bars, Devaux, Nevidal, Chambon, and Pacherie, 2020; Sahaï, Desantis, Grynszpan, Pacherie, and Berberian, 2019; Sahaï et al., 2022). Furthermore, in joint actions where two participants were to act simultaneously on the same device, IB was observed irrespectively of who initiated the action first (Grynszpan et al., 2019; Obhi and Hall, 2011; van der Wel, Sebanz, and Knoblich, 2012).

    • Towards integrating joint action research: Developmental and evolutionary perspectives on co-representation

      2022, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      Further, under some conditions joint effects have been found despite the absence of attention getters (Ruissen and de Bruijn, 2015; Ruys and Aarts, 2010; Sebanz et al., 2003; Tsai et al., 2008). Likewise, the referential coding account cannot explain findings showing an effect only with (believed) intentional compared to non-intentional co-actors (Sahaï et al., 2019; Stenzel et al., 2012, 2014; Tsai et al., 2008, see Table 1). Another possibility is that the two mechanisms are activated depending on some external factors, namely task settings and instructions, which indeed can readily explain some contradictory findings in the literature.

    • Social, affective, and non-motoric bodily cues to the Sense of Agency: A systematic review of the experience of control

      2022, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      In van der Weiden et al. (2019), participants performed a computerized version of the Social Simon Task: self-agency ratings were always higher than other-agency ratings, even when the other participant had to respond to the stimulus. Although the study by van der Weiden et al. (2019) may indicate that the Social Simon Effect does not translate in modulations of explicit SoA, Sahaï et al. (2019) provided evidence that implicit SoA may instead be influenced. First, they reported that the Social Simon Effect occurred exclusively when the other agent was a person, and not a computer.

    View all citing articles on Scopus
    View full text