The role of head and hand movements for infants’ predictions of others’ actions

Other people are a critical component of human life. For that reason, abilities underlying fluent social interaction and the understanding of others’ behavior are important. Consequently, much contemporary psychological research is dedicated to unraveling how action understanding develops early in life. Predictive looks, defined as looks to the (sub)goal of an observed action before that action is completed, are considered a hallmark of action understanding (Gredebäck & Falck-Ytter, 2015; Hunnius & Bekkering, 2014). Predictive looks are considered useful for action understanding because end points or turning points of actions frequently provide information relevant for understanding the purpose of the action: for drinking, one lifts a cup to the mouth, for clapping, the hands need to touch one another (Kilner, Friston, & Frith, 2007; Newtson, Engquist, & Bois, 1977; Uithol & Paulus, 2014). Hence, action prediction can form an important basis for action understanding, although action understanding requires more than prediction alone (Prinz, 2006). Many screen-based studies on action prediction have focused on infants’ ability to predict others’ goals based on observing the movements of the actor’s hand while leaving out the actor’s head (Ambrosini, Reddy, de Looper, Costantini, Lopez, & Sinigaglia, 2013; Elsner, Bakker, Rohlfing, & Gredebäck, 2014; Falck-Ytter, Gredebäck, & von Hofsten, 2006; Gredebäck & Kochukhova, 2010; Kanakogi & Itakura, 2011; Stapel, Hunnius, & Bekkering, 2015). The reason for leaving the actor’s head out of the stimulus videos is that the actor’s head movements and gaze turns are known to trigger gaze shifts in the infant from the actor to the object the actor looks at, a phenomenon called gaze-following (Johnson, Slaughter, & Carey, 1998). Although this approach of studying action prediction in isolation helps to learn to appreciate the role of body movements, it leaves the question of how infants predict actions in real-life situations unanswered (Rosander & von Hofsten, 2011, provided evidence for infants’ ability to predict real world actions). The current study aims to start bridging the gap between live studies (which typically display the actor’s head) and screen-based action prediction studies (which typically do not display the actor’s head) by comparing in a screen-based study infants’ predictions of an actor’s action when the actor’s head is visible or invisible.

The work of Flanagan and Johansson (2003) has formed an important source of inspiration for developmental studies on action prediction. Their adult participants displayed remarkably similar patterns of eye–hand coordination when stacking blocks themselves and when observing the experimenter stacking blocks. It appeared almost as if the eyes of the observers guided the hands of the experimenter, as the observers’ gaze landed on the location where the experimenter’s hand would go ahead of the actual hand action. These predictive looks (gaze preceding the hand) were taken as an indication that observers run the same computations based on the same action plans when observing an action as when performing the action themselves. In Flanagan and Johansson’s study, participants only saw the hands of the experimenter and not his face. This setup allowed the researchers to rule out the possibility that observers coordinated their own gaze with the experimenter’s gaze rather than with his hands when observing the block-stacking actions.

Developmental studies examining predictions of others’ actions have displayed actors performing manual actions without the actor’s face in view (Ambrosini et al., 2013; Elsner et al., 2014; Falck-Ytter et al., 2006; Gredebäck & Kochukhova, 2010; Kanakogi & Itakura, 2011; Stapel et al., 2015), following Flanagan and Johansson (2003). Results of these studies demonstrated that infants are indeed capable of predicting others’ actions, which shows from the infants’ predictive looks at the target of an action prior to action completion (Falck-Ytter et al., 2006; Hunnius & Bekkering, 2010; Kanakogi & Itakura, 2011; Stapel et al., 2015).

Studies on developmental action prediction elucidate, at least partially, the potential mechanisms underlying action prediction. Own motor ability, and thereby motor development, seems crucial for predicting others’ actions (Ambrosini et al., 2013; Elsner et al., 2014; Falck-Ytter et al., 2006; Gredebäck & Kochukhova, 2010; Kanakogi & Itakura, 2011; Stapel et al., 2015). This is in line with the mirror neuron or motor account of action prediction, which postulates that similar (sensorimotor) brain areas are recruited when acting and when perceiving actions of others (Prinz, 2006; Wilson & Knoblich, 2005; Wolpert, Doya, & Kawato, 2003). Some scholars have suggested that the mirror system is readily functioning at birth (Craighero, Leo, Umiltà, & Simion, 2011; Lepage & Théoret, 2007), which implies that infants should be capable of predicting actions regardless of their own motor repertoire. However, action prediction does not need to rely on motor processes, as visual experience can provide an alternative route to action prediction. Through statistical or associative learning, infants may learn that certain bodily movements, such as reaching, are often followed by certain end effects, such as holding an object in hand. The formed action–effect associations may then in turn be used to predict the action end effect when perceiving the action again (Elsner & Hommel, 2001; Paulus, van Dam, Hunnius, Lindemann, & Bekkering, 2011). Data of Hunnius and Bekkering (2010) illustrate that infants can indeed predict actions based on solely observational experience. In their study, 6-month-old infants made predictive eye movements towards the ear of an actor. Predictive gaze to the ear more frequently occurred when the actor brought a phone to her ear than when she brought a cup to her ear. Most likely, these predictive eye movements sprang from seeing others bringing phones to their ear, as infants of that age are not yet motorically capable of bringing objects to their ear themselves. Note that in the study by Hunnius and Bekkering, the actor did not move her head or eyes in a way that could reveal where the movement would end.

Apart from visual experience and motor processes, predictions of others’ actions may also rely on detecting direction cues in the stimuli, or on processing of socially relevant information. These latter two explanations are frequently mentioned in the gaze-following literature. Gaze direction, sometimes inferred from head orientation but most explicitly signaled by the eye direction, provides socially relevant information by revealing where the other agent is attending to (Driver et al., 1999; Langton & Bruce, 1999). That is, any given space, be it a room inside a house or a marketplace outside, contains an abundance of visual information that cannot be processed in detail all at once (Desimone & Duncan, 1995). Therefore, it is useful to look and see what the other intentional agent is attending to. The other agent might have selected a certain object of attention for a good reason, and it may prove to be a relevant object of attention for us as observers too (Triesch, Teuscher, Deák, & Carlson, 2006). This explanation of why we follow gaze assumes the ability to attribute intention to other agents, and to use that attribution to follow the gaze of the agent. Whether infants have the capacity to attribute intentionality (Gergely, Nádasdy, Csibra, & Bíró, 1995; Luo & Baillargeon, 2005; Sirois & Jackson, 2007) and productively use intention attributions to follow gaze is disputed (Butterworth & Jarrett, 1991; Johnson et al., 1998). Scholars favoring a more lenient view postulate that infants deduce from the social partner’s gaze direction where the person is looking at, without attributing intentionality to that partner. They may learn from experience that looking in the same direction as another person often leads to seeing interesting objects or events (Butterworth & Jarrett, 1991), which may through reward-based learning lead them to exhibit the same (gaze-following) behavior in subsequent situations.

Despite the controversy around the mechanisms underlying gaze following, it is undisputed that infants develop the capacity to follow gaze. The earliest report described that 30% of the tested 2-month-olds followed gaze, gradually increasing over age to 100% at 14 months (Scaife & Bruner, 1975). More complex gaze-following, such as following the experimenter’s gaze to an object located behind oneself and distinguishing smaller gaze angle differences, emerges in the second year of life (Butterworth & Jarrett, 1991). Both head and eye movements of the experimenter can independently elicit gaze-following behavior, but both cues combined result in the highest gaze-following scores (Butterworth & Jarrett, 1991; Corkum & Moore, 1995).

At first sight, it might seem that action prediction relies on different mechanisms when it is cued by head movements or hand movements, respectively. It seems a common intuition amongst scholars that predictions based on head and eye movements spring from mechanisms for social understanding and directional cueing, whereas predictions based on hand movements spring from sensorimotor mechanisms such as motor simulation or action–effect associations. However, head and eye movements not only convey directional and socially relevant cues which might allow intention attribution, head and eye movements also could potentially be simulated by the motor system. Furthermore, head and eye movements may also have formed the basis for action–effect associations (Hommel, Müsseler, Aschersleben, & Prinz, 2001) acquired through learning from observation (see for evidence on the acquisition of action–effect associations through observational learning: Paulus et al., 2011). Likewise, there is no strong theoretical reason why goal-directed hand movements could not bring about predictive eye movements through attributing intentions to the observed actor, if one would want to assume that infants have the capacity to attribute intentions. As such, all four explanations mentioned—observational learning, motor simulation, intention attribution, and detecting directional signals—count as viable explanations for both hand- and head-based action prediction. Though the current study was hence not designed to disentangle the mechanism underlying action prediction based on hand movements or on head movements, it may still yield insights relevant for the debate on these mechanisms.

Recent live-interaction studies provide some first indications that infants prioritize hand over head movements as joint attention between infants (ranging between 11- and 24-months of age) and their parents were more frequently established through attending to each others’ hands and hand-held objects than by means of gaze-following (Yu & Smith, 2013, 2017). These studies suggest that although infants are capable of following gaze and might use head turns as cues for predicting the subsequent action, they might rely more on hand than on head movements when predicting others’ actions.

The current study aimed to test the relative contributions of head and hand movements for infants’ action prediction. To that end, 13-month-old participants were presented with video clips of a model stacking rings to a pole while the participants’ gaze was tracked. In half of the videos, the model’s head was visible. In the other half of the videos, her head was invisible because a black rectangle occluded her head and neck. Looking time analyses were used to describe potential differences in duration of looks to the critical aspects of the scene, namely the model’s hand, her head, and the target location of the action. Due to infants’ preference for faces, we expected participants to look longer at the region in the scene where the head was displayed when it was visible compared to when the head was occluded. As a consequence of looking at the head during the action, infants were expected to look shorter at the model’s hand and the action target when the actress’ head was visible. In addition to looking time, gaze-arrival time was calculated, which is the difference in time between arrival of the model’s hand and participant’s gaze at the target. Positive gaze-arrival times indicated predictive gaze as the eyes precede the hand, whereas negative gaze-arrival times signaled reactive gaze. As the model portrayed natural actions, her head always started turning towards the target before movement onset of the hand. Hence, if infants made use of cues from head orientation, the gaze might be more ahead of the action when the head was visible compared to when it was not visible. Alternatively, infants might focus their attention on the model’s head when possible, at the expense of predicting the target. A face provides more than movement information only; it also carries other socially relevant information such as emotions, gender, and identity (Haan & Nelson, 1997; Quinn, Yahr, Kuhn, Slater, & Pascalis, 2002; Serrano, Iglesias, & Loeches, 1995). Processing this additional information might be prioritized over predicting the ongoing action and hence lead to later gaze-arrival. In addition to the conditional contrasts, exploratory analyses were conducted to obtain a description of gaze behavior within the head-visible condition. It was deemed of interest to explore whether infants would display more looks to the target following looks at the head or looks at the hand. If target looks would often follow from looks at the head, then the question is whether gaze landed earlier at the target following either a head or a hand look. As the head turn was the first sign of the impending action, infants might be quicker to look at the target if they focused on the head rather than on the hand. Lastly, to assess whether such a dichotomy of hand- versus head-based target looks was justified, we counted the number of trials in which the infants triangulated by looking at the hand as well as the head before looking at the target.

The current study aimed to investigate whether infants’ predictions of target-directed actions are affected by the visibility of the actor’s target-directed head movements. To that end, 13-month-old infants observed videos of a model reaching and grasping rings, transporting and stacking them on a pole. Analyses of the infants’ gaze showed that when the head was visible, infants looked at the model’s head and looked shorter at the hand performing the action as compared to when the head was invisible. During the transporting phases, infants followed rather than predicted the action. Infants predicted the targets of the reaching phases. Critically, gaze-arrival times at the target were not found to be different for when the model’s head was visible or invisible, neither during the transporting nor during the reaching phases. Interestingly, when the head was visible, looks to target arrived later if the infants focused on the head of the model instead of (solely) on her moving hand.

The infants displayed predictive gaze for the reaching actions. In contrast, the infants merely followed the transporting action. These results mimic two other studies (Gredebäck, Stasiewicz, Falck-Ytter, von Hofsten, & Rosander, 2009; Sciutti, Lohan, Gredebäck, Koch, & Rohlfing, 2016) reporting predictive gaze in 14-month-olds for reaching but not transporting actions. Potentially, infants preferred to look at the object that was being transported rather than at the target because the object was colorful and moving (Dannemiller, 2005).

More important for the central question of the current study are the differences between the head-visible and the head-invisible conditions. As expected, participants looked at the model’s head when it was visible and hardly looked to the same region when the head was occluded. Faces convey many types of socially relevant information such as gender, identity, and emotion (Haan & Nelson, 1997; Quinn et al., 2002; Serrano et al., 1995). Infants are frequently confronted with faces from early on and orient already as newborns towards faces or face-like stimuli (Goren, Sarty, & Wu, 1975). They are already capable of discriminating basic facial emotional expression at 3 months of age (Serrano et al., 1995; Young-Browne, Rosenfeld, & Horowitz, 1977). More importantly, between 3 and 9 months infants gradually start to focus on faces which are embedded in complex, dynamical scenes (Frank, Vul, & Johnson, 2009). The current results are in line with these prior findings, as the participants looked at the model’s head at the expense of looking at the model’s hand when her head was visible.

The finding that infants looked less at the model’s hand when the head was visible was in line with the hypothesis. Interestingly, one would expect that shorter looking at the hand may also affect the predictive eye movements. Two competing hypotheses were formulated which described the potential effect of infants’ shifted attention towards the model’s face. It could either have led to earlier looks to the target because the head moved prior to the hand, or, alternatively, to later target looks because looks to the face could take critical processing time. Overall, the gaze-arrival time data showed no loss due to the visibility of the face as the gaze-arrival time of looks to the target were not found to be different for the head-visible and invisible conditions, an observation which held for both the transporting actions and the reaching actions. The second implication of these results is that infants did not benefit from the visibility of the head. Importantly, the findings suggest that prior results on infants’ action prediction in screen-based tasks (Ambrosini et al., 2013; Elsner et al., 2014; Falck-Ytter et al., 2006; Gredebäck, & Kochukhova, 2010; Kanakogi, & Itakura, 2011; Stapel et al., 2015) may very well hold in real-life settings as well. Our results are furthermore in line with recent findings (Yu & Smith, 2017) that illustrate that infants coordinate joint attention with their parents based on hand movements rather than through gaze-following.

A maybe even more important implication flowing from our findings is that infants do not predict others’ action outcomes through gaze-following, despite being capable of gaze-following at this age (Butterworth & Jarrett, 1991; Scaife & Bruner, 1975), but through observing other’s hand movements. During the transporting actions, target looks were hardly ever preceded by looks at only the face and not the hand, but were frequently preceded by looks at the hand and not the face. During reaching actions, the frequency of looks to target after a look to the head and not the hand was not found to be different from the frequency of target looks after a look to the hand and not the head. In other words, goal-directed eye movements could be based on either head or hand movements. However, first looking at the face during the reaching actions led to reactive target looks whereas first looking at the hand led to predictive target looks. Similarly, when infants looked at the model’s head and hand prior to looking at the target, the looks to the target arrived later in comparison to when target looks occurred after only a look to the hand. This held for both the transporting actions and the reaching actions. Together, the results indicate that infants’ target looks may spring from gaze-following, but are quicker when based on observing the actor’s hand instead.

The finding that action prediction seems to be based on observation of hand movements rather than head movements is congruent with the motor account or mirror system account of action prediction (Prinz, 2006; Wilson & Knoblich, 2005; Wolpert et al., 2003). Subsets of mirror neurons in the macaque brain were found to increase in firing rate during observation of some specific but not merely all manual actions (Rizzolatti, Fadiga, Gallese, & Fogassi, 1996). This action-specific mirror system activity is thought to reflect action simulation of not-yet-finished movements, generating predictions of the ongoing action (Kilner et al., 2007). Hence, according to these views, observing hand movements can be sufficient to predict the action and the head movement can thus safely be ignored. This is in line with the behavior displayed by the infant participants here: looking at both hand and head before looking at the target hardly occurred, and in cases infants did look at both hand and head, gaze arrived later at the target than in purely hand-based target looks. From a strong motor perspective (e.g., Gredebäck & Falck-Ytter, 2015), 13-month-olds should be capable of predicting both the transporting and reaching actions as they at least have a minimal capacity to perform both actions, which was not found in the current study.

The current data allows for and does not rule out alternative explanations for the mechanisms underlying action prediction. The study did not include explicit manipulations of potentially perceived intentionality (Johnson et al., 1998), direction cues (Butterworth & Jarrett, 1991), observational experience (Hunnius & Bekkering, 2010) or tests of motor processes (Elsner, D’Ausilio, Gredebäck, Falck-Ytter, & Fadiga, 2013; Kanakogi & Itakura, 2011) that could help disentangle which mechanism best explains developmental action prediction. Prior studies on action prediction in infancy provide strong evidence that both observational experience (Henrichs, Elsner, Elsner, Wilkinson, & Gredebäck, 2014; Hunnius & Bekkering, 2010) and motor processes (Ambrosini et al., 2013; Elsner et al., 2014; Falck-Ytter et al., 2006; Gredebäck & Kochukhova, 2010; Kanakogi & Itakura, 2011; Stapel et al., 2015) play a role in infants’ predictions of others’ actions (see also Hunnius & Bekkering, 2014).

In conclusion, the current study demonstrated that 13-month-olds’ gaze-arrival time at another’s action target was not different for a situation in which the other’s head was visible or invisible for the observing infant. This suggests that results from prior screen-based studies on action prediction in infants, in which the model’s face was not presented (Ambrosini et al., 2013; Elsner et al., 2014; Falck-Ytter et al., 2006; Kanakogi & Itakura, 2011; Stapel et al., 2015), are likely to generalize to real-world settings in which infants do have visual access to the actor’s head. While both gaze- and hand-following was observed, only looks to the hand led to predictive target-looks. In sum, 13-month-old infants were found to predict other’s reaching actions based on the hand but not the head movements of the other person.