Communication for coordination: gesture kinematics and conventionality affect synchronization success in piano duos

Successful coordination is partly dependent on the quality of the visual signal given by the leader—particularly at piece onset, where there is no prior audio to indicate when the first notes should fall or at what tempo. Musicians commonly use head gestures to signal piece onsets, regardless of their instrument; head gestures were therefore our focus here, though we acknowledge that much of the upper body, as well as facial expressions and breathing, can be involved. The current study investigated how head movement kinematics communicate beats, and tested four kinematic properties of head gestures that we predicted could help observers detect communicated beats more successfully. This section of the paper develops these hypotheses.

For both conductors and instrumentalists, the kinematics rather than trajectories of cueing gestures have been shown to indicate the position of the beat, or tactus (Luck & Toiviainen, 2006). “Followers” attempting to synchronize with instrumentalists’ cueing gestures tend to perceive beats as aligning with major peaks in the leader’s head acceleration, rather than with points of direction change in head trajectories (Bishop & Goebl, 2017). This was observed among pianists and violinists in a study of synchronization in duo performance. Performers’ head movements were measured using accelerometers and Kinect sensors as they took turns cueing each other in at the starts of short passages. An aim of the current study was to replicate these findings (H1) using an expanded version of the same procedure and a more precise motion measurement system.

The easiest gestures to synchronize with are presumably those that convey beat position clearly and reliably. If followers aim to align their starting notes with peaks in leaders’ head acceleration, then leader/follower synchronization should be most successful when the leader’s starting notes align precisely with his or her own head acceleration peaks. The current study tested this hypothesis (H2) by calculating the offset of leaders’ first note onsets from major peaks in leaders’ head acceleration curves, and relating the magnitude of these offsets to success in note synchronization.

The clarity of a gesture, and how readily others synchronize with it, might also be influenced by its articulation. Wöllner, Parkinson, Deconinck, Hove and keller, ( 2012) found that observers synchronized finger-taps more successfully with quantitatively averaged conductor gestures, which were low in jerk, than with individual conductor gestures, which were higher in jerk. Jerk, the third derivative of position, indicates the smoothness of acceleration changes. The authors also observed more successful synchronization with marcato gestures, where the differences between acceleration maxima and minima were large, than with legato gestures, where the differences between acceleration maxima and minima were small. Here, we hypothesized that gestures high in smoothness (low in jerk) would be clearer and synchronized with more successfully than gestures with high jerk (H3).

We also tested the possibility that musicians synchronize more successfully with gestures marked by a larger-magnitude indication of the beat than with gestures marked by a smaller-magnitude indication of the beat (H4). Gesture magnitude was quantified in terms of how far the head travelled along the forwards–backwards axis while indicating the beat. Instrumentalists sometimes exaggerate their gestures at piece entrances and other places where exchanging visual cues benefits synchronization, and a test of how gesture magnitude affects synchronization should indicate whether this is an effective strategy.

Observers’ success at synchronizing with instrumentalists’ cueing gestures might also relate to the prototypicality of those gestures (Wöllner, Parkinson, Deconinck, Hove, & keller, 2012). People synchronize most successfully with gestures that are similar to those they produce themselves (Keller, Knoblich, & Repp, 2007; Wöllner & Cañal-Bruland, 2010). This effect is generally attributed to the strengthening of action prediction mechanisms that occurs with experience. According to this theoretical perspective, people use their own action planning systems to simulate observed movements—a process that may or may not yield overt motor output (Calvo-Merino, Grezes, Glaser, Passingham, & Harrad, 2006; van der Wel, Sebanz, & Knoblich, 2013). They then predict the course of observed gestures using the same mechanisms that they use to predict the course of their own gestures. When the observed or performed gesture is similar to gestures a person has performed in the past, action-perception links are stronger and prediction is facilitated. In the current study, we expected that highly prototypical cueing gestures would be more likely than highly idiosyncratic cues to align with followers’ own gesture tendencies, and would therefore be easier to predict and synchronize with (H5).

Theories of embodied music cognition posit that we use our own bodies to interpret the musical gestures produced by others (Leman, 2012). In other words, we understand others’ motor intentions by overtly or internally mirroring aspects of their actions (Jacob & Jeannerod, 2005; Jeannerod, 2003). Our ability to internally simulate others’ gestures is thus central to the concept of embodiment. Simulation mechanisms facilitate the translation of gestures into sound and the translation of sound into expressive gestures (Leman & Maes, 2014).

As stated above, observed actions can be simulated without overt replication, though in some cases the process clearly shapes motor output. For example, an imitation bias is observed among people who are asked to make speeded movements that are either congruent or incongruent (e.g., in terms of magnitude or direction) to irrelevant movements viewed simultaneously on a computer screen (Grosjean, Zwickel, & Prinz, 2009; Schubö, Aschersleben, & Prinz, 2001). Incongruent movements are performed less accurately than congruent movements, indicating an unintentional assimilation of observed motion parameters into the observer’s own performed actions.

At times, people overtly mimic each other’s behaviour. This mimicry has cognitive benefits: when people perform gestures that are high in similarity and coordinated in time, their attention is drawn towards each other and their perception and memory for each other’s behaviour is facilitated (Macrae, Duffy, Miles, & Lawrence, 2008). Furthermore, moving in rhythmic coordination with others can promote social bonding, improving participants’ ratings of partner trust and likeability (Hove & Risen, 2009) and increasing the likelihood of prosocial behaviour (Wiltermuth & Heath, 2009).

The current study included a test of whether coordination of body gestures occurs between duo performers at piece onset. Previously, some correlation in head and upper body sway has been observed within pairs of duo pianists. Goebl and Palmer ( 2009) found that duo pianists’ head movements were more synchronized when they performed under reduced auditory feedback conditions (unable to hear themselves or unable to hear their partner) than when they performed with normal two-way auditory feedback. Despite the heightened synchrony of head movements, however, note synchronization under reduced auditory feedback conditions was poor. Keller and Appel ( 2010) tracked the upper body movements of piano duos and found that the further the body movements of the primo performer (who usually plays the higher-pitched part) lagged behind those of the secondo (who usually plays the lower-pitched part), the greater note asynchrony became. Since primo note onsets consistently led secondo note onsets, the authors suggested that congruence between leader/follower relations at the levels of note onsets and body sway may be important for successful ensemble coordination.

Still unclear is whether leader/follower coordination of cueing gestures occurs at piece onset, and to what extent this coordination of body movement relates to note synchronization. In line with theories of embodiment, we hypothesized that coordinating cueing gestures would help performers gauge each other’s intended timing and, therefore, facilitate note synchronization (H6). Followers’ gestures were expected to mimic the form of leaders’ gestures and follow a parallel timecourse.

This study aimed to assess how kinematic measures affect the synchronizability of ensemble musicians’ cueing-in gestures. Motion capture recordings were made of nine piano duos performing short musical passages under alternating leader and follower conditions. The assigned leader for each trial was responsible for cueing in the follower at a particular tempo, with the aim of synchronizing their performance of the passage as precisely as possible. During a subsequent gesture-following task, a subset of the motion capture recordings of leader performances were presented (with audio) to an independent sample of 10 skilled musicians, who tapped in time with the leaders’ performed beats. Using data from this test, a measure of “synchronizability” (i.e., average leader–follower first beat asynchrony) was obtained for each leader gesture.

The alignment between followers’ first taps (for gesture-following task participants) or performed beats (for interactive duo performance task participants) and extremes in leaders’ head position, velocity, and acceleration curves was examined. Our focus was exclusively on leader–follower synchronization at piece onset, though a similar investigation of how gesture kinematics affect synchronization across the first few beats of a piece could also be made. It was expected that followers’ first taps would align with acceleration peaks in leaders’ gestures (H1), confirming previous findings (Bishop & Goebl, 2017). It was also expected that the precision of alignment between leaders’ first note onsets and their own head acceleration peaks (H2), as well as increased gesture smoothness (H3), magnitude (H4), and prototypicality (H5) would improve the synchronizability of leaders’ gestures. Finally, the hypothesis that increased similarity in the movements made by leader–follower pairs at the time of piece onset relates to improved leader–follower note synchronization was tested (H6), using data from the recording sessions.

Our first experiment investigated pianists’ synchronization with their duo partners’ cueing-in gestures under interactive conditions, while two-way communication was possible. The hypothesis that followers synchronize their piece onsets with peaks in leaders’ head acceleration was assessed. We also tested for coordination in duos’ body movements around piece onsets.

A second experiment was carried out with the aim of identifying the kinematic properties that improve cueing gesture synchronizability. Audio-visual recordings of pianist performances, collected during the first experiment, were used as stimuli for a beat-tapping task, which was completed by a sample of 10 musicians. The average accuracy of these musicians’ synchronization was taken as an indicator of gesture synchronizability, serving as a more reliable measure than the accuracy of individual followers’ synchronization during the interactive duo performance experiment, due to the larger sample size.

A central aim of this study was to test the hypothesis that points of peak acceleration in instrumentalists’ cueing gestures communicate beat position (H1). This prediction was addressed with an analysis of how followers’ first note onsets aligned with leaders’ gestures. More specifically, we assessed the alignment between followers’ first onsets and extremes in leaders’ head position, velocity, and acceleration curves. If beats were to be communicated via head position, it is logical to expect that beat locations would align with points of path reversal, as these occur in all gestures, regardless of their trajectory. If beats were to be communicated via head velocity or acceleration, they would likely coincide with either maxima or minima in the velocity or acceleration curves.

Peaks and troughs, therefore, were identified in the cue window of each leader’s position, velocity, and acceleration curves. Peaks were defined as points preceded by five consecutively increasing observations and followed by five consecutively decreasing observations that were outside the 99% confidence interval for a surrounding window of 300 ms. Troughs were defined as points preceded by consecutively decreasing observations and followed by five consecutively increasing observations that were likewise outside the 99% confidence interval for a surrounding 300 ms window. A constant rather than tempo-adjusted window size was used, as adjusting for tempo would have required a subjective judgement of at what hierarchical level of the beat each performer had gestured (e.g., 2 beats per bar vs. 4 beats per bar).

Cue gestures were assumed to have at least two points of path reversal, so peak-trough pairs separated by no more than one beat were identified. Since we expected the cue gesture to be more prominent than other movements made during the cue window, the peak-trough pair spanning the greatest range in position, velocity, or acceleration values was selected. The time interval between each selected peak and trough and the follower’s first note onset was calculated as an indication of the precision of their alignment. Peak-to-onset and trough-to-onset intervals were averaged across trials to produce a mean interval for each follower. Interval durations were divided by performers’ average interbeat intervals to achieve normalized values with units of interbeat intervals (IBIs). Sample head position, velocity, acceleration, and jerk curves are given in Fig.  4.

If either peaks or troughs in a given dimension indicate beats, then peak- or trough-to-onset intervals could be expected to cluster around two points: interval lengths of approximately 1 IBI would occur if the selected gesture feature preceded the follower’s onsets by one IBI (communicating a preparatory beat), while interval lengths of approximately 0 IBIs would occur if the feature and the follower’s onsets were synchronized. For our purposes, a clustering of intervals around either value was taken as an indication that the point communicated beat position. To assess the reliability of alignment between the selected peaks and troughs and followers’ first onsets, the proportion of average intervals approximating either 0 IBIs or 1 IBI (±0.2 IBIs) was calculated. Separate analyses were done for followers from the interactive duo performance task and gesture-following task.

Statistics for peak- and trough-to-onset interval distributions are presented in Table 2. To make sense of these data, we have to consider both (1) how reliably followers’ onsets aligned with each landmark and (2) around which values each distribution centered. Reliable alignment with a particular landmark (high percentages in columns 3 and 6 of Table 2), plus a median value near 0 or 1 IBI, would be evidence that the landmark communicates beat position.

Interval distributions for the interactive duo performance and gesture-following tasks are shown in Figs.  5 and 6, respectively. For head position, we found that followers in both tasks aligned their first onsets more closely with peaks than troughs. Medians for peak-to-onset distributions were near 1 IBI, while medians for trough-to-onset distributions were not. For head velocity, neither peaks nor troughs seemed to communicate beats, as alignment percentages were low and medians were not reliably near to either 0 or 1 IBI. For head acceleration, the results were a bit more complex: followers aligned their onsets more closely with peaks than troughs in the interactive task, while in the gesture-following task, onsets aligned slightly more closely with troughs than peaks. The period of deceleration between acceleration peak-trough pairs might have communicated beats to participants in the gesture-following task—a possibility that is considered in the discussion. For both tasks, however, followers aligned their first onsets more reliably with acceleration landmarks than with position or velocity landmarks.

Across motion parameters, a timing difference was noticeable between the interactive duo performance and gesture-following tasks: gesture-following participants’ first taps tended to align with a later point on leaders’ gesture curves than did interactive duo followers’ first onsets. For example, peaks in position and acceleration preceded interactive duo followers’ first onsets by slightly less than one beat and gesture-following participants’ first taps by slightly more than one beat. Correspondingly, leader-follower asynchronies were greater for gesture-following participants than for interactive duo participants, Z = 20.44, p < 0.001, r = 0.56 (gesture-following task M = −0.16 IBIs, SD = 0.64 IBIs; interactive duo task M = −0.01 IBIs, SD = 0.17). ² This timing difference could reflect better anticipation of the beat among interactive duo followers or a task-dependent difference in how beats were perceived.

The potential effects of leader gesture/follower onset alignment on note synchronization were assessed as an additional test of which gesture parameters were most useful in communicating beat position during the gesture-following task. Only position peak-to-onset, velocity peak-to-onset, and acceleration peak- and trough-to-onset interval distributions were considered, since their medians were close to 1 IBI (Table  2, column 7). For each distribution, trials with intervals approximating 0 or 1 IBI (±0.1 IBIs; “aligned”) were compared to trials without intervals approximating 0 or 1 IBI (“not aligned”), using mean absolute asynchronies of first tapped beats as the dependent variable. Significantly improved synchronization (at \(\alpha =0.01\)) was observed when first taps aligned with position peaks, Z = 2.95, p = 0.003, r = 0.11 (aligned M = 0.33 IBIs, SD = 0.29; not aligned M = 0.41 IBIs, SD = 0.33), acceleration peaks, Z = 4.83, p < 0.001, r = 0.15 (aligned M = 0.33 IBIs, SD = 0.32; not aligned M = 0.41 IBIs, SD = 0.34), and acceleration troughs, Z = 7.13, p < 0.001, r = 0.22 (aligned M = 0.27 IBIs, SD = 0.27; not aligned M = 0.42 IBIs, SD = 0.34). No significant difference was observed for velocity peaks, Z = 2.02, p = 0.04. These findings provide evidence that leaders’ head trajectories and acceleration patterns are used as cues to beat position.

In this section, analyses testing the potential effects of gesture kinematics and leader expertise on leader–follower synchronization are presented, using data from the gesture-following task. Asynchronies obtained from the gesture-following task were not normally distributed, so the results of non-parametric tests are reported.

Leaders’ first onsets aligned more precisely with acceleration peaks than with other kinematic landmarks, as evidenced by the high percentage of average acceleration peak-to-onset intervals that approximated 1 IBI. There was also some alignment with position peaks, but no evidence of alignment with position troughs, suggesting that head position also makes some contribution to the communication of beats. The first onsets performed by followers during the interactive duo performance task likewise aligned most reliably with peaks in leaders’ head acceleration, with a high percentage of average acceleration peak-to-onset intervals approximating 1 IBI. Acceleration peaks preceded both leaders’ and interactive duo followers’ first onsets by slightly less than 1 IBI. Alignment with position peaks was again more reliable than alignment with position troughs, indicating that beats tend to be communicated when leaders are near points of backwards-to-forwards path reversal.

The first taps performed by followers during the gesture-following task showed greater alignment with position peaks than with position troughs, as was the case for participants in the interactive duo performance task. In contrast to recording followers, however, participants in the gesture-following task showed even more reliable alignment with acceleration troughs than with acceleration peaks (63% of average trough-to-onset intervals and 50% of average peak-to-onset intervals approximated 0 or 1 IBI). Gesture-following task participants' first taps followed acceleration peaks by slightly more than 1 IBI and acceleration troughs by slightly less than 1 IBI, suggesting that perceived beats may have fallen in between these points.

The slight delay in gesture-following task participants’ first taps relative to interactive duo followers’ first onsets, apparent in the shifted distribution medians, was presumably a result of the difference in task completed by the two groups. The reduced information available to followers during the gesture-following task meant that only movement cues could be used to predict piece onsets. The access followers in the interactive duo performance task had to other cues, such as facial expressions and the sound of breathing, might have influenced their prediction processes, leading to earlier onsets.

The significance of acceleration patterns in communicating beat position may relate to the kinematics of sound-producing gestures. The sound-producing striking gestures used in drumming or piano-playing are similar in form to the head-nodding gestures studied here, as in both cases, changes in gesture trajectory are involved in communicating beats. Research on air drumming has shown that, when people are instructed to mime drumming gestures in synchrony with a sounded rhythm, acceleration peaks in their hand gestures slightly precede audio onsets, while “hits” (points of downwards-to-upwards path reversal) lag slightly (and more variably) behind audio onsets (Dahl 2014). Sharp decelerations from peak acceleration points, therefore, indicated beat position. In piano-playing, peaks in finger acceleration correspond to moments of key impact and, as such, sounded beats (Dalla Bella & Palmer, 2011; Goebl & Palmer, 2008). If peak accelerations typically lead the sound onsets produced by percussive sound-producing gestures, then intrinsic knowledge of this association could shape our performance and perception of non-sound-producing gestures, including the cueing-in gestures intended to communicate timing information.

Our finding that gesture synchronizability improved with increasing gesture smoothness and magnitude is in line with prior research suggesting that synchronization is more successful with averaged conductor gestures that are low in jerk (Wöllner et al., 2012). During joint action tasks, people tend to reduce the variability and increase the magnitude of their gestures, and the positive effects of gesture smoothness and magnitude observed here show that gesture predictability can improve as a result. These effects might have played a particularly strong role in the context of the gesture-following task, given that leaders’ upper body movements were the only source of timing information prior to piece onset. Normally, facial expressions are also involved, and likely to help with securing the follower’s attention and discriminating the cueing-in gesture from other preparatory gestures. The sound of the leader’s breathing also often acts as a cue to piece onset, but was not part of the audio presented to followers during the gesture-following task. These factors, along with the inability of followers to interact with the leader, would account for the much higher variability in synchronization success that gesture-following task participants achieved, relative to recording followers.

Contrary to our hypothesis, gesture prototypicality related to a decline, rather than an improvement in gesture synchronizability. Our measure of prototypicality (i.e., the average lag-0 cross-correlation coefficient for each gesture) gave preference to flatter, less distinctive curves. Given the relationship between gesture magnitude and synchronizability, it is therefore not surprising that synchronization was less successful with gestures scoring high in prototypicality. We should also note that none of the pianists in this study performed gestures with particularly idiosyncratic trajectories, in contrast to our previous study, in which a few musicians displayed noticeably idiosyncratic movement styles. Future studies might specifically recruit such individuals in order to disentangle the effects of gesture magnitude from individuality in movement style.

The hypothesis that synchronization would be more successful among interactive duo performance task participants when the follower’s movements mirrored the leaders’ movements was not supported. Both this and our hypothesis that synchronizability would improve with increasing gesture prototypicality derived from the idea that observers use their own motor systems to interpret and predict others’ gestures. While there was evidence that some leader–follower pairs performed head movements that were similar in form and closely aligned in time, leader–follower gesture alignment had no effect on the success of note synchronization. Different results might arise when cueing gestures are more directly tied to sound onset. For example, when pianists perform together on a single piano, they often use exaggerated wrist movements to help synchronize chords. Aligning wrist movements could help pianists regulate their timing and prove beneficial for note synchronization.

During the gesture-following task, synchronization was more successful with gestures performed by pianists who had either conducting experience or substantial experience performing in small ensembles than with gestures performed by pianists who had little ensemble experience. This difference in synchronization success suggests that conducting and ensemble performance experience improve pianists’ cueing gestures similarly. During the interactive duo performance task, slightly superior synchronization was observed among ensemble-experienced duos. The absence of similarly enhanced synchronization among conductor duos suggests that pianists whose cueing experience comes from conducting, rather than ensemble-playing, may be skilled at leading but less skilled at following. In the literature, it has been suggested that good duo coordination may depend on at least one member of the pair having good ensemble “following” skills, which include strong anticipation and timing adaptation abilities (Keller, 2008).

The reduced form of the gesture-following task presented musicians with a situation that differed substantially from normal duo performance. Followers were forced into an exclusive follower role, in which their responses had no effect on leaders’ behaviour. Normally, people attempting to synchronize their actions will adapt to each other, regardless of their assigned role (Goebl & Palmer, 2009; Konvalinka, Vuust, Roepstorff, & Frith, 2010). Followers were also forced to rely exclusively on leaders’ movement cues to predict piece onsets, while in normal performance conditions, a range of other cues would be available, including facial expressions and the sound of breathing. Pianists who participated in the interactive duo performance task were not told how their performances would be presented during the gesture-following task, so their movements were natural, and not deliberately exaggerated. Future research should investigate how much use musicians make of their co-performers’ gestures when other cues are available. It should also be noted that the performance arrangements to which this study is most relevant are those in which performers have a direct view of each other, and in the future, other viewing angles should be considered.

The results of this study show that acceleration patterns communicate beats in skilled musicians’ cueing-in gestures. Musically-trained observers aligned their performed beats with the periods of sharp deceleration that followed acceleration peaks. The communicative quality of cueing gestures depended on their smoothness, magnitude, and prototypicality; both ensemble performance and conducting experience improved the quality of cueing gestures given.

These results may have implications for artificial musical systems employing gesture modelling and recognition functions. In recent years, there has been increased interest in the development of systems that people can interact with musically, some of which use gesture recognition to modulate sound output. Identification of which kinematic landmarks in musicians’ gestures indicate beat positions would benefit systems designed to synchronize discrete output with users’ rhythmic body movements (Dahl, 2014). Some recently-developed systems, designed to fill the role of an accompanist or duet partner, output music in real time to accompany human performances. Output timing is continuously adjusted in response to fluctuations in the human performer’s timing to maintain coordination. There have been some attempts to introduce expressive and receptive visual communication capabilities to such systems in the form of prescribed gestures, given at prescribed times (Maezawa & Yamamoto, 2016), but an improved understanding of how beats are communicated naturally and how to communicate beats clearly would allow for more robust and natural communication between the system and the human performer.