The current experiments investigated how visual motion perception and concurrent movement production mutually interact. Participants were asked to produce hand movements (without visual feedback) while observing the motion of an independent stimulus. By varying the direction of the movements relative to the direction of the motions, it was possible to show that, on average, perceived directions were repelled by produced directions and produced directions were repelled by perceived directions. As far as we know, this is the first time that CEs in action and perception have been found in the same experiment. However, contrary to what was expected based on two recent models (Hamilton et al.,
2004; Schubö et al.,
2001), the size of the CEs in production and perception did not covary across or within participants, nor did the size of the effects systematically decrease with an increase in angular separation between the movement and motion directions. Interestingly, the CE in perception actually turned into an AE when the angular separation increased beyond 90°, that is when the movements were produced with a downward component while the motions had an upward component. The only prediction of the Schubö and Hamilton models (Table
1) that found empirical support was the mutual interference prediction, which states that influences on action and on perception should co-occur.
For economical reasons the influence of concurrent motions on the CEs in production was only tested in Experiments 1 and 2. However, given the null results in Experiments 1 and 2 it is very unlikely that a monotonic relationship had been observed if CEs in production had also been measured in Experiments 3 and 4.
While movements were employed in action and in perception in the current study we do not think that overlap on another dimension than direction is necessary for specific interference effects to show up. For example, the same effect would be expected if a line instead of a moving dot was used. Indeed in a similar but unpublished experiment we also found a CE in action with a static line. In addition, also a static object without orientation information induces CEs. This is because its location specifies a direction. For example, in a study by Tipper et al. (
1997) hand movements were influenced by stationary distractors. Tipper and colleagues accounted for their results by proposing that representations of the performed hand movements to the target and of the potential hand movements to the distractors interfered. We differ from this account, based on our findings of interference also in perception, in that we suggest that interference takes place between the representations of
perception and action events. Importantly, the reported effect is not unspecific, that is, effecting a general shift in one direction rather it shifts rightward hand movements to the right while leftward hand movements are shifted to the left.
In the experiments movement direction was blocked. This was an attempt to minimize interference by non-motor representations which could occur if visual cues for movement directions were used. Therefore, the current experiments cannot exclude the possibility that the categorical influence (i.e., CE size did not depend on the amount of overlap, see below) was caused by the small amount of reprogramming or the short movement time that was needed for task performance, which might have prevented development of detailed spatial representations.
2 Although, in the end, this is an empirical question and it would be interesting to look at it in future studies, there are two reasons why this does not challenge our explanation. First, while it is true that movements were always performed only in one direction during a block, participants were still required to produce a specific direction and as can be inferred from Table
2 succeeded in doing so. Additionally, they were retrained on the specific direction at the beginning of each block. Therefore, there is reason to believe that participants encoded the specific direction; simply coding, for example, ‘right upward’ would not suffice to achieve the criterion. Second, even if coding of categorical movements was caused by the small need to reprogram, the current findings are still interesting, because they show that interaction between action and perception can take place at a categorical level and that categorical coding seems to be the preferred coding if possible. If the need to invoke more detailed representations added to this interference effect an amount of overlap component would then be a question of further studies in which different directions are required for each movement. Interestingly, we are not aware of any study that reported such an effect. The problem with such a paradigm would be, how to measure an effect on action, given the large expected variability when movements change from trial to trial (simply having two movement directions would probably not suffice to exclude categorical coding). It is also unlikely that longer movement times would add an amount of overlap component to the CE, given that comparing Experiments 1 and 2 of Grosjean et al. (
2008) suggests that if anything, the size of CE in action decreases with longer movement times.
The size of the effects (<1°) obtained in the present study are somewhat smaller compared to that of related effects, such as the induced shift of a moving dot by surrounding tilted lines (e.g., Westheimer,
1990). Arguably, this limits the practical relevance of such phenomena to situations in which visual motion perception and movement production require high levels of accuracy (e.g., endoscopic surgery). Nonetheless, the generality and robustness of these effects have now been demonstrated in a number of tasks and for a variety of perception-action dimensions (e.g., amplitude, weight, and direction; Hamilton et al.,
2004; Müsseler & Hommel,
1997; Schubö et al.,
2001,
2004; Zwickel et al.,
2007,
2008), which has made them very useful for testing models of the perception-action interface. Importantly, despite the small size of the effect, the CE was obtained at a high level of significance which makes it unlikely that we failed to support the hypotheses because of too low statistical power. Additionally, amount of overlap
had an influence for an angular separation of more than 90° but this influence was in opposition to the prediction of the models. We now consider whether other models of perception-action, action-action and perception-perception interactions could explain the present findings.
Two particularly relevant models have been proposed by Tipper et al. (
1997) and Welsh and Elliott (
2004) to account for perception-action interference effects in pointing tasks. Tipper et al. (
1997) observed that hand movements to target objects deviated away from near distractor objects but toward far distractors. Repulsion from the near distractor was attributed to the inhibition of a potential movement to the distractor location, the distributed representation of which was assumed to overlap with the representation of the movement to the target. As in the Schubö model, which was inspired from this model, the inhibition of overlapping representational elements was posited to lead to a shift of the resultant reach away from the close distractor. To account for the attraction to the far distractor, it was further assumed that the amount of inhibition was proportional to the saliency of the distractor object, with saliency being related to the proximity of the distractor to the moving hand. Therefore, far distractors led to less inhibition than near distractors. As a consequence, the resultant reaching movement will actually combine representational elements of reaches to the target and distractor, thereby leading to an attraction effect for far distractors. This model could, in principle, account for the change from repulsion to attraction that was observed in the present experiments. However, it also predicts that the amount of repulsion should gradually diminish with smaller amount of overlap, which is inconsistent with the current results.
The response activation model of Welsh and Elliott (
2004) is similar to the model of Tipper et al. (
1997) in that it postulates that interference effects arise from the parallel activation of reaches to the target and distractor. It differs from it, however, in that whether attraction or repulsion occurs depends not on the spatial but on the temporal relationship between distractor and target stimuli. If distractor onset precedes target onset by enough time, the distractor response can be inhibited and repulsion arises. If, however, the target follows the distractor very close in time, the amount of inhibition is not large enough to prevent a combined response and attraction occurs. The time course assumptions of the response activation model cannot, however, explain the present pattern of results either. Namely, the absence of an amount of overlap effect for angular separations smaller than 90° and the reversal of the effect beyond this value.
It is interesting to note that the horizontal axis (midline) has been found to play a special role in line copying tasks (Meulenbroek & Thomassen,
1991,
1992; Van Sommers,
1984). In these tasks, participants had to produce rapid small back-and-forth movements in directions that were self-selected from a set of possible directions (Meulenbroek & Thomassen,
1992) or instructed by the experimenter (Meulenbroek & Thomassen,
1991; Van Sommers,
1984). In both cases, the possible directions were visually shown. Participants movements had a tendency to veer away from the horizontal, that is, to produce lower orientations than required when producing movements below but close to the horizontal, and higher orientations than required when producing movements above but again close to the horizontal. These results underline the qualitative difference between movements produced below and above the horizontal. It is unclear, however, what predictions about the CE could be derived from these results, given that in the current experiment the visual motions did not coincide with the required directions. What is more, any general form of repulsion from the horizontal would be subtracted away by comparing motion and no-motion trials and would not show up in the present effects.
For action-action interactions, interference effects (AE) that are influenced by amount of overlap have also been reported. For example, Swinnen, Dounskaia, Levin, and Duysens (
2001) asked participants to produce vertical movements with their left hand while concurrently performing rapid movements in certain directions with their right hand. The results showed that the movement of the left hand was biased in the direction of the concurrent right hand movement. The AE was largest for orthogonal directions and smallest for vertical movements with the right hand. Amount of overlaps between these two extremes led to intermediate effect sizes. This pattern of interference was attributed to the spread of neural activity via interhemispheric connections. When the two directions were similar and therefore led to the activation of similar movement population vectors (representations) in each hemisphere, spreading activation did not lead to a strong change in direction coding. Less similar vectors, however, resulted in more interference. Thus, this model predicts an influence of amount of overlap as well.
Amount of overlap effects were also observed for perception-perception interactions (e.g., Marshak & Sekuler,
1979; Westheimer,
1990). For example, in Marshak and Sekuler (
1979)’s experiment, participants watched random dots that moved in two different directions. One of these directions was always horizontal. Participants’ task was to judge the direction of the other motion. The results showed that this motion was perceived as being repelled from the horizontal motion. Additionally, the size of the repulsion decreased with larger angles between the two motions. Mahani, Carlsson, and Wessel (
2005) proposed a model to account for such perceptual interference effects in which motion repulsion is considered a side-effect of clustering algorithms (see also Navalpakkam & Itti,
2007). The underlying idea is that when concurrent features are assigned to two different tasks, features of the two tasks might become mixed. According to Mahani et al. (
2005) the probability of misclassifying features of a given motion is higher for features that are less typical for one motion and more typical for the other motion. This loss of “untypical” and the gain of more “typical” features leads to a CE. However, this mechanism would also predict an amount of overlap effect.
Given that amount of overlap effects have been found in action-action and perception-perception paradigms but not in the current paradigm, it seems that interference effects between perception and action differ from interference effects within perception and action. In the following we will argue that this difference depends on the kind of representations that are inhibited. Specifically, we will argue that while detailed spatial relationships are preserved for action-action and perception-perception interactions, this information is “lost” when functionally independent actions and perceptions interfere. Instead, interactions between perception and action are mediated by the activation of categorical representations. Indeed, to the best of our knowledge, there is only one study that reported an interference of detailed spatial representations between action and perception (Ehrenstein, Cavonius, & Lewke,
1996). Importantly, in this study, action and perception were not functionally independent (Zwickel et al.,
2007,
2008), which led to an AE.
A categorical account
One way to account for the the absence of an amount of overlap effect within the range of about 90° is to assume that, although specific angles are represented at some level of the system, interference arose between categorical representations of the movement and motion directions. For example, all hand movements within the upper-right quadrant of the graphics tablet may have led to the activation of the same “right upward” representation, irrespective of the specific angle that was required. Similarly, stimulus motions that moved roughly vertically may have always activated the same “upward” representation. Therefore, at this level of the system, the amount of overlap manipulation in Experiments 2 and 3 did not lead to activation of different representations. This means that, for example, the 35° movement to the right led to the same activation of the “right upward” representation as the 10° movement to the right did. Given the same activation in these two cases no difference in interference size would be expected. In addition, no correlation between the CE sizes in perception and action would be expected as the variance of the CE sizes in action and perception would only be caused by additional sources of variance that are unrelated to the processes and representations that lie at the origin of the interference effects themselves. In other words, a large CE in perception would not be caused by the a difference in representation at the level where action and perception interfere but by earlier or later processing. While this would explain why no influence of amount of overlap and no monotonic relationship would be expected, the next section will deal with the issue why mutual interference should still arise in this model.
While movement production and motion perception would activate one categorical representation at the level of interference, these categories themselves could be represented in a distributed fashion, such that, for example, activating the category “upward” would involve the activation of elements that code for motions slightly to the right and left of vertical as well. In this way, upward movements to the left or right would lead to the activation of a common set of elements with upward motions, but the amount of representational overlap would not depend on the specific directions involved. By assuming, as did Schubö et al. (
2001), that these common elements are inhibited, a CE would arise and it would not vary in size as long as the movement has an upward component.
For downward movements, however, a different categorical representation, such as “right downward”, could have been activated. Given that it would not overlap with the representation of an upward motion, no inhibition would be necessary to keep the motor and perceptual activities separate. However, as discussed above (Mahani et al.,
2005), some elements between the perceptual and motor representations may be exchanged by chance, leading to assimilation. Interestingly, this would predict that the size of the AE should increase with an decrease in amount of overlap because a random exchange of elements from less similar movements and motions would involve an exchange of more dissimilar elements.
The involvement of categorical representations in perceptual processes is not new (e.g., Braine,
1978; Postma & Laeng,
2006) and spatial categories have already been shown to influence, among other things, the perception of locations (Huttenlocher, Hedges, & Duncan,
1991) and visual search times (Wolfe, Friedman-Hill, Stewart, & O’Connell,
1992). In Wolfe et al. (
1992), visual search times were found to be faster when distractors and targets came from different categories (e.g., tilted to the right vs. tilted to the left) than when distractors and targets belonged to the same category (e.g., both tilted to the right). The present results add to this by showing that categorical representations also play a critical role in determining how concurrent perception and action interact.
When similarity is linked to comparability, the present account is related to the explanation of contrast and assimilation advanced by Aarts and Dijksterhuis (
2002). Aarts and Dijksterhuis (
2002) found that priming individuals with a certain speed could have a contrastive or assimilative effect on later speed judgments. However, the direction of the effect depended on whether the prime and test stimuli were perceived as comparable or not. For example, whether priming individuals with a turtle led to higher speed estimates of a human depended on whether participants believed that animals and humans are comparable or not. When participants read about the similarity between animals and humans, a CE was observed. However, reading about differences between humans and animals led to AEs. In the current context, one might argue that CEs were observed as long as participants perceived hand movements and stimulus motions as similar or comparable. AEs resulted, however, when decreasing the amount of overlap led to no perceived similarity or comparability between what was produced and perceived.
Taken together with the findings reviewed in the previous section, the present results suggest that the nature of interference effects depends on the type of events that are involved. When both events come from the same domain, that is within action or perception, interference seems to arise between detailed, as opposed to categorical, representations. Two studies are especially informative in this regard because they showed that the interference between spatial features of stimuli and responses depends not only on their physical properties but is more flexible. Hommel (
1993) instructed participants to press one of two buttons in response to the frequency of a tone (low/high). The tones were presented at task-irrelevant left or right locations. In another condition, participants were instructed to turn on a light in response to the frequency of the tones by pressing a button on the opposite side of the light. Shorter reaction times were obtained for spatial correspondence between the locations of the tone and the button in the button condition, and between the locations of the tone and the light in the light condition. Thus, the interference effects were modulated by how the participants represented the goal of their actions (button vs. light location). Similarly, Stevanovski, Oriet, and Jolicoeur (
2002) showed that interference effects change as a function of whether a stimulus (e.g., <) is interpreted as an arrow head (i.e., pointing to the left) or a headlight (i.e., projecting to the right). Analogously, in the current experiments, the observed interference might have occurred at a level where physical properties of the stimuli and actions played a minor role. It is thus not surprising that other researchers have already suggested that categorical codes underly various types of interference effects (e.g., de C. Hamilton, Joyce, Flanagan, Frith, & Wolpert,
2005; Hommel,
1998; Kunde & Wühr,
2004; Lindemann et al.,
2006).
Assimilation versus contrast
One issue that has not been considered until now is that some studies reported facilitatory effects between produced and perceived events (e.g., Brass, Bekkering, & Prinz,
2001; Craighero, Fadiga, Rizzolatti, & Umiltà,
1999; Stürmer, Aschersleben, & Prinz,
2000; Vogt, Taylor, & Hopkins,
2003). However, in these studies, action and perception where not functionally independent from each other. For example, Vogt et al. (
2003) showed that grasping movements where performed faster when the go signal depicted a picture of a congruent versus incongruent grasp end position. Crucially, in all these studies, hand movements were performed in response to the interfering pictures and therefore action and perception where not functionally independent. There are also two studies that have reported AEs between two functionally independent produced and perceived events (Repp & Knoblich,
2007; Wohlschläger,
2000). Interestingly, their experiments differ from the current ones and those of Hamilton et al. (
2004) and Schubö et al. (
2001) in that they used
ambiguous visual or auditory stimuli. For example, Wohlschläger (
2000) investigated the influence of directional hand movements on the perception of an ambiguous motion that could be interpreted as rotating clock- or counterclockwise. What he found was that the direction of perceived motion was biased in the direction of the produced movement. The critical difference between this study and those considered so far could be that, under ambiguous stimulus conditions, one often relies on non-perceptual information to achieve a stable percept. This may have lead to more integration (assimilation) between perceptual and action-related information than in the type of experiments used here, where participants may have been able to keep their perceptual and motor tasks more separate from each other.
Similarly, Repp and Knoblich (
2007) found that the direction of movements on a piano keyboard influenced perceived changes in pitch of an ambiguous tone sequence. The tone sequence was more often judged as rising when participants moved from left to right than when they moved from right to left. However, this influence was obtained for expert piano players, but not for non-expert piano players. The authors suggested that this effect was related to the existence of overlearned movement-auditory effect associations for the experts. Given the relative novelty of the movements and motions used in the current study, the difference between the results of Repp and Knoblich (
2007) and the current ones could be related to their use of ambiguous stimuli and/or the involvement of overlearned movement-effect associations. That said, future research is necessary to resolve the apparent conflict between these effects and the type of effects considered here.