A brief glimpse of another person leads to a variety of spontaneous categorizations, including sex, race, and age. Recent work has investigated the temporally extended process underlying such social categorizations. For example, recent models have argued that a target’s multiple facial cues converge the moment that they become available during visual processing to weigh in on multiple partially active category representations (e.g., male and female). These partially active category representations then continuously complete over time to form a stable, integrated perception (Freeman, Ambady, Rule, & Johnson, 2008).

The basis of this proposal lies in that, at the neural level, a representation of a social category would be reflected by a pattern of activity across a population of neurons. As such, categorizing a face would entail ongoing changes in a pattern of neuronal activity. Monkey neurophysiological work has shown that within 100 ms neuronal populations reflect approximately half of a face’s visual information, with the remaining half accruing over the following hundreds of milliseconds (Rolls & Tovee, 1995). Thus, in early portions of the process, there would be a partial commitment to multiple categories because only a rough gist would be available. As more features are accrued, the pattern of neuronal activity would gradually sharpen into an increasingly confident representation (e.g., female), while other competing representations (e.g., male) would fall out (Freeman & Ambady, 2011a; Grossberg, 1980; Spivey, 2007). In some cases, top-down factors (e.g., stereotypes or goals) might also help determine the pattern to which the system gravitates, thereby influencing categorization.

Such an account was recently proposed by Freeman and Ambady (2011a) in their dynamic interactive model (DIM), a recurrent connectionist model of social categorization. A primary aim of the model is to explain how the bottom-up extraction of facial, vocal, and bodily cues drives categorization, which is simultaneously constrained by top-down social cognitive processes. Its assumption of a gradual accumulation of perceptual evidence shares similarities with other categorization models in the cognitive literature, such as the extended generalized context model (EGCM; Lamberts, 2000), as well as a wealth of stochastic accumulator, random-walk, and diffusion models (e.g., for a review, see Usher & McClelland, 2001). Besides focusing on social percepts, DIM differs from such models in that (a) category representations are constructed not just by the continuous processing of features, but also by the modulation of top-down social factors, and (b) consistent with other connectionist categorization models (Usher & McClelland, 2001), categories compete through lateral inhibition rather than accumulate evidence independently.

The initial evidence for DIM came from a mouse-tracking paradigm. Participants typically categorize faces by moving the mouse from the bottom center of the screen to one of two responses in either of the top corners. When categorizing faces that have partial overlap with the opposite category, the model predicts that mouse trajectories would exhibit a continuous, partial, and parallel attraction toward the opposite-category response. For example, when categorizing category-atypical targets (e.g., a male face with slight feminine features), trajectories should exhibit a partial attraction toward the female response before settling onto the male response. A number of studies have demonstrated these continuous-attraction effects in sex, race, and age categorization, which have been taken to reflect partially active category representations smoothly competing over time until stabilizing onto a single categorization (Freeman & Ambady, 2011b; Freeman et al., 2008; Freeman, Pauker, Apfelbaum, & Ambady, 2010).

One problem with such studies, however, is that features have been assumed to be sampled over time through a uniform process, with categories gradually activating and resolving over time. It is possible, however, that a continuous competition need not always be a smooth one. Most readers are familiar with instances in which we nearly fully commit to one category interpretation, but after further analysis, realize that the person belongs to a different category. Consider, for instance, a long-haired man. Initial visual analysis may give rise to a nearly 100%-confident interpretation that the target is a woman. Hundreds of milliseconds later, however, further analysis of the internal facial features may indicate that the target is actually a man, leading to a rapid shift in interpretation. Thus, it may be possible that in some cases abrupt processing shifts are required in order to integrate another’s features into the categorization process, as they dynamically become available.

Indeed, Lamberts’s work with the EGCM has shown that such nonuniformity may arise due to multiple features (Lamberts, 2000): Some features may be highly salient but not very diagnostic of the categories in question, whereas other features may be highly diagnostic but not very salient. Accordingly, salient features will tend to be incorporated into a categorization more quickly, sometimes leading to early incorrect response tendencies. For instance, in cases in which a salient feature was associated with the opposite category and participants were signaled to respond very quickly, responses were consistently biased toward the opposite-category alternative. When participants were signaled to respond later, however, correct responses were restored (e.g., Lamberts, 1995). Other studies manipulating exposure time rather than signal-to-response time converged on similar results. Macrae and Martin (2007), for example, conducted a priming study with short- and long-haired male and female faces, finding standard facilitation effects after a long exposure but category-reserved facilitation after a short exposure.

However, it is currently unclear how the early processing of salient features and the later processing of less salient (but potentially more diagnostic) features coordinate within a single instance of a natural, unconstrained categorization. Moreover, besides features that are especially salient, some features are also accrued earlier due to mere chance, because visual–perceptual processing is inherently probabilistic (Hegdé, 2008; Lamberts, 2002). Thus, if participants were given unlimited exposure and unlimited time to respond, would the early processing of some features nevertheless elicit early commitments to an incorrect category, and if so, how would these be overcome to yield correct categorizations?

Extant research, which initially led to the DIM, has answered this question by arguing that features cascade activation onto competing category representations, which smoothly resolve over time. Consider again the case of a long-haired man. In the mouse-tracking paradigm, extant research has predicted that participants’ hand movements should exhibit a continuous partial attraction toward the “female” response before smoothly settling into the “male” response, which has been shown in a number of studies (Freeman & Ambady, 2011a; Freeman et al., 2008). In some instances, however, one might reasonably expect that an early commitment to an incorrect category might gain so much confidence (e.g., ~100% female) that it must be abruptly intervened on and corrected, once additional sex-specifying cues are able to be processed. Thus, integrating the later accrual of one set of facial features into categorization may require an abrupt break from the earlier accrual of another set of features. In the mouse-tracking paradigm, this would be evidenced not by a continuous partial attraction toward the incorrect-category response (which is evidence of smooth competition), but by an initial straight pursuit toward the incorrect response, followed by a later abrupt redirection straight toward the correct response (see Supplemental Fig. 1 and Resulaj, Kiani, Wolpert, & Shadlen, 2009).

The present research

The present work was designed to investigate whether smooth competition may not always be sufficient, and whether abrupt shifts are sometimes required to integrate a face’s multiple cues. First, such shifts were examined using computer-generated faces whose conflicting cues could be controlled (Study 1) and then using real faces that would permit greater generalizability (Study 2), each with a mouse-tracking paradigm. Overall, it was predicted that as facial cues became more conflicting, abrupt shifts would become increasingly more likely. Importantly, in cases not involving such shifts, it was predicted that conflicting cues would nevertheless trigger smooth competition, demonstrating the coexistence of both abrupt shifts and smooth competition.

The design also reflected an interest in the context-dependent nature of these shifts. Whereas some facial cues are highly diagnostic of a category, others are less diagnostic but nonetheless reliable indicators (see Schyns, 1998). Short hair, for example, is a cue that relates to the male category, and long hair a cue that relates to the female category (Goshen-Gottstein & Ganel, 2000). Nevertheless, there are many exemplars of short-haired women and long-haired men. If perceivers were to encounter an “alternative” social context, in which short hair covaried with women and long hair covaried with men, could hair cues be flexibly reassociated with the opposite category? If perceivers were to adapt to such a counternormative context, it would be the normatively sex-typical targets—the short-haired men and long-haired women—that would elicit the greatest competition. This malleability to the social context was additionally explored.

Study 1

In the present study, participants categorized the sex of faces varying along a morph continuum in a mouse-tracking paradigm.

Method

A group of 32 undergraduate participants were recruited in exchange for partial course credit or monetary compensation. The face stimuli comprised eight computer-generated face identities morphed along an eight-point continuum from very male (morph −4) to very female (morph +4), excluding the 50%/50% ambiguous face (morph 0), using FaceGen (see Fig. 1). On each of the 64 trials, participants clicked a “Start” button located at the bottom center of the screen, which was then replaced by the face. Faces were presented in randomized order and categorized by clicking a “male” or “female” response in the top-left and top-right corners of the screen (see the supplemental materials). Which category appeared on the left versus the right was counterbalanced across participants. If a trial’s time exceeded 3,000 ms, a “time out” would occur and the data for that trial would be discarded. As in previous research (e.g., Freeman et al., 2010), if participants initiated movement later than 400 ms following face presentation, a message would appear after the trial encouraging them to start moving earlier on future trials. This encouraged early initiation and ensured that participants’ movements were online with the categorization process. To record and analyze the trajectory data, the MouseTracker software package was used (Freeman & Ambady, 2010). The screen resolution was 1,024 × 768 pixels, and standard Windows mouse-sensitivity settings were used with an 800 DPI mouse (resulting in a speed of 0.00125 in./pixel); thus, it took 1.28 or 0.96 in. of wrist/hand movement to traverse the screen from left to right or top to bottom, respectively.

Fig. 1
figure 1

Sample stimuli. In Study 1, computer-generated faces varying along a morph continuum were used. In Study 2, real faces bearing a sex-typical or sex-atypical hair cue were used

Full size image

Results and discussion

For comparison, all trajectories were remapped rightward such that the selected response was at the top right and the unselected response at the top left. They were rescaled into a standard x, y coordinate space: top left at [−1, 1.5] and bottom right at [1, 0], leaving the start position of the mouse at [0, 0]. Trajectories were normalized (linearly interpolated) into 101 time steps (100 time bins) to permit averaging of their full length across multiple trials. To capture abrupt shifts in movement, x-flips were computed: that is, the number of times that a trajectory switched direction along the x-axis (Dale, Roche, Snyder, & McCall, 2008). To capture standard smooth, graded-competition effects, as had been done in previous studies, the maximum deviation (MD) was computed: the maximum perpendicular deviation (out of the 101 time steps) between the observed trajectory and an idealized response trajectory (a straight line between the trajectory’s start- and endpoints). This indexes the partial, simultaneous activation of a competing representation of the opposite category. Prior to analysis, every trajectory was plotted and checked for aberrant movements (e.g., uninterpretable looping). Aberrant movements and “time outs” totaled 1.8% of the trials, and these were discarded prior to analysis. See Freeman and Ambady (2010) for further details on the trajectory-analytic techniques and validation of the measures. A generalized estimating equations (GEE) regression approach was used to incorporate the trial-by-trial data while accounting for the intracorrelations among individual participants and the individual eight face identities (Zeger & Liang, 1986). In all of the GEE analyses, unstandardized regression coefficients are reported.

Before the analyses were conducted, a coder inspected every trajectory and marked those containing an abrupt horizontal reversal, in which an initial pursuit toward the incorrect response was abruptly redirected midflight toward the correct response. The coder was blind to the hypotheses of the study and to the trajectories’ stimulus and condition information. The marking of these reversals was also attempted using a quantitative method, in which trajectories exceeding an MD threshold of 0.9 (indicating a large deviation toward the incorrect category) were marked as reversals. This quantitative method exhibited high agreement with the subjective coding (intraclass correlation coefficient [ICC] = .83). To permit greater accuracy, the coding measure was used here. Reversals and nonreversals are plotted in Fig. 2a, and their x-coordinate velocity dynamics between 0 and 900 ms (in windows of 60 ms) are plotted in Fig. 2b. Note that the reversal trajectories exhibit a spike in velocity early on in the direction of the incorrect category. Between 120 and 420 ms, velocities were significantly different between the reversal and nonreversal trajectories (all ps < .003, Bonferroni corrected), solidifying evidence for the early nature of the reversal.

Fig. 2
figure 2

At the top are mean trajectories (a) and velocity profiles (b) for trials that were marked as having a reversal (white trajectory) or no reversal (black trajectory) in Study 1. At the bottom are mean trajectories (c) and velocity profiles (d) for trials that were marked as having a reversal (white trajectory) or no reversal (black trajectory) in Study 2

Full size image

Morph values were converted into absolute values and rescaled (−.5 = sex-specified, .5 = sex-ambiguous). The x-flips were regressed onto this measure of ambiguity, which revealed that trajectories exhibited more x-flips as a face’s sex increased in ambiguity, B = 0.46, Z = 9.67, p < .0001 (Fig. 3a). Similarly, regressing the trajectory reversals (0 = no reversal, 1 = reversal) onto ambiguity (using logistic regression) revealed that trajectory reversals increased with ambiguity, B = 1.87, Z = 8.06, p < .0001 (Fig. 3b). Thus, category ambiguity increased abrupt shifts in movement and trajectory reversals.

Fig. 3
figure 3

Results of Study 1. Panels a and b plot shifts in movement (x-flips) and trajectory reversals as a function of sex-category ambiguity, demonstrating greater abrupt shifts with increasing category ambiguity. Panel c plots the maximum deviation as a function of sex-category ambiguity only for trials that did not contain a trajectory reversal, demonstrating greater smooth graded-competition effects with increasing category ambiguity

Full size image

To examine the possibility of graded-competition effects on trials not involving full-blown reversals, trials without a reversal were examined using MD, a standard measure of continuous, graded competition (indexing simultaneous, partial attraction toward the opposite category). Regressing MD onto sex-category ambiguity revealed that nonreversal trajectories’ MDs increased with increasing ambiguity, B = 0.07, Z = 4.40, p < .0001 (Fig. 3c).

Thus, increases in ambiguity led to corresponding increases in full-blown trajectory reversals and in abrupt shifts in movement between categories. However, when trajectory reversals were removed, smooth graded-competition effects were nevertheless observed. The malleability of these coexistent competitive effects was examined in the next study.

Study 2

In the present study, participants sex-categorized real faces bearing a sex-typical or sex-atypical hair cue. In a “normative” task, participants were presented with a majority of sex-typical targets and a minority of sex-atypical targets (as generally occurs in the real world). In a “counternormative” task, however, participants were presented with an alternative social world: a minority of sex-typical targets and majority of sex-atypical targets. If perceptual cues can be flexibly reassociated with different categories as determined by the social context (i.e., long hair reassociated with men and short hair reassociated with women), then greater competition should be observed for sex-atypical targets in the normative task, but for sex-typical targets in the counternormative task. The cursor recording and settings were identical to those in Study 1.

Method

Undergraduate participants were randomly assigned to the normative (N = 34) or counternormative (N = 37) task. Photographs of 80 sex-typical (short-haired males, long-haired females) and 80 sex-atypical (long-haired males, short-haired females) faces were obtained from public-domain websites (see Fig. 1). These were grayscaled, standardized by brightness and contrast, and resized to 200 × 200 pixels. Hair was retained in the cropping, but all other extrafacial information was removed. In the normative task, 80 sex-typical and 20 randomly selected sex-atypical targets were presented; in the counternormative task, 80 sex-atypical and 20 randomly selected sex-typical targets were presented (each set contained equal numbers of males and females). The same mouse-tracking paradigm and preprocessing were used as in Study 1.

Results and discussion

Unlike in Study 1, the trials of Study 2 had correct responses associated with them (since the faces had a true sex category). Aberrant movements, “time outs,” and incorrect responses totaled 7.5% of the trials in the normative task and 8.0% of the trials in the counternormative task. These were discarded prior to analysis. Again, trajectories were marked for reversals using both subjective coding and the MD threshold measure, which showed high agreement (ICC = .88); the subjective codings were used for greater accuracy. Plots and velocity profiles, respectively, of the reversals and nonreversals appear in Fig. 2c and d. As in Study 1, the reversal trajectories exhibited a spike in velocity early on in the direction of the incorrect category, and this spike was significant between 120 and 540 ms (all ps < .003, Bonferroni corrected).

All x-flips and coded trajectory reversals were regressed (separately) onto target typicality (−.5 = typical, .5 = atypical), task context (−.5 = normative, .5 = counternormative), and the interaction. A significant effect of typicality indicated that trajectories for sex-atypical targets exhibited more x-flips than did those for sex-typical targets, B = 0.14, Z = 5.02, p < .0001. The effect of context was not significant, B = 0.03, Z = 1.10, p = .27. More importantly, the interaction was significant, B = 0.26, Z = 4.81, p < .0001 (Fig. 4a). Whereas sex-atypical targets induced more x-flips when in a normative context (simple B = 0.16, Z = 4.06, p < .0001), sex-typical targets induced more x-flips when in a counternormative context (simple B = −0.10, Z = 2.71, p < .01).

Fig. 4
figure 4

Results of Study 2. Panels a and b plot shifts in movement (x-flips) and trajectory reversals, respectively, as a function of target sex typicality and task context. Panel c depicts the maximum deviation as a function of target sex typicality and task context only for trials that did not contain a trajectory reversal

Full size image

The regression with coded reversals converged on the same pattern. There was a significant effect of typicality (B = 0.57, Z = 5.12, p < .0001), a nonsignificant effect of context (B = −0.04, Z = −0.35, p = .72), and a significant interaction (B = 1.22, Z = 5.53, p < .0001; Fig. 4b). Whereas sex-atypical targets were more likely to elicit a reversal in a normative context (simple B = 0.57, Z = 4.09, p < .0001), sex-typical targets were more likely to elicit a reversal in a counternormative context (simple B = −0.65, Z = 3.79, p = .0001).

As in Study 1, graded-competition effects were also inspected on trials not involving reversals. MD was regressed onto typicality, task context, and the interaction. This revealed a significant effect of typicality, with the trajectories for sex-atypical targets exhibiting higher deviation, B = 0.05, Z = 5.69, p < .0001. The effect of context was also significant, B = 0.03, Z = 3.86, p = .0001. More importantly, a significant interaction (B = 0.04, Z = 2.28, p < .05; Fig. 4c) indicated that, on trials without a reversal, sex-atypical targets induced higher deviation in a normative context (simple B = 0.06, Z = 4.03, p < .0001). For sex-typical targets, deviation did not reliably differ between the two contexts (simple B = 0.02, Z = 1.22, p = .22).

As in Study 1, abrupt shifts were found to coexist with graded competition effects.Footnote 1 Importantly, social context influenced these categorization dynamics. In a counternormative context in which sex-atypical targets were the norm, the targets with sex-typical rather than sex-atypical hair cues elicited the greatest competition. This suggests that the social categorization system rapidly reassociated perceptual cues with another category in order to adapt to the social context.

General discussion

To form coherent perceptions of others, perceivers must rapidly integrate a variety of visual cues into rigid categories. The authors of previous work have argued that this involves partially active categories smoothly competing over time (Dale, Kehoe, & Spivey, 2007; Freeman et al., 2008). The present results suggest that in some cases smooth competition may be insufficient, and that instead abrupt cognitive shifts are sometimes needed, as evidenced by trajectories’ horizontal reversals. These shifts were more likely to occur as internal facial cues became increasingly ambiguous (Study 1) or when a peripheral cue, hair, was encountered in a task context in which it was not the norm for a given sex (Study 2). Thus, in some cases the integration of another’s facial cues requires a break from the dynamics of earlier integration, leading to initial commitments to the incorrect category that must be rapidly reversed later in processing.

These results are consistent with previous work finding that the early processing of specific features may lead to early incorrect response tendencies (e.g., Lamberts, 2000; Macrae & Martin, 2007). However, it has remained unclear how the transition from early incorrect tendencies to later correct tendencies unfolds within a single instance of categorization. The present studies suggest that this transition may be accomplished by either smooth competition or abrupt corrections, both of which increase with category atypicality. Future research could use mouse-tracking to examine the specific factors that determine smooth versus abrupt competitive effects on a trial-by-trial basis, which could bear numerous implications for models of both basic and social categorization.

One possibility is that the abrupt category shifts found here reflect dual processes in categorization (e.g., one coarse and one fine-grained). Such dual-process models have been prominent in research on recognition memory (Atkinson & Juola, 1973) and on distinguishing basic-level versus subordinate-level categorization (Jolicœur, Gluck, & Kosslyn, 1984). The shifts could also be explained by a single process responsible for categorization (e.g., modeled by DIM), along with a separate correction process involved in conflict monitoring that runs in parallel with continuous stimulus processing (e.g., Botvinick, Braver, Barch, Carter, & Cohen, 2001). That said, dual/separate processes may not be needed to explain the results. In dynamical systems such as the system proposed by DIM, such rapid shifts in the system’s trajectory are known as phase transitions and naturally emerge out of the system’s dynamics; they are theorized to be important for a variety of cognitive processes (Spivey, Anderson, & Dale, 2009). A challenge for current models of social categorization will be to show how both smooth competition and abrupt shifts emerge in the same task to integrate another’s facial cues. For now, the results provide an important refinement to previous work by showing that, although smooth competition often underlies social categorization, abrupt shifts are in some cases required, as well.

In Study 2, the categorization process adapted to the context. In a task in which sex-typical targets were the norm, sex-atypical targets elicited the greatest competition. However, in a task in which sex-atypical targets were the norm, the sex-typical targets instead elicited the greatest competition. This suggests that category-atypical cues were flexibly reassociated with the opposite category in order to adapt to the context: long hair with the male category, and short hair with the female category. Although previous work has documented contextual effects on a number of categorization outcomes, the present results provide new evidence that the underlying categorization dynamics and interrelations among cues and categories themselves are also rapidly altered by social context. As such, the results support recent models positing interactions between bottom-up sensory cues and top-down factors that drive social category dynamics in real time (Freeman & Ambady, 2011a).

Beyond the theoretical implications, the present work opens many questions for future research. Studies could explore individual differences in tendencies for smoother versus more abrupt processing modes, and how such differences might relate to perceivers’ conceptions of social categories, among many other factors. Researchers have placed a great deal of importance on social categorization because it powerfully shapes interpersonal interactions and has cognitive, affective, and behavioral effects (Macrae & Bodenhausen, 2000). The present work suggests that a given instance of categorization may involve qualitatively distinct styles of processing, and thus it will be important for future research to examine how these different processing styles might trigger different kinds of downstream social consequences.

In sum, smooth competition may not always be sufficient to handle the slight category incoherencies that we naturally encounter out in the social world. Instead, abrupt processing shifts may sometimes be required to bring another’s diverse visual cues into coherent perceptions.