What is the scanpath signature of syntactic reanalysis?

Abstract

Which repair strategy does the language system deploy when it gets garden-pathed, and what can regressive eye movements in reading tell us about reanalysis strategies? Several influential eye-tracking studies on syntactic reanalysis (Frazier and Rayner, 1982, Meseguer et al., 2002, Mitchell et al., 2008) have addressed this question by examining scanpaths, i.e., sequential patterns of eye fixations. However, in the absence of a suitable method for analyzing scanpaths, these studies relied on simplified dependent measures that are arguably ambiguous and hard to interpret. We address the theoretical question of repair strategy by developing a new method that quantifies scanpath similarity. Our method reveals several distinct fixation strategies associated with reanalysis that went undetected in a previously published data set (Meseguer et al., 2002). One prevalent pattern suggests re-parsing of the sentence, a strategy that has been discussed in the literature (Frazier & Rayner, 1982); however, readers differed tremendously in how they orchestrated the various fixation strategies. Our results suggest that the human parsing system non-deterministically adopts different strategies when confronted with the disambiguating material in garden-path sentences.

Introduction

Eye tracking is a very productive methodology in sentence processing research. Beginning with classic work such as Just and Carpenter, 1980, Frazier and Rayner, 1982, reading studies involving eye tracking continue to provide a rich array of empirical evidence that inform competing theories of human sentence parsing. This entire body of work rests on some degree of belief on an assumption articulated first by Just and Carpenter (1980), the eye-mind assumption. As they put it (p. 331): “there is no appreciable lag between what is being fixated and what is being processed.” Taken literally, this assumption is clearly false; this is evident from two facts: (i) preview effects which indicate that processing of a word can start even before the eyes fixate it for the first time (Rayner, 1998, Rayner, 2009); (ii) the spillover-effects where processing initiated at one word can continue even after the eyes move to fixate another word (Rayner & Duffy, 1986). We can therefore safely assume that no eye tracking researcher believes in the strict formulation of the eye-mind hypothesis. At the other extreme, if we were to assume that the eye-mind assumption is completely false, then the eye movement record would be difficult to interpret because fixation durations would have no straightforward relationship with processing difficulty. Clearly, this extreme position is also untenable given the largely replicable findings in the sentence comprehension literature (cf. Clifton, Staub, & Rayner, 2007, for a review of the empirical results).

This leaves us with an intermediate version of the eye-mind assumption: fixation durations reflect processing difficulty, but lags in processing and constraints arising from oculo-motor control (Rayner, 1998) have the potential to complicate the interpretation of the eye movement record. Indeed, theories of eye movement control such as E-Z Reader 10 (Reichle et al., 1998, Reichle et al., 2009) standardly assume such a lag.

A particularly interesting situation arises when the eyes, instead of making the prototypical forward saccade, carry out a regression. Such a regressive eye movement is interesting from the parsing perspective because it could be driven at least in part by parser actions that began when the launch site of the regression was fixated. The question then arises: to what degree are the eyes coupled to and directed by the parser’s actions? Are they tightly coupled (as the strict form of the eye-mind assumption would assert), are they completely uncoupled, or is there a loose coupling (as the intermediate form of the eye-mind assumption would predict)? Moreover, the degree of coupling could be modulated and depend on, for instance, the particular type of processing difficulty encountered by the parser.

This question has been the focus of a trio of papers by Frazier and Rayner, 1982, Meseguer et al., 2002, Mitchell et al., 2008. Frazier and Rayner suggested that when the parser initiates a reanalysis action, detaching a constituent from the incremental tree built so far and attaching it to another part of the tree, the eyes carry out a regressive saccade to follow the parser’s actions: as the parser intelligently searches for an alternative attachment site in the sentence, the eyes follow along. For example, in the sentence Since Jay always jogs a mile seems like a very short distance to him the noun phrase (NP) a mile is initially mis-attached to the verb jogs as its direct object; when the next word, seems, is processed, a reanalysis process begins whereby the NP is reattached as a subject of a main clause headed by the verb seems. Frazier and Rayner named this intelligent reanalysis process Selective Reanalysis, and argued that the eyes closely follow the parser’s processing steps: “The selective reanalysis hypothesis predicts that eye movements should regress from the disambiguating region to the ambiguous region of the sentence” (p. 204).

Note, however, that Selective Reanalysis does not presuppose the strict form of the eye-mind assumption; it is consistent with the intermediate version of the eye-mind assumption, since the eyes could be following—with a lag—the parser’s repair actions.

Twenty years after the Frazier and Rayner proposal, Meseguer et al. presented more evidence for Selective Reanalysis from Spanish. They examined garden-path sentences in Spanish where an adverbial phrase could attach high or low. They found evidence in favor of Selective Reanalysis (this work is discussed in detail below). In subsequent work, Mitchell et al. (2008) challenged the idea that the eyes regress in lock-step with the parser’s actions.

Mitchell et al.’s alternative proposal was that the cognitive system that drives parsing may want to avoid moving forward to take in new information when reanalysis is triggered. It therefore takes a “time-out,” which results in a regression (presumably because moving to the right would bring new information in, which is undesirable in the face of increased processing load).¹ In their analysis, Mitchell et al. were also concerned with eye movement patterns, but relied on “regression signatures” instead of qualitative assessments as Frazier and Rayner did; these are probabilities of the eye landing on any word preceding the word from which the regression started.

In these three papers, the crucial evidence for (and against) the coupling of eye movements with parsing actions dictated by Selective Reanalysis hinged on analyzing eye movement patterns rather than just fixation durations or regression probabilities; scanpaths are the principal object of inquiry. This makes sense because the question literally is: what patterns of regressive eye movements result when reanalysis begins? (Eye-tracking researchers often refer to ‘patterns’ of eye movements where they really mean fixation durations; in the present case, ‘patterns’ stands for scanpaths.)

Since there exists no suitable quantitative way to evaluate the similarity of one eye movement pattern with another, the three sets of authors mentioned above were forced to either look at scanpath patterns qualitatively (e.g. Frazier & Rayner, 1982, p. 196) or to reduce scanpaths to scalar duration measures and transition probabilities in order to derive conclusions about participants’ behavior (e.g. Frazier & Rayner, 1982, pp. 199–200). Due to the unavailability of a method for quantitatively studying spatio-temporal fixation patterns, subjective descriptions were necessary. It is worth quoting one such description (Frazier & Rayner, 1982, pp. 196–197) to underline the fact that a major issue of interest is indeed eye movement patterns, i.e. scanpaths, and not only transitional probabilities:

…three or four patterns of eye movement behavior occurred which we shall attempt to characterize. In some cases, subjects read the ambiguous noun phrase and upon reading the disambiguating region made very long fixations. These long fixations were also accompanied by very short saccades … Upon reading the end of the sentence, the subject then made a long regression to the beginning of the sentence and reread the sentence. The long fixations and short saccades in the disambiguating region and thereafter may also have been accompanied by short regressions, but the reader did not regress at that point back to the beginning of the sentence or to the ambiguous region. We shall characterize this behavior as chaos in that the reader apparently was having great difficulties understanding the sentence but seemed to have no insights as to what the nature of the processing difficulties were. This pattern of eye movements was particularly noticeable among three of the subjects and occurred less frequently with most of the other subjects.

Thus, it is clear that the debate about how eye movements are driven by a sequence of parsing actions needs a method for characterizing scanpath patterns and their relative similarities to each other or to theoretically proposed patterns of regressions.² In this paper, we provide such a method, along with freely available software for exploring eye movement patterns. We also reanalyze Meseguer et al.’s data (which they generously provided to us) using this method to demonstrate the gain in information when we rely on a formal characterization of scanpath patterns rather than qualitative evaluations of scanpaths and regression signatures. One contribution of this paper is to provide such an additional analytical tool for directly evaluating scanpaths where these are at issue theoretically.

The present investigation was motivated by two main aspects of the above-mentioned studies on reanalysis. First, although the analytical methods used by Frazier and Rayner, Meseguer et al. and Mitchell et al. are undoubtedly informative, it is possible that aggregated eye-tracking measures are misleading because they could in principle arise from a blend of several classes of fixation patterns. Increased transition probabilities in one condition might be caused by changes in different underlying populations of scanpaths. For example, if there is a higher probability of transitions from region 9 to region 2 and from 2 to 7, we cannot decide if this was caused by patterns that went with transitions 9 → 2 → 7 or by one class of patterns with occurrences of transition 9 → 2 and another with transitions 2 → 7. In principle, one could calculate transition probabilities that are conditional on the previous transitions. In a typical reading study, however, there are hardly enough data points to reliably estimate unconditioned transition probabilities, let alone conditional probabilities. In general, if the aggregated data stem from different populations, it can be very difficult to infer anything precise about the various strategies used by the reader as they relate to eye movement patterns.

Second, in both the Mitchell et al. and Meseguer et al. studies, many regressions ensued after reading the disambiguating material in the non-reanalysis condition. In fact Meseguer et al. recorded 700 regressions in the reanalysis condition and 667 in the non-reanalysis condition. This means that most regressions produced by participants cannot plausibly be explained by any of the competing theories, which predict that long-range regressions occur only in the garden-path condition. The observed regression patterns are, however, only problematic if we assume a strictly deterministic parser that always adopts the preferred structure during ambiguous segments of the sentence. In this framework garden-pathing takes place in only one condition; Frazier and Rayner (1982) made this assumption explicit by invoking the late closure and minimal attachment principles (Frazier, 1979) in order to explain the occurrence of regressive eye movement patterns in reanalysis conditions.

Evidence for Selective Reanalysis clearly exists in the above-mentioned data, but this evidence comes in the form of a slightly increased probability of regressions to particular words. However, if the eye-mind assumption—which is the basis for reasoning in favor of Selective Reanalysis—holds, then we have to ask what process is driving the numerous regressions in the non-reanalysis conditions. Even the Time-out hypothesis, which seems to partly reject the eye-mind assumption in the case of regressions, has not much to offer when it comes to explaining regressions in the non-reanalysis condition. Thus, the data suggest that there is a lot more to the issue of regressions than meets the eye.