Concurrent visual sequence learning

In the standard AGL experiment (Reber, 1967), the participants are asked to observe short strings of stimuli that are, unbeknownst to them, derived from a complex set of rules, the grammar. Afterwards a test phase follows. The participants receive new strings that either follow or do not follow the grammar and they have to judge whether a given string is grammatical or ungrammatical. The usual finding is that the participants judge the grammaticality of the strings better than chance-level, but concurrently are not able to explicate the underlying rules. Albeit debatable, Reber (1967) concluded from these findings that learning of the grammar is implicit. For instance, Perruchet and Pacton (1990) provided evidence that implicit grammar learning might be based on explicit knowledge about grammar fragments (e.g., single bigrams).

In contrast to the AGL, in the standard SRTT originated by Nissen and Bullemer (1987), the participants see several locations marked on the screen. These screen-locations are spatially mapped to response keys. In each trial, an asterisk (target) appears at one of the marked screen locations and the participants are asked to press the assigned response key as quickly and accurately as possible. Unbeknownst to the participants, the locations of the asterisk follow a regular sequence. This sequence is replaced by a random sequence after a few practice blocks and is re-introduced thereafter. This leads to an increase in response times and/or error rates which disappears as soon as the regular sequence reoccurs. Analogous to the AGL, most participants are unable to explicate their acquired sequence knowledge. Meanwhile, the standard SRTT has been modified to assess not only perceptual-motor learning, but also, for instance, pure perceptual learning or the concurrent learning of uncorrelated sequences (e.g., perceptual sequences: stimulus-color sequence, stimulus-location sequence; or a motor sequence: response-location sequence; Eberhardt et al., 2017, Goschke & Bolte, 2012; Haider et al., 2012; Haider et al., 2014; Howard et al., 1992; Mayr, 1996; Remillard, 2009, 2011).

Even though such additional variations of the SRTT might have attenuated the boundaries between the research topics investigated within the fields of the AGL and the SRTT experiments, the two fields are only loosely connected (Frost et al., 2015). This is somewhat surprising since the research questions in both fields are quite similar. In both fields, a central question concerns the building blocks of implicit learning (Conway, 2020; Conway & Christiansen, 2006; Eberhardt et al., 2017; Goschke & Bolte, 2012; Haider et al., 2018; Mayr, 1996). Yet, there is no agreement on what exactly the processed contents might be that become associated within implicit learning tasks. For instance, implicit learning might refer to contents represented in different modalities like vision, or audition (Abrahamse et al., 2010; Frost et al., 2015). Yet, alternatively, some authors now propose to look at a more fine-grained level within modalities, namely features like color, shape, or location for the visual modality which form the smallest entities that can become associated in an implicit learning task (e.g., Conway & Christiansen, 2006; Eberhardt et al., 2017; Haider et al., 2018). It is important to note that the conceptualization of features, as we use it here, stems from the Theory of Event Coding (TEC; Hommel, 2009; Hommel et al., 2001), a theory of perception and action planning. In short, according to the TEC, it is assumed that both perceptions and actions are represented in terms of distal events in the form of consciously available feature codes. These sensorimotor feature codes, for instance the feature code “left” as part of the feature “location” are made up of various proximal sensory and motor representations. These representations are distributed all over the cortex, such as color, shape, or location are represented in distinct parts of the visual cortex (Hommel, 2009). To plan or conduct an action means that such feature codes are bound together in so-called event files (Hommel, 2004). We, therefore, assume that implicit learning might be based on associations between such feature codes belonging to one feature. This would imply that two regularities can be learned concurrently as long as they belong to different features.

Such an account concerning the representations underlying implicit learning might help to better understand implicit learning. When looking at the theoretical accounts, it becomes obvious that in both fields the proposals only loosely define the characteristics of the representations underlying implicit learning. For instance, in SRTT learning, Keele and colleagues (2003) proposed the Dual-System Model.¹ In this model, the crucial assumption is that implicit learning in the unidimensional system takes place in multiple encapsulated modules which are each specialized to process information along a particular dimension. Due to this specificity, the modules can process information in parallel as long as they belong to different dimensions.

Thus, according to the Dual-System Model, it is the particular dimension processed in the unidimensional system that constitutes the building blocks of implicitly learned sequences. As already noticed by the authors themselves, the problem is that the term dimension is underspecified. It may refer to modalities, like vision, audition and so on. Alternatively, it might refer to features within modalities, like color, shape, and location.

In the field of AGL, Frost et al., (2015, 2019) proposed in their model of Perceptual Statistical Learning to interpret implicit learning as an interaction between domain-specific and domain-general learning processes. Domain-specific learning takes place in different cortical areas (e.g., learning of a visual regularity in the visual cortex). Hence, learning of specific regularities is constrained by properties of the respective cortices and therefore, modality-specific. However, the encoding of specific stimuli within a modality can also take place in different brain regions leading to stimulus-specific representations (e.g., learning of a regularity of colors in a specific part of the visual cortex). These representations are fed into a multi-modal region for further domain-general computations in the medial temporal lobe memory system (Frost et al., 2015).

According to Frost et al., (2015), the capacity of the domain-specific learning processes is limited. Factors, such as the complexity and similarity of the to-be-associated material, influence whether learning is modality- or stimulus-specific. These factors remain underspecified by the authors. We propose that two regularities instantiated by two distinct features (e.g., color, shape, etc.) might be processed concurrently and independently from each other.

Recently, Conway (2020) combined the implicit and explicit learning modes from Keele et al., (2003) with modality-specific and domain-general learning mechanisms similar to Frost et al., (2015) and also arrived at the question how learning might be affected by the input domain.

In summary, the exact understanding of the terms dimension or domain remains unclear in the above-described models and hence, the question about the content of the representations underlying implicit learning is still unanswered (Frost et al., 2015; Keele et al., 2003). We propose to equate the terms dimension and domain with feature, such as colors, shapes, and location in the visual domain.

A usual way to investigate the (minimal) building blocks of implicit learning is to train participants in an implicit learning task with two uncorrelated statistical regularities (Abrahamse et al., 2010; Conway & Christiansen, 2006; Goschke & Bolte, 2012). If the participants can successfully acquire these two regularities concurrently, the participants must have been able to keep the representations of the statistical regularities separate. A few studies investigated already this parallel learning of multiple regularities within AGL and SRTT designs (i.e., Conway & Christiansen, 2006; Deroost & Soetens, 2006; Li et al., 2018; Mayr, 1996; Witt & Willingham, 2006). The research can be summarized with regard to learning of multiple regularities that were instantiated in either different or the same modality, and in either different or the same feature.

As one of the first who investigated learning in two different modalities in the SRTT, Mayr (1996) trained participants to discriminate object-identities by pressing the assigned keys. The objects appeared in one of the four corners of a square on the screen. Importantly, both the object-identities and the screen-locations followed uncorrelated sequences, respectively. The findings confirmed that participants could learn the response (object-identities) and the perceptual sequence (screen-locations) simultaneously. Deroost and Soetens (2006) replicated this SRTT experiment, but additionally showed that small changes in the experimental design hampered already the learning of the perceptual sequence. Thus, it seems that simultaneous learning of a perceptual and a response sequence is possible, even though perceptual learning seems to be rather fragile (Rüsseler & Rösler, 2000).

In a modified AGL experiment of Conway and Christiansen (2006), the participants received randomly intermixed visual color strings and auditory tone strings that were both derived from two different grammars. The authors observed simultaneous learning of both grammars and concluded that concurrent learning of two grammars instantiated by two different modalities (visual versus auditory) is possible.

Goschke and Bolte (2012) succeeded in showing that participants could learn two uncorrelated perceptual sequences (one consisting of stimulus-locations, the other of visual target-letters) concurrently with a third uncorrelated response-location sequence in the SRTT. First, their results support the findings of Mayr (1996) that a perceptual and a response sequence can be learned concurrently. Furthermore, they showed that two sequences within the same modality (visual) instantiated by the features location and shape can also be learned concurrently.

Similarly, in the AGL, Conway and Christiansen (2006) showed that participants learned two grammars presented by two perceptual features (color and shape) concurrently. This result has also been replicated by Johansson (2009). Furthermore, Turk-Browne et al. (2008) showed that visual strings of colors and shapes can be learned concurrently in the AGL, even when they were displayed in the same object, as long as the regularities did not co-vary. Walk and Conway (2016) also provided evidence for concurrent learning of color and shape strings derived from two different sets of grammars.

In the already mentioned modified AGL design of Conway and Christiansen (2006), the authors also tested for learning of two grammars instantiated by the same perceptual feature (two sets of shape strings). In this condition, the participants were unable to learn the grammatical rules of the two shape sets. Thus, if the strings derived from different grammars are based on the same feature, simultaneous learning of grammatical rules is impossible. In the SRTT, Eberhardt et al., (2017) went one step further by asking for the constraints of concurrently learning two uncorrelated sequences in parallel. In their study, the participants received either a visual-color or a response-location sequence together with a sequence of screen-locations. The results revealed that while the participants learned the visual-color sequence simultaneously with the screen-location sequence (both sequences were perceptual sequences), they were not able to learn the screen-location sequence together with the response-location sequence (one perceptual and one response sequence). In a follow-up experiment, the authors could support these findings by showing that when the participants were instructed to code the response keys in terms of their locations, they did not learn the stimulus-location sequence. By contrast, they did so when they coded their responses in terms of the colors (Gaschler et al., 2012). These findings thus fit the assumption that the implicit learning system can be dedicated to process single features within the visual modality (Hommel, 2009). The stimulus-location sequence and the response-location sequence are both instantiated by the same feature (location) and could not be learned concurrently. In contrast, the stimulus-color and the stimulus-location sequence refer to distinct features (color and location) and could be learned concurrently.

Thus, the empirical findings reported so far suggest that implicit learning, either in the AGL or in the SRTT, relies on associations within features like color, shape, or location as the (minimal) building blocks. However, except for the Conway and Christiansen (2006) study, the experiments all used location as one of the features. This might be a limitation because “location” is suggested to be a special feature (Conway, 2020). First, it seems as if the location of a stimulus is processed more effectively than other perceptual features (Gaschler et al., 2012; Koch & Hoffmann, 2000). Second, location might not be even a pure perceptual feature because it might involve eye movements (Goschke & Bolte, 2012; Marcus et al., 2006; Willingham et al., 2000). To make a stronger point that features might be the (minimal) building blocks of implicit learning, more evidence is needed showing that concurrent learning of two regularities is possible, even when they are instantiated by two non-spatial perceptual features.

An a-priori power analyses (d = 0.8,²α = β = 0.05; Faul et al., 2007) yielded a required sample of 40 participants, that is 20 participants³ in each condition. We collected the data of 40 participants in an online experiment in Prolific (www.​prolific.​co; 22 women, mean age = 26.30, age range = 18–40 years, SD = 6.08). No participant reported to be color-blind. All participants received £2.50 in exchange for participation.

We used the same two different finite-state grammars as Conway and Christiansen (2006) to generate two sets of non-overlapping strings (see Fig. 1). Nine strings of each grammar were used in the training phase and 10 strings of each grammar in the test phase.⁴ The strings contained between three and seven elements. For one grammar, each letter was mapped onto a color vocabulary. For the other grammar, each letter was mapped onto a shape vocabulary. The assignment of vocabulary to grammar was counterbalanced across participants. Also, the assignment of letters to particular colors and shapes was counterbalanced across participants.

We used different sets of colors and shapes than Conway and Christiansen (2006).⁵ For colors, we chose magenta, blue, orange, cyan, and green. For shapes, we presented a triangle, diamond, circle, arch, and cross. All visual stimuli were presented in the center of the screen (80 × 80 pixels).

We used the same procedure as in the original experiment. For all participants, the experiment started with computer-presented instructions. Participants were told to observe the shape and color strings carefully, because they would be tested on what they have observed. Then, the training phase started with six training blocks containing the 18 training strings, each. For all participants, half of these strings were derived from one grammar and were presented as colors; the other half relied on the other grammar and was presented as shapes. In each training block, the same 18 training strings were repeated. The presentation order of the strings was entirely random so that the participants observed the grammars intermixed and could learn them simultaneously. Each string element (color or shape item) was presented for 500 ms, followed after an interval of 100 ms by the next string element. Strings were separated by 1.700 ms (see Fig. 2).

After training, the participants were randomly assigned to either the color-test condition or the shape-test condition. In the color-test condition, the participants received only new colored strings derived from either the color grammar (grammatical strings) or the shape grammar (ungrammatical strings) of the training. In the shape-test condition, the participants analogously received only grammatical and ungrammatical shape strings. All participants were informed that the strings they had observed during training were generated according to a complex set of rules. They were told to observe each string carefully and to judge if it follows the same set of rules as in the training phase (grammaticality judgement).

After the test phase, all participants completed an online questionnaire. First, they were asked if they had noticed anything during the experiment. Additionally, the color and shape stimuli were displayed, and participants were asked to indicate with which color and shape the strings usually began and ended. They were also asked to produce as many transitions as they believed had occurred in the training stimuli. For each correct begin, ending, and transition, the participants gained a point so that a score of explicit grammar knowledge could be calculated.

To assess the participants’ grammar knowledge from the test phase, we applied the same scoring system as in the original analysis (Conway & Christiansen, 2006). The participants’ judgments were scored as correct, when they classified a test string as grammatical that has been derived from that particular grammar during training (e.g., a color-test string was classified as grammatical, and the test string was based on the color grammar). If they classified a test string as ungrammatical and it was based on the other grammar than during training, the judgment was classified also as correct (e.g., a color-test string was classified as ungrammatical, and the test string was derived from the shape grammar). If the participants had learned the transitions between the respective features, they should be better than chance to correctly accept or reject the test strings as grammatical or ungrammatical (> 50% correct judgements). It is important to note that chance-level performance would lead to two possible interpretations. The first interpretation would be that participants did not learn the transitions. The second interpretation might be that the participants acquired abstract representations of both grammars without distinguishing between the perceptual characteristics. In this case, for instance, participants would judge most of the strings as grammatical because all strings are either based on the shape grammar or the color grammar. Due to the scoring system, their judgements would lead to chance-level performance. Hence, the scoring system cannot differentiate between a participant who did not learn at all and a participant who learned abstract grammar rules. However, a scoring system above chance-level would clearly identify a participant who learned both grammars separately (Conway & Christiansen, 2006).

We first report the frequentist analyses and afterwards additionally the Bayes factors. For all Bayes analyses, we tested one-sided hypotheses and used a default Cauchy prior distribution with r = 1/√2, truncated to only allow positive effect sizes (van Doorn et al., 2019). All Bayes analyses were conducted with JASP (2020) (Wagenmakers et al., 2018).

An a-priori power analyses (d = 0.6,⁶α = β = 0.05; Faul et al., 2007) yielded a required sample of 60 participants, that is 30 participants in each condition. Participants were excluded if they had made more than 20% errors in the training or test phase. No participant reported to be color-blind.

Sixty-four students of the University of Cologne participated in the laboratory experiment. No participant exceeded our error criterion. One participant in the shape-test condition and one participant in the color-test condition were excluded due to technical issues. This left 30 participants in the color-test condition and 32 participants in the shape-test condition (45 women, 1 not stated, mean age = 22.98, age range = 18–38 years, SD = 5.04). All participants received either 4€ or course credit in exchange for participation. In addition, they could earn 2€ extra money in the wagering task.

In the training phase, in every trial two targets appeared in the middle of the screen (100 × 100 pixels, 2 cm distance) in front of a grey background. The color-target always occurred on the left side, the shape-target on the right. Unbeknownst to the participants, either the colors followed a 6-element first-order sequence and the shapes a 7-element second-order sequence (magenta – blue – orange – cyan – green – red; diamond – cross – circle – arch – triangle – cross – star) or vice versa (blue – magenta – green – orange – red – magenta – cyan; cross – diamond – arch – star – circle – triangle). Due to the differences in sequence length, the correlated sequence was rather long (42 elements), making it highly unlikely that the participants would learn this long sequence (e.g., Schmidtke & Heuer, 1997). To ensure that the participants would attend to the colors and the shapes, we interspersed some differing trials. In 16.7% of the trials the color target was dotted and in another 16.7% of the trials the frame of the shape target was dashed. Participants had to indicate these differing targets by pressing the spacebar.

The experiment started with instructions displayed on the screen. The participants were told to pay close attention to the shape and the color-patch stimuli and to press the spacebar as soon as they detect a differing target. Afterwards, the training phase started with 7 training blocks containing 89 trials, each. In each trial, a color and a shape stimulus appeared on the screen for 250 ms. After an inter-trial-interval of 2150 ms the next targets appeared on the screen. The participants merely observed the color and shape targets. In some trials, either the color target was dotted, or the shape target was dashed. If this was the case, the participants had to press the spacebar within the inter-trial-interval of 2150 ms. If they missed a differing target, an error message (“Miss”) appeared for 250 ms and a 400 Hz tone sounded for 50 ms. If they pressed the spacebar, but both targets did not differ, the message “False Alarm!” appeared on the screen for 250 ms and the 400 Hz tone also sounded for 50 ms. Between blocks, the participants could take short breaks.

After the training phase, the participants were administered to the post decision wagering task (see Fig. 4) which served to assess the participants knowledge about either the color or the shape sequence. Accordingly, the participants were randomly assigned to either the color-test or the shape-test conditions.

The post decision wagering task consisted of 100 trials presented within one single block. In these trials, a single target (either a color patch in the color-test condition or a shape in the shape-test condition) appeared in the upper part of the screen. In the lower part of the screen, six colored response squares (color-test condition) or six shapes (shape-test condition) appeared. Their screen locations were mapped to the keys Y, X, C, B, N, and M on a German QWERTZ keyboard (the left most response square was mapped to the Y-key, etc.). The participant’s task was to press the response key assigned to the location of the response square containing the target. The arrangement of the colors (shapes) of the response squares changed from trial to trial such that the response keys did not follow any sequence.

While the target disappeared after 150 ms, the response squares remained on the screen until the participant`s response. Afterwards, the screen went black for 300 ms. Incorrect responses were not signaled.

In overall 20 trials, the so-called wagering trials, a question mark appeared instead of the target. The participants’ task here was to guess the color or shape of the current target. That is, they had to search among the six response squares the color or shape (depending on condition) they guessed the target would have been and to press the respective response key. Immediately after the participant’s response, the pictures of a 1 cent and a 50 cents coin appeared on the screen. The participants were instructed to bet on the correctness of their guessed target by pressing either the A-key (1 cent) or the K-key (50 cents). If their guess was correct, they won the amount of the wager; if not, they lost it. Importantly, the participants were not informed whether they guessed correctly in the respective trials. The participants were free to use low and high wagers as frequently as they were pleased to, but they were told to try to maximize their earnings.

The following example serves to illustrate the wagering phase further: A participant who observed the 6-element color sequence in the training phase (magenta – blue – orange – cyan – green – red) would react by simple key presses first to the color magenta and then in the next trial to blue. After that, instead of the next color orange, a question mark appears, and the participant is supposed to guess that the color orange would have followed in the training phase.

In the wagering trials, the arrangement of the response squares did not change from trial t-1 to t. The participants might remember the location of their last response square (but not the last color or shape) and attend to this position in order to predict the next target. If this were the case, changing the arrangement of the response squares would overwrite the representation of the last target (see Fig. 4). The maximum of extra-earnings was set to 2€. If the participants reached this maximum before the end of the block, the wagering task was terminated.

After having finished the post decision wagering task, a short interview followed to assess the participants’ reportable sequence knowledge. The participants were first asked if they had noticed anything during the experiment. Then, they were asked if they had noticed any structural regularity during their experiment and, if they affirmed, they were asked to describe it. Subsequently, they were informed about the two sequences and their lengths and asked to try to reconstruct both, the color and the shape sequences. Participants were categorized as having structural conscious knowledge when they were able to name three or more consecutive transitions of their test sequence. Finally, participants received their payment or course credit, were informed about their earnings in the wagering task and were then debriefed.

If the participants had learned the sequences, their correct guesses in the wagering task should be above chance-level.⁷ We set the chance-level to 20% since immediate repetitions of colors or shapes in the respective sequences were excluded during training.⁸ In order to assess whether the participants’ sequence knowledge was implicit or explicit, we also analyzed the percentage of correct answers under the condition of high wagers versus of low wagers. The rationale was that if their correct guesses were mainly driven by implicit sequence knowledge, they will place high or low wagers randomly and will be unable to maximize their overall earnings (Dienes & Seth, 2010). If, by contrast, the participants have acquired explicit sequence knowledge, they should be able to strategically place high wagers when they are certain that their guess of the last target-color or target-shape was correct (see Haider et al., 2011).

Again, we report results from frequentist analyses and Bayes factor analyses using JASP (2020), (Wagenmakers et al., 2018) with default Cauchy prior distribution with r = 1/√2 allowing only positive effect sizes (van Doorn et al., 2019).