Automatic effects of instructions: a tale of two paradigms

One technical difference was examined on an existing database (Meiran et al., 2016, N = 175). Specifically, we wanted to reject the claim that the differences between the two paradigms are due to stimulus type (i.e., a significant portion of the stimuli presented in the NEXT paradigms are pictures, but there were no picture targets in most of the studies employing the I-D paradigm). RTs were analyzed in a two-way ANOVA with a within-subjects independent variables congruency (congruent -incongruent) and stimulus type (symbols, letters, digits, and pictures). Mauchly's test indicated that the assumption of sphericity had been violated for the interaction between stimulus type and congruency (χ²(4) = 0.38, p < 0.001). Therefore, the degrees of freedom were corrected using Greenhouse–Geisser estimates of sphericity (ε = 0.67). As predicted, the interaction was not significant, F(2.68, 410.18) = 0.20, p = 0.880, η²_p = 0.0003, and permitted acceptance of H0, BF10 = 0.002. A focused contrast examining the AEI of pictures as compared to symbols, letters, and digits, pooled (which used the pooled error term) showed a non-significant difference as well with t(612) = 0.05, p = 0.962, with BF10 = 0.020, indicating support for H0 that stimulus type did not influence AEI.

When examining the influence of testing mode, 20 interactions were computed (10 for each phase; 5 in RT and 5 in PE). Two out of the 20 interactions had significant p-value and/or had BF value that enabled H1 acceptance: (1) The 2-way interaction between testing mode and congruency in PE, F(1, 75) = 14.06, p < 0.001, ²_p = 0.056, BF10 > 1000 (indicating a larger AEI in lab testing, 5.48% as compared to 0.35% in online testing); (2) The triple interaction among testing mode, congruency and instruction in RT, F(2, 75) = 2.82, p = 0.066, ²_p = 0.005, BF10 = 12.015. Since the spatial-control condition was added only as an additional reference, we examined if the interaction among testing mode, Congruency, and instruction in RT remains significant when omitting spatial-control and comparing the abstract and concrete conditions. The results show insignificant interaction, with an inconclusive BF10 = 0.417 (Full details are provided in Table S5). To ensure that any conclusions regarding testing mode do not reflect some speed-accuracy tradeoffs, all the aforementioned ten interactions (five for each phase) were tested once more, now on an integrated speed-accuracy measure, the Linear Integrated Speed–Accuracy Score (LISAS, Vandierendonck, 2017). Only one out of the ten interactions had BF value that enabled H1 acceptance (but showed non-significant p-value): The triple interaction among testing mode, congruency and instruction, F(2, 75) = 2.60, p = 0.081, ²_p = 0.005, BF10 = 13.425. Again, we examined if the interaction among testing mode, Congruency, and instruction was significant when comparing only the abstract and concrete conditions and found it inconclusive, BF10 = 1.072 (Full details are provided in Table S6 in the Supplementary Materials). Given the Testing mode results, and to be on the safe side, the B/ANOVA of the interaction between congruency and instruction in RT will be examined once on the entire sample and once in each testing mode, separately.

In the B/ANOVA, the computation of effects involving step violated the assumption of sphericity both in RT and in PE, which was indicated by Mauchly’s test. Therefore, degrees of freedom were corrected using the Greenhouse–Geisser correction. Mauchly’s test parameters and the GG estimates values are presented in Table S7 in Supplementary Materials. Results from G–G-corrected effects are presented in Tables 4 and 5 in bold.