Skip to main content
SpringerLink
Log in

Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

A growing body of empirical research on the topic of multisensory perception now shows that even non-synaesthetic individuals experience crossmodal correspondences, that is, apparently arbitrary compatibility effects between stimuli in different sensory modalities. In the present study, we replicated a number of classic results from the literature on crossmodal correspondences and highlight the existence of two new crossmodal correspondences using a modified version of the implicit association test (IAT). Given that only a single stimulus was presented on each trial, these results rule out selective attention and multisensory integration as possible mechanisms underlying the reported compatibility effects on speeded performance. The crossmodal correspondences examined in the present study all gave rise to very similar effect sizes, and the compatibility effect had a very rapid onset, thus speaking to the automatic detection of crossmodal correspondences. These results are further discussed in terms of the advantages of the IAT over traditional techniques for assessing the strength and symmetry of various crossmodal correspondences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Results of supplemental analysis on untransformed data after the removal of outliers (i.e., responses above 3sd SD from the individual means. Overall less than 1 % of the data were removed), but without discarding the first four trials of each block:

    Experiment 1: Congruency: F(1,9) = 17.49, p = .002; Modality: F(1,9) = 29.41, p < .001; Interaction: F(1,9) = 1.41, p = .26.

    Experiment 2: Congruency: F(1,9) = 16.37, p = .003; Modality: F(1,9) = 38.21, p < .001; Interaction: F(1,9) < 1, n.s.

    Experiment 3: Congruency: F(1,9) = 7.36, p = .024; Modality: F(1,9) = 12.07, p = .007; Interaction: F(1,9) = 2.01, p = .19.

    Experiment 4: Congruency: F(1,9) = 7.43, p = .023; Modality: F(1,9) = 5.586, p = .042; Interaction: F(1,9) < 1, n.s.

    Experiment 5: Congruency: F(1,9) = 5.44, p = .045; Modality: F(1,9) = 20.22, p = .001; Interaction: F(1,9) = 4.17, p = .072.

References

  • Bernstein IH, Edelstein BA (1971) Effects of some variations in auditory input upon visual choice reaction time. J Exp Psychol 87(2):241–247

    Article  PubMed  CAS  Google Scholar 

  • Bien N, ten Oever S, Goebel R, Sack AT (2012) The sound of size Crossmodal binding in pitch-size synesthesia: A combined TMS, EEG and psychophysics study. Neuroimage 59:663–672

    Google Scholar 

  • Blair IV (2002) The malleability of automatic stereotypes and prejudice. Personal Soc Psychol Rev 6:242–261

    Article  Google Scholar 

  • Bozzi P, Flores D’Arcais G (1967) Experimental research on the intermodal relationships between expressive qualities. Arch Psicol Neurol Psichiatr 28(5):377–420

    PubMed  CAS  Google Scholar 

  • Brang D, Rouw R, Ramachandran VS, Coulson S (2011) Similarly shaped letters evoke similar colors in grapheme–color synesthesia. Neuropsychologia 49:1355–1358

    Article  PubMed  Google Scholar 

  • Bremner A, Caparos S, Davidoff J, de Fockert J, Linnell K, Spence C (submitted) Bouba and Kiki in Namibia? Western shape-symbolism does not extend to taste in a remote population. Cognition

  • Chen Y-C, Spence C (2011) Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. J Exp Psychol Hum Percept Perform 37:1554–1568

    Article  PubMed  Google Scholar 

  • Chiou R, Rich AN (2012) Cross-modality correspondence between pitch and spatial location modulates attentional orienting. Perception 41:339–353

    Article  Google Scholar 

  • Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, Hillsdale, NJ

    Google Scholar 

  • Cohen Kadosh R, Henik A (2007) Can synaesthesia research inform cognitive science? Trends Cogn Sci 11(4):177–184

    Article  PubMed  Google Scholar 

  • Cohen Kadosh R, Cohen Kadosh K, Henik A (2007a) The neuronal correlate of bidirectional synesthesia: a combined event-related potential and functional magnetic resonance imaging study. J Cogn Neurosci 19(12):2050–2059

    Article  PubMed  Google Scholar 

  • Cohen Kadosh R, Henik A, Walsh V (2007b) Small is bright and big is dark in synaesthesia. Curr Biol 17(19):R834–R835

    Article  PubMed  CAS  Google Scholar 

  • Cowey A, Weiskrantz L (1975) Demonstration of cross-modal matching in rhesus monkeys, Macaca mulatta. Neuropsychologia 13(1):117–120

    Article  PubMed  CAS  Google Scholar 

  • Crisinel AS, Spence C (2009) Implicit association between basic tastes and pitch. Neurosci Lett 464(1):39–42

    Article  PubMed  CAS  Google Scholar 

  • Crisinel AS, Spence C (2010) A sweet sound? Food names reveal implicit associations between taste and pitch. Perception 39(3):417–425

    Article  PubMed  Google Scholar 

  • Davis R (1961) The fitness of names to drawings: a cross-cultural study in Tanganyika. Br J Psychol 52:259–268

    Article  PubMed  CAS  Google Scholar 

  • De Jong R, Liang CC, Lauber E (1994) Conditional andunconditional automaticity: a dual-process model of effects of spatial stimulus-response correspondence. J Exp Psychol Hum Percept Perform 20(4):731–750

    Article  PubMed  Google Scholar 

  • Demattè M, Sanabria D, Spence C (2006) Cross-modal associations between odors and colors. Chem Senses 31(6):531–538

    Article  Google Scholar 

  • Demattè M, Sanabria D, Spence C (2007) Olfactory-tactile compatibility effects demonstrated using a variation of the implicit association test. Acta Psychol 124(3):332–343

    Article  Google Scholar 

  • Dixon MJ, Smilek D, Cudahy C, Merikle PM (2000) Five plus two equals yellow. Nature 406(6794):365

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO (2007) Learning to integrate arbitrary signals from vision and touch. J Vis 7(5):1–14

    Article  PubMed  Google Scholar 

  • Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10(1):1–12

    PubMed  Google Scholar 

  • Fiedler K, Messner C, Bluemke M (2006) Unresolved problems with the “I”, the “A”, and the “T”: a logical and psychometric critique of the implicit association test (IAT). Eur Rev Soc Psychol 17:74–147

    Article  Google Scholar 

  • Freed DJ (1990) Auditory correlates of perceived mallet hardness for a set of recorded percussive sound events. J Acoust Soc Am 87(1):311–322

    Article  PubMed  CAS  Google Scholar 

  • Gallace A, Spence C (2006) Multisensory synesthetic interactions in the speeded classification of visual size. Percept Psychophys 68(7):1191–1203

    Article  PubMed  Google Scholar 

  • Greenwald AG, McGhee DE, Schwartz JLK (1998) Measuring individual differences in implicit cognition: the implicit association test. J Pers Soc Psychol 74(6):1464–1480

    Article  PubMed  CAS  Google Scholar 

  • Grossenbacher PG, Lovelace CT (2001) Mechanisms of synesthesia: cognitive and physiological constraints. Trends Cogn Sci 5(1):36–41

    Article  PubMed  Google Scholar 

  • Hinton L, Nichols J, Ohala JJ (eds) (2006) Sound symbolism. Cambridge University Press, Cambridge

  • Johnson A, Jepma M, De Jong R (2007) Colours sometimes count: awareness and bidirectionality in grapheme-colour synaesthesia. Q J Exp Psychol 60(10):1406–1422

    Article  Google Scholar 

  • Klapetek A, Ngo MK, Spence C (in press) Do crossmodal correspondences enhance the facilitatory effect of auditory cues on visual search? Atten Percept Psychophys

  • Klatzky RL, Pai DK, Krotkov EP (2000) Perception of material from contact sounds. Presence Teleoper Virtual Environ 9(4):399–410

    Article  Google Scholar 

  • Klein R, Brennan M, Gilani A (1987) Covert cross-modality orienting of attention in space. Paper presented at the Annual meeting of the Psychonomic Society, Seattle, WA

    Google Scholar 

  • Köhler W (1929) Gestalt psychology. Liveright, New York

    Google Scholar 

  • Köhler W (1947) Gestalt psychology: an introduction to new concepts in modern psychology. Liveright Publ. Corporation, New York, NY

    Google Scholar 

  • Kovic V, Plunkett K, Westermann G (2010) The shape of words in the brain. Cognition 114(1):19–28

    Article  PubMed  Google Scholar 

  • Ludwig VU, Adachi I, Matzuzawa T (2011) Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans. Proc Natl Acad Sci USA 108:20661–20665

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (1987) On cross-modal similarity: auditory–visual interactions in speeded discrimination. J Exp Psychol Hum Percept Perform 13(3):384–394

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (1989) On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness. J Exp Psychol Hum Percept Perform 15(3):586–602

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (2004) Cross-modal interactions in speeded classification. In: Calvert GA, Spence C, Stein BE (eds) The handbook of mutisensory processes. MIT Press, Cambridge, MA, pp 85–106

    Google Scholar 

  • Martino G, Marks LE (2001) Synesthesia: strong and weak. Curr Dir Psychol Sci 10(2):61–65

    Article  Google Scholar 

  • Meier B, Rothen N (2007) When conditioned responses “fire back”: bidirectional cross-activation creates learning opportunities in synesthesia. Neuroscience 147(3):569–572

    Article  PubMed  CAS  Google Scholar 

  • Melara RD, O’Brien TP (1987) Interaction between synesthetically corresponding dimensions. J Exp Psychol Gen 116(4):323–336

    Article  Google Scholar 

  • Melara RD, O’Brien TP (1990) Effects of cuing on cross-modal congruity. J Mem Lang 29(6):655–686

    Article  Google Scholar 

  • Mills CB (1999) Digit synaesthesia: a case study using a Stroop-type test. Cogn Neuropsychol 16(2):181–191

    Article  Google Scholar 

  • Mulvenna CM, Walsh V (2006) Synaesthesia: supernormal integration? Trends Cogn Sci 10(8):350–352

    Article  PubMed  Google Scholar 

  • Newman S (1933) Further experiments in phonetic symbolism. Am J Psychol 45(1):53–75

    Article  Google Scholar 

  • Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355

    Article  PubMed  Google Scholar 

  • Osgood CE (1960) The cross-cultural generality of visual–verbal synesthetic tendencies. Behav Sci 5(2):146–169

    Article  Google Scholar 

  • Osgood CE, Suci G, Tannenbaum P (1957) The measurement of meaning. University of Illinois Press, Urbana

    Google Scholar 

  • Oyama T, Yamada H, Iwasawa H (1998) Synesthetic tendencies as the basis of sensory symbolism: a review of a series of experiments by means of semantic differential. Psychologia 41:203–215

    Google Scholar 

  • Parise CV, Pavani F (2011) Evidence of sound symbolism in simple vocalizations. Exp Brain Res 214(3):373–380

    Article  PubMed  Google Scholar 

  • Parise CV, Spence C (2008) Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci Lett 442(3):257–261

    Article  PubMed  CAS  Google Scholar 

  • Parise CV, Spence C (2009) When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes. PLoS One 4(5):e5664

    Article  PubMed  Google Scholar 

  • Parise CV, Spence C (2012) Assessing the associations between brand packaging and brand attributes using an indirect performance measure. Food Qual Prefer 24:17–23

    Article  Google Scholar 

  • Parise CV, Spence C (in press) Audiovisual crossmodal correspondences. In Simner J, Hubbard EM (eds) Oxford handbook of synaesthesia. Oxford University Press, Oxford

  • Parise CV, Spence C, Ernst MO (2012) When correlation implies causation in multisensory integration. Curr Biol 22:46–49

    Article  PubMed  CAS  Google Scholar 

  • Parker A, Easton A (2004) Cross-modal memory in primates: the neural basis of learning about the multisensory properties of objects and events. In: Calvert GA, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, MA, pp 333–342

    Google Scholar 

  • Poffenberger A, Barrows B (1924) The feeling value of lines. J Appl Psychol 8(2):187–205

    Article  Google Scholar 

  • Premack D, Premack AJ (2003) Original intelligence: unlocking the mystery of who we are. McGraw-Hill, New York

    Google Scholar 

  • Rader C, Tellegen A (1987) An investigation of synesthesia. J Pers Soc Psychol 52(5):981–987

    Article  Google Scholar 

  • Ramachandran VS, Oberman LM (2007) Broken mirrors: a theory of autism. Sci Am Spec Ed 17(2):20–29

    Google Scholar 

  • Robson D (2011) Language’s missing link. New Sci 211(2821):30–33

    Article  Google Scholar 

  • Rogers SK, Ross AS (1968) A cross-cultural test of the Maluma-Takete phenomenon. Perception 4(1):105–106

    Article  Google Scholar 

  • Rousseeuw PJ, Ruts I, Tukey JW (1999) The bagplot: a bivariate boxplot. Am Stat 53(4):382–387

    Google Scholar 

  • Rudmin F, Cappelli M (1983) Tone-taste synesthesia: a replication. Percept Mot Skills 56:118

    Article  PubMed  CAS  Google Scholar 

  • Sapir E (1929) A study in phonetic symbolism. J Exp Psychol 12(3):225–239

    Article  Google Scholar 

  • Seo H-S, Arshamian A, Schemmer K, Scheer I, Sander T, Ritter G, Hummel T (2010) Cross-modal integration between odors and abstract symbols. Neurosci Lett 478:175–178

    Article  PubMed  CAS  Google Scholar 

  • Shepherd GM (2012) Neurogastronomy: how the brain creates flavor and why it matters. Columbia University Press, New York

    Google Scholar 

  • Simpson RH, Quinn M, Ausubel DP (1956) Synesthesia in children: association of colors with pure tone frequencies. J Genet Psychol Res Theory Hum Dev 89(1):95–103

    CAS  Google Scholar 

  • Soto-Faraco S, Lyons J, Gazzaniga M, Spence C, Kingstone A (2002) The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. Cogn Brain Res 14(1):139–146

    Article  Google Scholar 

  • Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):1–25

    Article  Google Scholar 

  • Spence C, Deroy O (2012) Are chimpanzees really synaesthetic? i-Perception 3:316–318

  • Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519–R521

    Article  PubMed  CAS  Google Scholar 

  • Stevens JC, Marks LE (1965) Cross-modality matching of brightness and loudness. Proc Natl Acad Sci USA 54(2):407–411

    Article  PubMed  CAS  Google Scholar 

  • Stumpf K (1883) Tonpsychologie. S. Hirzel, Leipzig

    Google Scholar 

  • Svartdal F, Iversen T (1989) Consistency in synesthetic experience to vowels and consonants: five case studies. Scand J Psychol 30:220–227

    Article  PubMed  CAS  Google Scholar 

  • Vallesi A, Mapelli D, Schiff S, Amodio P, Umiltà C (2005) Horizontal and vertical Simon effect: different underlying mechanisms? Cognition 96(1):B33–B43

    Article  PubMed  Google Scholar 

  • Van den Doel K, Pai DK (1998) The sounds of physical shapes. Presence 7(4):382–395

    Article  Google Scholar 

  • Walker P, Smith S (1985) Stroop interference based on the multimodal correlates of haptic size and auditory pitch. Perception 14(6):729–736

    Article  PubMed  CAS  Google Scholar 

  • Walker P, Bremner J, Mason U, Spring J, Mattock K, Slater A, Johnson S (2010) Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychol Sci 21(1):21–25

    Article  PubMed  Google Scholar 

  • Ward J, Huckstep B, Tsakanikos E (2006) Sound-colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all? Cortex 42(2):264–280

    Article  PubMed  Google Scholar 

  • Watson AB, Pelli DG (1983) QUEST-a Bayesian adaptive psychometric method. Percept Psychophys 33(2):113–120

    Article  PubMed  CAS  Google Scholar 

  • Weiskrantz L, Cowey A (1975) Cross-modal matching in the rhesus monkey using a single pair of stimuli. Neuropsychologia 13(3):257–261

    Article  PubMed  CAS  Google Scholar 

  • Westbury C (2005) Implicit sound symbolism in lexical access: evidence from an interference task. Brain Lang 93(1):10–19

    Article  PubMed  Google Scholar 

  • Xu J, Yu L, Rowland BA, Stanford TR, Stein BE (2012) Incorporating cross-modal statistics in the development and maintenance of multisensory integration. J Neurosci 32:2287–2298

    Article  PubMed  CAS  Google Scholar 

  • Zigler MJ (1930) Tone shapes: a novel type of synaesthesia. J Gen Psychol 3:276–287

    Article  Google Scholar 

Download references

Acknowledgments

Cesare Parise was supported by the Bernstein Center for Computational Neuroscience, Tübingen, funded by the German Federal Ministry of Education and Research (BMBF; FKZ: 01GQ1002).

Author information

Authors and Affiliations

  1. Department of Experimental Psychology, University of Oxford, Oxford, UK

    Cesare V. Parise & Charles Spence

  2. Max Planck Institute for Biological Cybernetics, Tübingen, Germany

    Cesare V. Parise

  3. Bernstein Centre for Computational Neuroscience, Tübingen, Germany

    Cesare V. Parise

  4. Department of Cognitive Neuroscience and Center of Excellence Cognitive Interaction Technology (CITEC), University of Bielefeld, Universitaetstr. 25, W3-246, 33615, Bielefeld, Germany

    Cesare V. Parise

Authors
  1. Cesare V. Parise
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles Spence
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cesare V. Parise.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (MPG 161 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Parise, C.V., Spence, C. Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test. Exp Brain Res 220, 319–333 (2012). https://doi.org/10.1007/s00221-012-3140-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-012-3140-6

Keywords

Search

Navigation