Pitch-height is often labeled spatially (i.e., low or high) as a function of the fundamental frequency of the tone. This correspondence is highlighted by the so-called Spatial–Musical Association of Response Codes (SMARC) effect. However, the literature suggests that the brightness of the tone’s timbre might contribute to this spatial association. We investigated the SMARC effect in a group of non-musicians by disentangling the role of pitch-height and the role of tone-brightness. In three experimental conditions, participants were asked to judge whether the tone they were listening to was (or was not) modulated in amplitude (i.e., vibrato). Participants were required to make their response in both the horizontal and the vertical axes. In a first condition, tones varied coherently in pitch (i.e., manipulation of the tone’s F0) and brightness (i.e., manipulation of the tone’s spectral centroid); in a second condition, pitch-height varied whereas brightness was fixed; in a third condition, pitch-height was fixed whereas brightness varied. We found the SMARC effect only in the first condition and only in the vertical axis. In contrast, we did not observe the effect in any of the remaining conditions. The present results suggest that, in non-musicians, the SMARC effect is not due to the manipulation of the pitch-height alone, but arises because of a coherent change of pitch-height and brightness; this effect emerges along the vertical axis only.