Structural decomposition of EEG signatures of melodic processing

Abstract

In the current study we investigate the EEG response to listening and imagining melodies and explore the possibility of decomposing this response according to musical features, such as rhythm and pitch patterns. A structural model was created based on musical aspects and multiple regression was used to calculate profiles of the contribution of each aspect, in contrast to traditional ERP components. By decomposing the response, we aimed to uncover pronounced ERP contributions for aspects of the encoding of musical structure, assuming a simple additive combination of these. When using a model built up of metric levels and contour direction, 81% of the variance is explained for perceived, and 57% for imagined melodies. The maximum correlation between the parameters found for the same melodic aspect in perception vs. imagery was 0.88, indicating similar processing between tasks. The decomposition method is shown to be a novel analysis method of complex ERP patterns, which allows subcomponents to be investigated within a continuous context.

Introduction

In processing a musical stimulus, the information is thought to be grouped and categorized on multiple levels (see Lerdahl and Jackendoff, 1983). In probing the characteristics of these processes, it can be very challenging to isolate the different musical aspects and separate their related responses. The current study proposes a method to decompose the EEG response to simple melodies, both heard and imagined, and to isolate the brain response common to separate musical aspects of the stimulus.

The investigation into how the melodic structure is stored (as a sequence of pitches, as interval jumps, as a tonal contour, as pitch functions within the key, or as a combination of these) has been very informative in behavioral studies. Differences in reaction times have revealed hierarchical processing of tonality and chord structure (Bharucha and Krumhansl, 1983), the time course of recognition memory has indicated that contour (the relative up–down pattern of pitches in a melody) may be processed separately from the absolute pitch pattern (Dowling et al., 1995, Dowling et al., 2001). Recognition rates have shown varying levels of attention over different melodic and metric accents (Jones and Boltz, 1989) and sophisticated scrambling methods have shown interactions of global and local perception (i.e. Tillmann and Bigand, 2001), among others.

For a number of these musical aspects the EEG response has also been investigated, albeit mostly in the context of ERP components that have been described earlier, such as the P300 oddball response, the mismatch negativity (MMN) and language syntax related components such as the N400. Effects on the P300-complex were found for a number of oddball stimuli, such as minor vs. major mode for musicians (Halpern et al., 2007) and unexpected pitches in scales (Krohn et al., 2007). MMNs, occurring when infrequently occurring stimuli are perceived, were also seen for rare chord modulations (Koelsch et al., 2003) (where the authors refer to an early right anterior negativity as a music-syntactic MMN), out-of-key pitches (Brattico et al., 2006), and contour violations (Trainor et al., 2002). ERPs in response to musical rhythms have also been investigated, for instance for different metric levels such as the note, beat and bar level (Jongsma et al., 2004), as well as subjective accents (Brochard et al., 2003). On a more global level, parallel to phrases in language, musical phrase endings often induce a so-called closure-positive shift (Neuhaus et al., 2006).

In a single note, multiple characteristics or functions are combined, such as its absolute and relative pitch, the interval jump it has just made, its absolute and relative duration, hierarchical metric level, role in the harmony and more. How these aspects combine to form a representation is not clear. Jones and Ralston (1991) describe different aspects of the stimulus (i.e. the rhythmic pattern, the pitch pattern) as each having their own accent structures, and assume that the different accent structures combine in some way to form our representation. To investigate these aspects individually, they need to be isolated from each other. This is practically impossible for melodies that inherently contain these different layers, and can only be achieved by keeping constant as many of the other aspects as possible. The studies described above tend to use factorial designs, contrasting different conditions to investigate their different effects on the event-related brain response. Although there is generally an assumption that different ERP components combine additively, very few studies have tried to separate the effects of processing different aspects of a stimulus. A relatively new method of separating such intercorrelated variables is using multiple linear regression analysis (e.g. Schaefer and Desain, 2006, Hauk et al., 2006, Hauk et al., 2009, Dambacher et al., 2006) to test the degree to which a variable (or in the current study musical aspect) predicts data across all trials, in a continuous context. For fMRI data, the use of continuous regressors imitating the ongoing modulations is more common (see Cohen, 1997) and has already been proposed in the music domain by Janata et al. (2002).

To investigate the cognitive involvement in processing melodies separately from the auditory response, the investigated constructs also need to be isolated from the processing of sound, distinguishing the low-level perceptual mechanisms from the higher level cognitive processes. As we have a rich capacity for imagining music, imagery offers a means to get around auditory information processing while still evoking the representation of the melody. By investigating the relationship between perceived and imagined modalities, the level of similarity between the different tasks can also be addressed. This issue becomes particularly important, as the two have been shown to be entwined, both in behavioral and EEG studies.

In previous work that has looked at imagery of music, specific attention has been given to the spatial location of music imagery in the brain, as measured with PET or fMRI. Brain structures implicated in musical imagery were shown to be very similar to those recruited by actual perception, although involvement of primary auditory areas is not always found (c.f. Halpern et al., 2004, Kraemer et al., 2005). However, looking at imagery for melodies in EEG, Janata (2001) found that of a series of imagined notes, only the first elicited an N1 component as is generally seen for actually perceived notes. The subsequent imagined notes did not elicit such a component, most likely indicating the response to the first note to signify a state or task change. Other investigations of music imagery in EEG have focused specifically on musicians, for instance on the motor component induced by imagining the sound of an instrument participants played themselves (Kristeva et al., 2003), or the MMN elicited when a note imagined from notation deviates from a note actually presented (Yamamoto et al., 2005). When specifically investigating imagined rhythmic patterns, Desain and Honing (2003) showed that classification of internally rehearsed rhythms from the EEG signature is possible well above chance level, and preliminary results also show detectability of imagined natural overlearned music from the EEG (Schaefer et al., 2008). More basic investigations include work by Meyer et al. (2007), who found a reduced N1 and almost absent P2 response (see also Scherg et al., 1989) for imagined piano triads. However, these investigations do not address the multiple layers of a musical stimulus, such as metric levels or pitch structure.

In the current study, the full ERP trace of listening and imagining melodies is investigated, and the possibilities of decomposing these traces according to musical phenomena, here referred to as musical aspects, are explored. These musical phenomena or aspects add multiple levels of structure to a musical stimulus, and together form the melody. Examples are the absolute pitch sequence, the relative pitch pattern, the sequence of implied chords, and the rhythmic structure, which is multi-leveled itself. By taking this structure as the basis for the decomposition, we thus use a priori knowledge about the stimulus to decompose the response, similarly to Windsor et al. (2006) and Desain et al. (2008). A structural symbolic model can be built to represent these layers of structure, defined by simply labeling notes according to different musical aspects. Each aspect can have a number of levels (such as for instance metric depth), but is reduced into a set of components on separate levels (note, beat, bar, etc.) that are either absent or present. This structure can be expressed in a matrix of ones and zeros, shown more elaborately in the method description below. By regressing EEG data according to this structure, the contributions of different musical aspects to melodic processing can be calculated.

If we assume that these structural levels combine linearly in the ERP response, we can find the parameters that are associated with these musical aspects by using simple least-squares linear regression. By regressing the data with only one, or a few components that together correspond to the levels of a musical aspect, the prediction for only that component (such as beat) or aspect (combined levels of meter) can be formed. By calculating how much variance is explained by each aspect or component, the size and significance of that part of the model can be assessed. This procedure is very similar to the deconvolution algorithms used in the analysis of fMRI data (Glover, 1999). In this way, we have different predictor variables for different time segments, depending on the note within the sequence. Furthermore, as we fit the data separately per EEG channel, the topology of the component responses can be investigated, yielding insight into the different cognitive modules involved in the processing of the various aspects of musical structure. In the model we created, we used a three-layered rhythmic structure (‘First’, or the start of a phrase; ‘Beat’, or the pulse of the rhythm, and ‘Note’, the lowest rhythmic level using every event) and contour direction, based on the up–down pattern of the pitch sequence. These particular aspects were selected based on previous report of a distinguishable response in the EEG signal, such as Janata (2001) and Brochard et al. (2003) for rhythm and Trainor et al. (2002) for contour direction changes.