EEG2Mel: Reconstructing Sound from Brain Responses to Music

Here we train and validate models with the Naturalistic Music Electroencephalogram Dataset – Tempo (NMED-T) and Naturalistic Music Electroencephalogram Dataset – Hindi (NMED-H) [9, 10]. Both datasets collect long recordings of passive brain responses to participants listening to music. The NMED-T contains recordings from twenty participants listening to ten different songs that were selected by the researchers because of the lack of familiarity and differences in tempo. On the other hand, the NMED-H has recordings from twelve participants listening to four pop songs in Hindi. These publicly available datasets have been central to various signal processing studies along with some studies looking into music retrieval from cortical activity [8, 11]. To aid with replication we provide guiding python notebooks as examples to our methods and analyses for others to follow. The supplementary materials also provide examples for others to listen to and visually inspect. Both datasets provide preprocessed versions of the data which include standard corrections for faulty channels, line-noise filtering, and muscle artefact corrections [9, 10]. Here we use those preprocessed versions of the data, as our focus is on passive cortical signals and not on any correlated motor behaviors that may show up as muscle artefacts.

Our modeling approach contrasts with previous publications showing success with decoding and reconstructing complex stimuli, where generative models such as VAEs and GANs reconstructed image classes and where VCCA was able to reconstruct music stimuli spectra at the expense of needing a multi-view of extracted features [7, 8]. We choose a straightforward approach with a sequential CNN based regressor mapping EEG input directly to the time-aligned music spectra. Table 1 shows a summary of how the model architecture is constructed; the models contain five convolutional layers with the last convolutional layer being the first layer that reduces the dimensionality of the input. A Max Pooling layer with a small pool size is chosen over a Global Average Pooling layer used previously during EEG classification [6] using the same dataset, because it was necessary to limit shrinkage since the output layer was the size of the spectral target.

Regularization in the model was implemented through dropout layers and maintaining the size of intermediate fully connected layer as small as possible. Dropout layers between convolutional layers were kept with 10% dropout and the dropout layer between the fully connected layers was kept at 15%. The amount and strength of dropout layers were tested demonstrating no evident effect with less layers and weaker layers, stronger values were not helpful between convolutional layers but did show some improvement between fully connected layers. Activation and Kernel L2 regularization was tried but opted out due to showing signs of a vanishing gradient. Number of filters for convolutional layers were kept at a base 8 while increasing by a power of 2 for every subsequent layer. Increasing the number of filters showed training loss outpacing validation loss resulting in overfitted model runs, while decreasing the number of filters made training loss stagnate too early which resulted in underfitting. For all intermediate layers a Rectified Linear-Unit activation was used with a linear output activation. Non-monotonic activation functions Swish and Mish were tried due to their success in improving image processing in deep networks, but they did not present any evident advantage over ReLu during training. Lastly, we use Adaptive Moment Estimation (Adam) as an optimizer with a 0.0015 learning rate and initialized weights with a He uniform distribution which have shown advantages in similar training procedures [6, 12].

The first four minutes of all recordings were used and cut up into five second chunks. To balance the train and test set distributions across time, we assign every other chunk to either train or test at a 75/25 ratio. Then all chunks were split into 1 second examples and randomly shuffled for training and validation. Five second chunks were also useful in securing consecutive 1 second examples to be reconstructed. These reconstructed five second music spectra were inverted into waveforms and used as examples in a behavioral experiment to validate the quality of brain to music reconstruction. Generalization stays within participants and across unseen chunks of time that are balanced to be sampled from the beginning, middle, and end of the songs. Other EEG studies also keep generalization within participants [13, 5], and a recent study has shown that weak correlations across participants in the NMED-T could be why generalizing to unseen participants is difficult without many recorded participants in the dataset [14].

Using the NMED-T, four reconstruction models were trained and evaluated depending on their varying input and target representations. Figure 2 outlines the combinations between input and target representations along with their visual qualities. Each arrow in the figure is a model showing the designated input to target mapping. Prior classification studies have shown a preference for input representations either being raw voltage or a Power Spectral Density (PSD), and their ability to boost performance in CNNs [5, 6]. As a regression task, we also find it important to compare target representations since linear and mel-spectrogram representations have tradeoffs. The linear spectrum scales low frequency activity equally to higher frequency, which allows it to be more easily inverted back into a listenable waveform. On the other hand, the mel-spectrogram applies a non-linear scaling across frequency ranges that has made it easier to map in classification tasks [15, 16], but also adds more noise during the inversion to a listenable waveform.

The reconstruction models were all trained until they reached a plateau in their loss with varying epoch ranges ( 100-300) needed for the models to converge their training and validation performance. Given that the mean-squared error (MSE) loss was only indicative of the convergence of the model during training and not a metric that informs how image reconstruction compares across models, we decided to classify the output as an objective quantitative measure. Simply put, the four trained regressors were used to create new datasets from their outputs given the original NMED-T as input and keeping the same train-test split. Then those new output datasets were passed through a classifier that would attempt to classify the name of the song from each spectral image reconstruction. This type of validation has been used before in the image stimuli domain when trying to objectively test the quality of reconstructed image classes from generative models [7]. The classifiers were CNN based following the architecture from a previous study where the EEG was classified into song name classes [6]. Table 2 demonstrates a summary of the results from the classified outputs across modeling approaches. The raw input representation showed to not give the best results in our deep CNN regressors but it did remain well above the chance performance rate of 10%. Meanwhile, the mel-spectrogram worked better across both the raw and PSD input representations with performance comparable to studies where only EEG classification was conducted [17, 13, 18].

Classifying outputs to their stimuli classes reveals fidelity of semantically relevant reconstructions. This was especially useful in the image stimuli domain when the models being evaluated focused at the stimuli class level [7]. Furthermore, their results of classification stayed well above chance (2.5% chance) for forty classes with performance 4̃0%. The results from Table 2 also stand well above chance for 10 classes (10% chance). Given that our training method maps EEG to the target stimuli directly rather than estimating stimuli class densities in the model’s latent space, common image reconstruction metrics were used to see how different each reconstruction was from the target image. Structural Similarity Index (SSI) and Peak Signal to Noise Ratio (PSNR) were chosen because of the ability to compare the reconstruction of perceived image features and how much noise the reconstructions contain from a lossy compression comparison respectively [19]. Figure 3 summarizes the results from both metrics across the test set of the image reconstructions from the regressor models trained on the PSD input representation along with their linear and mel-spectrogram targets. SSI measures, as a normalized ratio, the similarity between images where an SSI score of 1 points to the images being identical. The mean SSI is higher in the mel-spectrogram reconstruction than with the linear-spectrogram by  14%. PSNR on the other hand, measures as a ratio the logged maximum image power over its mean squared error which gives a relative difference between images and not a normalized score. We also see in Figure 3 that mel-spectrogram has a higher mean PSNR which further validates how the mel-spectrogram makes the EEG to music spectra reconstruction closer to the target.

To better understand what musical features the model learned from mapping to the mel-spectrogram, we took SSI and PSNR scores across frequencies. Figure 4 demonstrates the results of this analysis; for every example we paired up each 1 second frequency bin from target to model output to calculate both an SSI and PSNR value. Boxplots were created for all 128 frequency bins, and Figure 4 shows that frequencies vary around a 0.4 SSI score. Meanwhile, the PSNR plot shows notably lower values for the first 5 frequency bins. We show a visual example of this difference by taking a reconstruction and zooming into the ten lowest and highest frequency bins. This visual comparison demonstrates how many more pixel differences the lower frequencies have when compared to the higher frequencies. This went against our expectations as we believed it would be easier to map to lower frequencies because of pattern regularity [20]. Further testing is needed as future studies adopt this methodology but we speculate that this could be attributed to these model architectures not explicitly learning temporal dependencies such as in an RNN.

The above-mentioned metrics were useful in evaluating the success of the spectral image reconstructions, but here we are also interested in whether that allows for the retrieval of perceptually interpretable music reconstruction. This means that our spectral image reconstructions should be good enough to then be processed by common out of the box signal processing libraries that invert spectrograms to soundwaves, and then be identified by listeners as matching the original stimulus. For a practical experimental design and for testing the generalization of our methodology, we decided to train models and produce reconstructed spectral outputs from the NMED-H. Instead of a total of ten songs like in the NMED-T, the four songs from the NMED-H made it easier to test reconstructions across time slices and recorded participants; the training procedure in the deep CNN regressors stayed the same. Models trained on this dataset converged sooner and provided a steeper gradient descent during training which could be attributed to the familiarity of songs from participants [21]. Because the best performing models using the NMED-T used the PSD and mel-spectrogram as input-target representations, we focused our training on the NMED-H with that specific representation pairing. Figure 5 provides a visual summary of the quality of reconstructions from the NMED-H.

As a tradeoff for efficient modeling procedures and feature mapping within the proposed methodology, processing the outputs to listenable waveforms faces several steps of lossy transformations. The first comes from the actual deep learning models themselves where the input to target mapping assumes that the music signal is hidden/scrambled in cortical activity and attempts to recover that into a spectral representation. Second, those model outputs must be denormalized and set to a decibel range. Lastly, the spectrograms are transformed into waveforms by approximating the Short-Time Fourier Transform (STFT) magnitude from the mel power spectrum and reconstruction of the phase is done using Griffin Lim Algorithm (GLA). The mel-spectrogram in comparison to the linear is more lossy in the last two steps because mel is scaled as a non-linear function that represents human auditory perception across frequencies. Nevertheless, our modeling procedure finds it easier to map mel targets, so we commit to lossy tradeoffs and adjust to produce listenable examples to present to participants during the behavioral evaluation. During the denormalization step we use the following transformation:

In Equation 1, $\operatorname{mel}^{T}$ is the mel-spectrogram we aim to denormalize set as a matrix transpose and $\operatorname{Max\,dB}$ is set as a constant to 100 while $\operatorname{ref\,dB}$ is set as a constant within a song but varies across songs. To avoid presenting jarring noise to participants, $\operatorname{ref\,dB}$ becomes a free parameter set to [46 dB, 46 dB, 52 dB, 43 dB] for songs 1-4 respectively. The parameter value was set simply by randomly sampling 5 second examples and listening to how they would sound with values between 40 dB – 60 dB. The evaluation of which value produced the least jarring example was decided by the researcher; therefore we encourage any other studies following this methodology to play around with their own constraints relevant to their own studies. In Figure 5 it is demonstrated how well the waveforms come out despite the lossyness of the inversion process. The reconstructed spectrograms in Figure 5 shows how information across frequencies are recovered and how that translates to alignment in amplitude from target to reconstructed waveform.

The produced listenable examples were then presented to participants in a listening task to evaluate whether their quality was good enough to be identifiable. The experiment was a two-alternative AB-X task in which a participant had to pick between sounds A or B that matches the unlabeled sound X. The target sound X was a 5 second example from one of the four song stimuli in the NMED-H. Sounds A or B contained a non-corresponding foil and a corresponding 5 second reconstruction from our deep regressor model that was taken from spectra to a waveform. Participants were tested on a total of 24 trials plus 4 practice trials at the start of the experiment. The reconstructed model output produced a total of forty-eight examples which then half of the examples were used for one version of the experiment and the other half for a second version. Each version of the experiment controlled for balancing presentations across time by picking examples that belonged to the beginning, middle, and end of the songs. Furthermore, the experiment also contained the same number of examples per song while controlling for each trial to have reconstructions coming from different participants from the NMED-H. Trials were all randomized and foils balanced to minimize the repetition of reconstructions from a specific song or participant. During the AB-X discrimination task, participants were given a max of 30 minutes to finish and were allowed to listen to the sound samples as many times as they liked before making their choices, while also not taking longer than 1 minute per presentation. This experiment collected responses from a total of 16 participants with 8 participants for each version of the task. On average, participants had an 85% success rate (50% chance) with max performance of 95.83% and minimum performance of 66.67%. The average performance lines up with output classification, adding evidence of not just robust image reconstruction but also the perceived interpretability of listeners when inverting the spectrum.