BrainDecoder: Style-Based Visual Decoding
of EEG Signals^††thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Prior work [17] [19] [20] has demonstrated the ability of CLIP image embeddings to facilitate both semantic and stylistic transfer when the generator model is conditioned accordingly. Building on these findings, our approach aims to extract image-related information from EEG signals by aligning them with CLIP image embeddings. To achieve this, we process the EEG signals and their corresponding ground truth images (i.e., the ones that the EEG subject was watching when the signal was taken) using an EEG encoder and a CLIP image encoder, respectively, and aim to correlate the outputs. Let $F_{\text{image}}$ be an encoder that processes the input EEG signal $x$, and $E_{\text{image}}$ the CLIP image encoder applied to the input image $I$. We call $F_{\text{image}}$ the EEG-image encoder because it is trained to align with the image as encoded by $E_{\text{image}}$. We employ Mean Squared Error (MSE) as the loss function to measure the similarity between the EEG and image representations:

To effectively encode the EEG data, we extend upon previous approaches [2] [21] [6] by utilizing an LSTM-based encoder architecture followed by fully connected layers.

Recent approaches for visual brain signal decoding [11] [22] have sought to align brain signals with CLIP [15] text embeddings obtained from captions generated by pretrained image caption generators. However, since CLIP was trained on image-text pairs publicly available on the Internet with often short captions, those methods using longer generated captions, particularly with Stable Diffusion [16], have been less effective. Although Stable Diffusion allows up to 77 tokens as input, empirical evidence suggests that the effective token length of CLIP is considerably shorter [23]. Accordingly, we adopt a simpler labeling approach: we make the caption by appending the class label of the image to the text “an image of”. We show empirically that this method improves performance over previous approaches and that more fine-grained information can be captured by the EEG-image encoder instead. We use the CLIP text encoder $E_{\text{text}}$ that embeds the caption $C$ to train the EEG-text encoder $F_{\text{text}}$ that encodes the corresponding EEG signal $x$. As in the image alignment step, we use MSE as the loss function to quantify the similarity between the EEG and the text representations:

Similar to the image processing pipeline, an LSTM-based encoder is used for EEG signal encoding.

After training the EEG-image and EEG-text encoders, we leverage the resulting EEG embeddings to generate images. Our method uses a pretrained latent diffusion model (e.g., Stable Diffusion [16]), with the EEG embeddings from both encoders serving as distinct conditioning inputs. This is achieved through a decoupled cross-attention mechanism [17]. We hypothesize that by aligning the EEG signals in CLIP image space, the EEG encoder can capture detailed semantics and style that may not be easily conveyed through text alone. This approach is analogous to the way latent diffusion models incorporate both text and image prompts as conditioning factors. The reconstructed visual stimuli are defined as:

Here, $\hat{y}$ represents the reconstructed image, and $\mathrm{SD}$ denotes the pretrained Stable Diffusion, conditioned on the outputs of both the EEG-image encoder $F_{\text{image}}$ and the EEG-text encoder $F_{\text{text}}$.

We utilize the Brain2Image [21] [2], an EEG-image pair dataset with 11,466 EEG recordings from six participants, for our experiments. These recordings were captured using a 128-channel EEG system as the participants were exposed to visual stimuli for 500 ms. The stimuli consisted of 2,000 images with labels spanning 40 categories, derived from the ImageNet dataset [24]. Each category included 50 easily recognizable images to ensure clarity in the participants’ neural responses.

For the EEG encoders, we extend from previous approaches [21] [2] [6] and use a 3-layered LSTM network with a hidden dimension of 512. The output of the network is then passed through a fully connected linear network with a BatchNorm [25] and LeakyReLU [26] activation function in between. Only the EEG encoders are trained in our framework, keeping the framework computationally efficient. We use the Adam [27] optimizer with a weight decay of 0.0001. The initial learning rate is set to 0.0003 and a lambda learning rate scheduler is used with a lambda factor of 0.999.

In order to align with CLIP image space, we follow the approach outlined in the IP-Adapter [17] framework, utilizing the CLIP-Huge model to process the images. For aligning EEG with CLIP text space, we process the captions using the CLIP-Large model which is used by Stable Diffusion 1.5. The captions are generated by concatenating “an image of” with the class label. For ablation studies, we employ the LLaVA-1.5-7b model [28] for layout-oriented caption generation and BLIP [29] for general caption generation. For visual reconstruction, we employ Stable Diffusion version 1.5, aligning our method with recent results for fair comparison and we employ a PNDM scheduler [30] with 25 inference steps.

We employ the following metrics to objectively assess the performance of our framework. ACC: The $N$-way Top-$K$ Classification Accuracy [8] [31] evaluates the semantic accuracy of the reconstructed images. We set $N=50$ and $K=1$. IS: The Inception Score [32] assesses the diversity and quality of the generated images. FID: Fréchet inception distance [33] measures the distance from the ground truth images. SSIM: The Structural Similarity Index Measure [34] evaluates the quality of images. CS: CLIP Similarity [35] [11] reflects how well the generated images capture the semantic and stylistic content of the ground truth images.

Fig. 2 presents sample outputs of BrainDecoder. Beyond capturing the high-level semantics, our method demonstrates the ability to retain fine-grained visual features, including color and texture. Notably, there is also a resemblance in the color composition of the background in addition to the main object’s color. This capability is further illustrated in the example of the electric locomotive class. The object’s color is depicted as light blue—matching the visual stimuli—despite the range of potential color variations. This demonstrates the model’s ability to recover nuanced visual attributes with a high fidelity.

This is further demonstrated in Fig. 3, where we compare our results with prior studies. Notably, in the second image, our method is able to reconstruct not only the wooden texture of the chair, but the grass in the background as well, which was absent in the results by other methods.

Table. I shows the quantitative results of BrainDecoder compared to baselines [2] [12] [11] [7]. We evaluate our methodology on 5 evaluation metrics in §III-C. BrainDecoder outperforms the state-of-the-art in both reconstruction fidelity and generation quality. Notably, BrainDecoder achieves a surprising 95.2% on the 50-way top-1 classification accuracy metrics, showing that the trained EEG encoders are able to extract rich information from the brain signals very well.

We conduct an ablation study to understand the contributions of each component. Rows 3-5 of Table II show visual decoding with the EEG-image encoder yields a higher SSIM (0.2239) than with only the EEG-text encoder (0.1845). This supports our premise that aligning in CLIP’s image space facilitates style transfer. Furthermore, the framework achieves the best performance when both encoders are used, indicating the complementary nature of the two encoders.

Additionally, we empirically show that captions generated by Vision Language Models (VLMs) are suboptimal for EEG-based visual decoding. We compare our label caption method with two VLMs: BLIP [29] and layout-oriented LLaVA [28]. A key challenge in image reconstruction from brain signals is preserving the visual layout. We hypothesize that associating EEG signals with detailed layout-oriented CLIP text embeddings might help. Using the LLaVA model, we generate layout-oriented captions following the instruction: “Write a description of the image layout. EXAMPLE OUTPUT: [object] is in the top left of the image, facing right.” Fig. 4 shows example layout-oriented captions. Notably, rows 1-3 of Table II indicate that the simple label caption (“an image of [class name]”) performs best, while layout-oriented captions (row 1) perform the worst. This further supports our premise that simple label captions are more effective for EEG encoders and complex prompts are harder for CLIP to fully interpret.