Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

4 Oct 2024 | Dongyang Li1*, Chen Wei1*, Shiying Li1, Jiachen Zou1, Haoyang Qin1, Quanying Liu1†
This study presents an end-to-end EEG-based zero-shot visual reconstruction framework, which includes a tailored brain encoder called the Adaptive Thinking Mapper (ATM) and a two-stage multi-pipe EEG-to-image generation strategy. The ATM projects neural signals from different sources into a shared subspace as clip embeddings, while the two-stage strategy first aligns the high-level clip embedding with EEG and then refines the EEG embedding into image priors using a prior diffusion model. A blurry image is also decoded from EEG to maintain low-level features. In the second stage, both high-level clip embedding, blurry image, and caption from EEG latent are input into a pre-trained diffusion model to generate images. The framework is evaluated on both EEG and MEG data, demonstrating state-of-the-art performance in classification, retrieval, and reconstruction tasks. The results highlight the portability, low cost, and high temporal resolution of EEG, making it suitable for a wide range of brain-computer interface (BCI) applications. The code for this framework is available at <https://github.com/neclab-sustech/EEG_Image_decode>.This study presents an end-to-end EEG-based zero-shot visual reconstruction framework, which includes a tailored brain encoder called the Adaptive Thinking Mapper (ATM) and a two-stage multi-pipe EEG-to-image generation strategy. The ATM projects neural signals from different sources into a shared subspace as clip embeddings, while the two-stage strategy first aligns the high-level clip embedding with EEG and then refines the EEG embedding into image priors using a prior diffusion model. A blurry image is also decoded from EEG to maintain low-level features. In the second stage, both high-level clip embedding, blurry image, and caption from EEG latent are input into a pre-trained diffusion model to generate images. The framework is evaluated on both EEG and MEG data, demonstrating state-of-the-art performance in classification, retrieval, and reconstruction tasks. The results highlight the portability, low cost, and high temporal resolution of EEG, making it suitable for a wide range of brain-computer interface (BCI) applications. The code for this framework is available at <https://github.com/neclab-sustech/EEG_Image_decode>.
Reach us at info@study.space
[slides and audio] Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion