Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity

Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity

29 Mar 2024 | Ruijie Quan, Wenguan Wang*, Zhibo Tian, Fan Ma, Yi Yang
Psychometry is an omnifit model designed to reconstruct images from functional Magnetic Resonance Imaging (fMRI) data, offering a more generalized approach compared to individually trained models. The model incorporates an Omni MoE (Mixture-of-Experts) module that enables all experts to collaboratively capture inter-subject commonalities while addressing individual differences through subject-specific parameters. Additionally, it employs a retrieval-enhanced inference strategy called Ecphory, which retrieves relevant CLIP embeddings from pre-stored memories to enhance fMRI representations. These components allow Psychometry to efficiently capture both inter-subject commonalities and individual variabilities, leading to high-quality image reconstructions. The model is trained once on aggregated fMRI data from multiple subjects, eliminating the need for separate models for each subject. This approach reduces computational resources and training time, while also improving performance compared to existing methods. Psychometry outperforms state-of-the-art methods in both qualitative and quantitative evaluations, demonstrating its effectiveness in reconstructing images from fMRI data. The model's ability to capture both commonalities and individual differences across subjects makes it a significant advancement in the field of brain-computer interfaces and image reconstruction from human brain activity.Psychometry is an omnifit model designed to reconstruct images from functional Magnetic Resonance Imaging (fMRI) data, offering a more generalized approach compared to individually trained models. The model incorporates an Omni MoE (Mixture-of-Experts) module that enables all experts to collaboratively capture inter-subject commonalities while addressing individual differences through subject-specific parameters. Additionally, it employs a retrieval-enhanced inference strategy called Ecphory, which retrieves relevant CLIP embeddings from pre-stored memories to enhance fMRI representations. These components allow Psychometry to efficiently capture both inter-subject commonalities and individual variabilities, leading to high-quality image reconstructions. The model is trained once on aggregated fMRI data from multiple subjects, eliminating the need for separate models for each subject. This approach reduces computational resources and training time, while also improving performance compared to existing methods. Psychometry outperforms state-of-the-art methods in both qualitative and quantitative evaluations, demonstrating its effectiveness in reconstructing images from fMRI data. The model's ability to capture both commonalities and individual differences across subjects makes it a significant advancement in the field of brain-computer interfaces and image reconstruction from human brain activity.
Reach us at info@study.space