Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

Abstract

AbstractIn recent years, substantial strides have been made in the field of visual image reconstruction, particularly in its capacity to generate high-quality visual representations from human brain activity while considering semantic information. This advancement not only enables the recreation of visual content but also provides valuable insights into the intricate processes occurring within high-order functional brain regions, contributing to a deeper understanding of brain function. However, considering fusion semantics in reconstructing visual images from brain activity is semantic-to-image guide reconstruction and basically ignores underlying neural computational mechanisms, which is actually not real reconstruction from brain activity. In response to this limitation, our study introduces a novel approach that combines linear mapping with nonlinear reconstruction to reconstruct visual images perceived by subjects based on their brain activity patterns. The primary challenge associated with linear mapping lies in its susceptibility to noise interference. To address this issue, we leverage a flexible denoised deep convolutional neural network, which surpasses the performance of traditional linear mapping. Our investigation encompasses linear mapping as well as the training of shallow and deep autoencoder denoised neural networks, including a pre-trained state-of-the-art denoised neural network. The outcome of our study reveals that the amalgamation of linear image decoding with nonlinear noise reduction significantly enhances the quality of reconstructed images from human brain activity. This suggests that our methodology holds promise for decoding intricate perceptual experiences directly from brain activity patterns without semantic information. Moreover, the model has strong neural explanatory power because it shares structural and functional similarities with the visual brain.