Brain2GAN: Feature-disentangled neural coding of visual perception in the primate brain

28 April 2023

Thirza Dado, Paolo Papale, Antonio Lozano, Lynn Le, Feng Wang, Marcel van Gerven, Pieter Roelfsema, Yağmur Güçlütürk & Umut Güçlü

Abstract

A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, we analyzed the relationship between multi-unit activity of macaque visual cortex and latent representations of state-of-the-art deep generative models, including feature-disentangled representations of generative adversarial networks (i.e., w-latents of StyleGAN) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled representations explain increasingly more variance than the alternative representations over the ventral stream. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.

Preprint / GitHub repository / Poster

Test set stimulus (top) reconstruction (bottom) from brain activity.

Time-based reconstruction: meaningful information is extracted from the stimulus-evoked brain responses in time.