A contrastive coding account of category selectivity in the ventral visual stream

Abstract

ABSTRACTModular and distributed theories of category selectivity along the ventral visual stream have long existed in tension. Here, we present a reconciling framework, based on a series of analyses relating category-selective tuning within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, visual category-selective tuning naturally emerges for classic categories of faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits. Finally, these pre-identified units from a single model can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of the human visual system, even under a constrained sparse-positive encoding procedure. The success of this model indicates that the nature of category-selective tuning in the human brain (e.g. for faces or scenes) is dependent on the learning diet, and best understood in the context of the full scope of experienced visual input. Broadly, we offer a unifying theoretical account where category-selective tuning naturally emerges as a consequence of positive information routing through hierarchical population codes, in order to disentangle the statistics of visual experience.