Convolutional neural networks as a model for brain area MSTd

Abstract

AbstractAccuracy-optimized convolutional neural networks (CNNs) have emerged as highly effective models at predicting neural responses in brain areas along the primate ventral stream, but it is largely unknown whether they effectively model neurons in the complementary primate dorsal stream. We explored how well CNNs model the optic flow tuning properties of neurons in dorsal area MSTd and we compared our results with the Non-Negative Matrix Factorization (NNMF) model proposed by Beyeler, Dutt, & Krichmar (2016), which successfully models many tuning properties of MSTd neurons. To better understand the role of computational properties in the NNMF model that give rise to MSTd-like optic flow tuning, we created additional CNN model variants that implement key NNMF constraints — non-negative weights and sparse coding of optic flow. While the CNNs and NNMF models both accurately estimate the observer’s self-motion from purely translational or rotational optic flow, NNMF and the CNNs with nonnegative weights yield substantially less accurate estimates than the other CNNs when tested on more complex optic flow that combines observer translation and rotation. Despite their poor accuracy, however, neurons in the networks with the nonnegativity constraint give rise to tuning properties that align more closely with those observed in primate MSTd. Interestingly, the addition of the sparsity constraint has a negligible effect on the accuracy of self-motion estimates and model tuning properties. Across all models, we consistently observe the 90-degree offset in the preferred translation and rotation directions found in MSTd neurons, which suggests that this property could emerge through a range of potential computational mechanisms. This work offers a step towards a deeper understanding of the computational properties and constraints that describe optic flow tuning primate area MSTd.Significance StatementOne of the most exciting developments in visual neuroscience over the past decade is that convolutional artificial neural networks optimized to accurately categorize natural images effectively model neural activity in ventral visual areas of the primate brain. We explored whether accuracy-optimized neural networks account for well-established properties of MSTd, a brain area in the complementary primate dorsal stream that is involved in self-motion perception during navigation. Our findings indicate that such networks depart substantially from MSTd-like tuning, which suggests the computational goal of MSTd may not be to accurately estimate self-motion. We found that adding computational constraints inspired by an existing MSTd model that performs dimensionality reduction on afferent motion signals improves the correspondence with MSTd.