Papers

Deep convolutional neural networks are important models of the visual cortex that account relatively well for brain activity and are able to perform ethologically relevant functions. However, it is unknown which combination of factors, such as network architecture, training objectives, and data best align this family of models with the brain. Here we investigate the statistics of training data. We hypothesized that stimuli that are naturalistic for mice would lead to higher similarity between deep network models and activity in mouse visual cortex. We used a video-game engine to create training datasets in which we varied the naturalism of the environment, the movement statistics, and the optics of the modelled eye. The naturalistic environment substantially and consistently led to greater brain similarity, while the other factors had more subtle and area-specific effects. We then hypothesized that differences in brain similarity between the two environments arose due to differences in spatial frequency spectra, distributions of color and orientation, and/or temporal autocorrelations. To test this, we created abstract environments, composed of cubes and spheres, that resembled the naturalistic and non-naturalistic environments in these respects. Contrary to our expectations, these factors accounted poorly for differences in brain similarity due to the naturalistic and non-naturalistic environments. This suggests that the higher brain similarities we observed after training with the naturalistic environment were due to more complex factors. 

The primate visual cortex contains various regions that exhibit specialization for different stimulus properties, such as motion, shape, and color. Within each region, there is often further specialization, such that particular stimulus features, such as horizontal and vertical orientations, are over-represented. These asymmetries are associated with well-known perceptual biases, but little is known about how they influence visual learning. Most theories would predict that learning is optimal, in the sense that it is unaffected by these asymmetries. However, other approaches to learning would result in specific patterns of perceptual biases. To distinguish between these possibilities, we trained human observers to discriminate between expanding and contracting motion patterns, which have a highly asymmetrical representation in the visual cortex. Observers exhibited biased percepts of these stimuli, and these biases were affected by training in ways that were often suboptimal. We simulated different neural network models and found that a learning rule that involved only adjustments to decision criteria, rather than connection weights, could account for our data. These results suggest that cortical asymmetries influence visual perception and that human observers often rely on suboptimal strategies for learning.

Background:Cortical blindness is a form of severe vision loss that is caused by damage to the primary visual cortex (V1) or its afferents. This condition has devastating effects on quality of life and independence. While there are few treatments currently available, accumulating evidence shows that certain visual functions can be restored with appropriate perceptual training: Stimulus sensitivity can be increased within portions of the blind visual field. However, this increased sensitivity often remains highly specific to the trained stimulus, limiting the overall improvement in visual function. Objective:Recent advances in the field of perceptual learning show that such specificity can be overcome with training paradigms that leverage the properties of higher-level visual cortical structures, which have greater capacity to generalize across stimulus positions and features. This targeting can be accomplished by using more complex training stimuli that elicit robust responses in these visual structures. Methods:We trained cortically blind subjects with a complex optic flow motion stimulus that was presented in a location of their blind field. Participants were instructed to train with the stimulus at home for approximately 30 minutes per day. Once performance plateaued, the stimulus was moved deeper into the blind field. A battery of pre- and post-training measures, with careful eye tracking, was performed to quantify the improvements. Results:We show that 1) optic flow motion discrimination can be relearned in cortically blind fields; 2) training with an optic flow stimulus can lead to improvements that transfer to different tasks and untrained locations; and 3) such training leads to a significant expansion of the visual field. The observed expansion of the visual field was present even when eye movements were carefully controlled. Finally, we show that regular training is critical for improved visual function, as sporadic training reduced the benefits of training, even when the total numbers of training sessions were equated. Conclusions:These findings are consistent with the hypothesis that complex training stimuli can improve outcomes in cortical blindness, provided that patients adhere to a regular training regimen. Nevertheless, such interventions remain limited in their ability to restore functional vision.

The visual system of mammals is comprised of parallel, hierarchical specialized pathways. Different pathways are specialized in so far as they use representations that are more suitable for supporting specific downstream behaviours. In particular, the clearest example is the specialization of the ventral ("what") and dorsal ("where") pathways of the visual cortex. These two pathways support behaviours related to visual recognition and movement, respectively. To-date, deep neural networks have mostly been used as models of the ventral, recognition pathway. However, it is unknown whether both pathways can be modelled with a single deep ANN. Here, we ask whether a single model with a single loss function can capture the properties of both the ventral and the dorsal pathways. We explore this question using data from mice, who like other mammals, have specialized pathways that appear to support recognition and movement behaviours. We show that when we train a deep neural network architecture with two parallel pathways using a self-supervised predictive loss function, we can outperform other models in fitting mouse visual cortex. Moreover, we can model both the dorsal and ventral pathways. These results demonstrate that a self-supervised predictive learning approach applied to parallel pathway architectures can account for some of the functional specialization seen in mammalian visual systems.

Neurons in the dorsal visual pathway of the mammalian brain are selective for motion stimuli, with the complexity of stimulus representations increasing along the hierarchy. This progression is similar to that of the ventral visual pathway, which is well characterized by artificial neural networks (ANNs) optimized for object recognition. In contrast, there are no image-computable models of the dorsal stream with comparable explanatory power. We hypothesized that the properties of dorsal stream neurons could be explained by a simple learning objective: the need for an organism to orient itself during self-motion. To test this hypothesis, we trained a 3D ResNet to predict an agent's self-motion parameters from visual stimuli in a simulated environment. We found that the responses in this network accounted well for the selectivity of neurons in a large database of single-neuron recordings from the dorsal visual stream of non-human primates. In contrast, ANNs trained on an action recognition dataset through supervised or self-supervised learning could not explain responses in the dorsal stream, despite also being trained on naturalistic videos with moving objects. These results demonstrate that an ecologically relevant cost function can account for dorsal stream properties in the primate brain.

Visual perceptual learning (VPL) is an improvement in visual function following training. Although the practical utility of VPL was once thought to be limited by its specificity to the precise stimuli used during training, more recent work has shown that such specificity can be overcome with appropriate training protocols. In contrast, relatively little is known about the extent to which VPL exhibits motor specificity. Previous studies have yielded mixed results. In this work, we have examined the effector specificity of VPL by training observers on a motion discrimination task that maintains the same visual stimulus (drifting grating) and task structure, but that requires different effectors to indicate the response (saccade vs. button press). We find that, in these conditions, VPL transfers fully between a manual and an oculomotor response. These results are consistent with the idea that VPL entails the learning of a decision rule that can generalize across effectors.

Dynamic latent variable modelling has provided a powerful tool for understanding how populations of neurons compute. For spiking data, such latent variable modelling can treat the data as a set of point-processes, due to the fact that spiking dynamics occur on a much faster timescale than the computational dynamics being inferred. In contrast, for other experimental techniques, the slow dynamics governing the observed data are similar in timescale to the computational dynamics that researchers want to infer. An example of this is in calcium imaging data, where calcium dynamics can have timescales on the order of hundreds of milliseconds. As such, the successful application of dynamic latent variable modelling to modalities like calcium imaging data will rest on the ability to disentangle the deeper- and shallower-level dynamical systems’ contributions to the data. To-date, no techniques have been developed to directly achieve this. Here we solve this problem by extending recent advances using sequential variational autoencoders for dynamic latent variable modelling of neural data. Our system VaLPACa (Variational Ladders for Parallel Autoencoding of Calcium imaging data) solves the problem of disentangling deeper- and shallower-level dynamics by incorporating a ladder architecture that can infer a hierarchy of dynamical systems. Using some built-in inductive biases for calcium dynamics, we show that we can disentangle calcium flux from the underlying dynamics of neural computation. First, we demonstrate with synthetic calcium data that we can correctly disentangle an underlying Lorenz attractor from calcium dynamics. Next, we show that we can infer appropriate rotational dynamics in spiking data from macaque motor cortex after it has been converted into calcium fluorescence data via a calcium dynamics model. Finally, we show that our method applied to real calcium imaging data from primary visual cortex in mice allows us to infer latent factors that carry salient sensory information about unexpected stimuli. These results demonstrate that variational ladder autoencoders are a promising approach for inferring hierarchical dynamics in experimental settings where the measured variable has its own slow dynamics, such as calcium imaging data. Our new, open-source tool thereby provides the neuroscience community with the ability to apply dynamic latent variable modelling to a wider array of data modalities.

Many brain functions are difficult to localize, as they involve distributed networks that reconfigure themselves on short timescales. One example is the integration of oculomotor and visual signals that occurs with each eye movement: The brain must combine motor signals about the eye displacement with retinal signals, to infer the structure of the surrounding environment. Our understanding of this process comes primarily from singleneuron recordings, which are limited in spatial extent, or fMRI measurements, which have poor temporal resolution. We have therefore studied visual processing during eye movements, using magnetoencephalography (MEG), which affords high spatiotemporal resolution. Human subjects performed a task in which they reported the orientation of a visual stimulus while executing a saccade. After removal of eye movement artifacts, time-frequency analysis revealed a signal that propagated in the beta-frequency band from parietal cortex to visual cortex. This signal had the characteristics of perisaccadic "remapping", a neural signature of the integration of oculomotor and visual signals. These results reveal a novel mechanism of visual perception and demonstrate that MEG can provide a useful window into distributed brain functions.

Although the structure and function of the human visual system are determined in large part during early development, there is ample evidence for adult plasticity as well. Such plasticity has important consequences for restoring vision after cortical damage and for improving function in healthy people. Although these applications have shown promising results, they are often limited by pathological specificity: improvements obtained through perceptual training fail to generalize beyond the trained stimulus feature or location. Efforts to reduce specificity have focused on the design of training tasks, but less is known about the effects of stimulus structure on the specificity of perceptual learning. Here, we leverage physiological findings from the dorsal visual pathway of the primate brain to explore the hypothesis that learning specificity is related to the complexity of the training stimulus. Specifically, because neurons in higher-level structures of the dorsal visual pathway exhibit little stimulus specificity, we reasoned that training with more complex stimuli would reduce the specificity of learning. We trained human observers on stimuli of varying complexity, ranging from simple sinewave gratings to complex optic flow fields. Our results show that training with more complex stimuli reduces specificity for spatial position and stimulus features. Such changes are associated with increased spatial integration. These findings were captured by a computational “reweighting” model that decoded the outputs of simulated neurons in areas middle temporal (MT) and medial superior temporal (MST) of the primate visual cortex. Our results suggest that the addition of more complex stimuli into perceptual learning paradigms provides a simple and effective way to minimize specificity in learning.

Inhibiting inappropriate actions in a context is an important part of the human cognitive repertoire, and deficiencies in this ability are common in neurological and psychiatric disorders. An anti-saccade is a simple oculomotor task that tests this ability by requiring inhibition of saccades to peripheral targets (pro-saccade) and producing voluntary eye movements toward the mirror position (anti-saccades). Previous studies provide evidence for a possible contribution from the basal ganglia in anti-saccade behavior, but the precise role of different components is still unclear. Parkinson’s disease patients with implanted deep brain stimulators (DBS) in subthalamic nucleus (STN) provide a unique opportunity to investigate the role of the STN in anti-saccade behavior. Previous attempts to show the effect of STN DBS on anti-saccades have produced conflicting observations. For example, the effect of STN DBS on anti-saccade error rate is not yet clear. Part of this inconsistency may be related to differences in dopaminergic states in different studies. Here, we tested Parkinson’s disease patients on anti- and pro-saccade tasks ON and OFF STN DBS, in ON and OFF dopaminergic medication states. First, STN DBS increases anti-saccade error rate while patients are OFF dopamine replacement therapy. Second, dopamine replacement therapy and STN DBS interact: L-dopa reduces the effect of STN DBS on anti-saccade error rate. Third, STN DBS induces different effects on pro- and anti-saccades in different patients. These observations provide evidence for an important role for the STN in the circuitry underlying context-dependent modulation of visuomotor action selection.

Smooth pursuit eye movements have frequently been used to model sensorimotor transformations in the brain. In particular, the initiation phase of pursuit can be understood as a transformation of a sensory estimate of target velocity into an eye rotation. Despite careful laboratory controls on the stimulus conditions, pursuit eye movements are frequently observed to exhibit considerable trial-to-trial variability. In theory, this variability can be caused by the variability in sensory representation of target motion, or by the variability in the transformation of sensory information to motor commands. Previous work has shown that neural variability in the middle temporal (MT) area is likely propagated to the oculomotor command, and there is evidence to suggest that the magnitude of this variability is sufficient to account for the variability of pursuit initiation. This line of reasoning presumes that the MT population is homogeneous with respect to its contribution to pursuit initiation.  At the same time, there is evidence that pursuit initiation is strongly linked to a subpopulation of MT neurons (those with strong surround suppression) that collectively generate less motor variability. To distinguish between these possibilities, we have combined human psychophysics, monkey electrophysiology, and computational modeling to examine how the pursuit system reads out the MT population during pursuit initiation. We find that the psychophysical data are best accounted for by a model that gives stronger weight to surround-suppressed MT neurons, suggesting that variability in the initiation of pursuit could arise from multiple sources along the sensorimotor transformation.

The resting brain has been extensively investigated for low frequency synchrony between brain regions, namely Functional Connectivity (FC). However the other main stream of brain connectivity analysis that seeks causal interactions between brain regions, Effective Connectivity (EC), has been little explored. Inherent complexity of brain activities in resting-state, as observed in BOLD (Blood Oxygenation-Level Dependant) fluctuations, calls for exploratory methods for characterizing these causal networks. On the other hand, the inevitable effects that hemodynamic system imposes on causal inferences in fMRI data, lead us toward the methods in which causal inferences can take place in latent neuronal level, rather than observed BOLD time-series. To simultaneously satisfy these two concerns, in this paper, we introduce a novel state-space system identification approach for studying causal interactions among brain regions in the absence of explicit cognitive task. This algorithm is a geometrically inspired method for identification of stochastic systems, purely based on output observations. Using extensive simulations, three aspects of our proposed method are investigated: ability in discriminating existent interactions from non-existent ones, the effect of observation noise, and downsampling on algorithm performance. Our simulations demonstrate that Subspace-based Identification Algorithm (SIA) is sufficiently robust against above-mentioned factors, and can reliably uncover the underlying causal interactions of resting-state fMRI. Furthermore, in contrast to previously established state-space approaches in Effective Connectivity studies, this method is able to characterize causal networks with large number of brain regions. In addition, we utilized the proposed algorithm for identification of causal relationships underlying anti-correlation of default-mode and Dorsal Attention Networks during the rest, using fMRI. We observed that Default-Mode Network places in a higher order in hierarchical structure of brain functional networks compared to Dorsal Attention Networks.