Let's move forward: Image-computable models and a common model evaluation scheme are prerequisites for a scientific understanding of human vision

Behavioral and Brain Sciences · 2023 · 1 citations

• Computer Science
• Artificial Intelligence
• Computer Science

In the target article, Bowers et al. dispute deep artificial neural network (ANN) models as the currently leading models of human vision without producing alternatives. They eschew the use of public benchmarking platforms to compare vision models with the brain and behavior, and they advocate for a fragmented, phenomenon-specific modeling approach. These are unconstructive to scientific progress. We outline how the Brain-Score community is moving forward to add new model-to-human comparisons to its community-transparent suite of benchmarks.

Unsupervised neural network models of the ventral visual stream

Proceedings of the National Academy of Sciences · 2021 · 328 citations

• Artificial Intelligence
• Artificial Intelligence
• Computer Science

Deep neural networks currently provide the best quantitative models of the response patterns of neurons throughout the primate ventral visual stream. However, such networks have remained implausible as a model of the development of the ventral stream, in part because they are trained with supervised methods requiring many more labels than are accessible to infants during development. Here, we report that recent rapid progress in unsupervised learning has largely closed this gap. We find that neural network models learned with deep unsupervised contrastive embedding methods achieve neural prediction accuracy in multiple ventral visual cortical areas that equals or exceeds that of models derived using today's best supervised methods and that the mapping of these neural network models' hidden layers is neuroanatomically consistent across the ventral stream. Strikingly, we find that these methods produce brain-like representations even when trained solely with real human child developmental data collected from head-mounted cameras, despite the fact that these datasets are noisy and limited. We also find that semisupervised deep contrastive embeddings can leverage small numbers of labeled examples to produce representations with substantially improved error-pattern consistency to human behavior. Taken together, these results illustrate a use of unsupervised learning to provide a quantitative model of a multiarea cortical brain system and present a strong candidate for a biologically plausible computational theory of primate sensory learning.

Explanatory models in neuroscience: Part 1 -- taking mechanistic abstraction seriously

arXiv (Cornell University) · 2021 · 49 citations

• Computer Science
• Artificial Intelligence
• Computer Science

Despite the recent success of neural network models in mimicking animal performance on visual perceptual tasks, critics worry that these models fail to illuminate brain function. We take it that a central approach to explanation in systems neuroscience is that of mechanistic modeling, where understanding the system is taken to require fleshing out the parts, organization, and activities of a system, and how those give rise to behaviors of interest. However, it remains somewhat controversial what it means for a model to describe a mechanism, and whether neural network models qualify as explanatory. We argue that certain kinds of neural network models are actually good examples of mechanistic models, when the right notion of mechanistic mapping is deployed. Building on existing work on model-to-mechanism mapping (3M), we describe criteria delineating such a notion, which we call 3M++. These criteria require us, first, to identify a level of description that is both abstract but detailed enough to be "runnable", and then, to construct model-to-brain mappings using the same principles as those employed for brain-to-brain mapping across individuals. Perhaps surprisingly, the abstractions required are those already in use in experimental neuroscience, and are of the kind deployed in the construction of more familiar computational models, just as the principles of inter-brain mappings are very much in the spirit of those already employed in the collection and analysis of data across animals. In a companion paper, we address the relationship between optimization and intelligibility, in the context of functional evolutionary explanations. Taken together, mechanistic interpretations of computational models and the dependencies between form and function illuminated by optimization processes can help us to understand why brain systems are built they way they are.

Explanatory models in neuroscience: Part 2 -- constraint-based intelligibility

arXiv (Cornell University) · 2021 · 25 citations

• Computer Science
• Artificial Intelligence
• Cognitive science

Computational modeling plays an increasingly important role in neuroscience, highlighting the philosophical question of how computational models explain. In the context of neural network models for neuroscience, concerns have been raised about model intelligibility, and how they relate (if at all) to what is found in the brain. We claim that what makes a system intelligible is an understanding of the dependencies between its behavior and the factors that are causally responsible for that behavior. In biological systems, many of these dependencies are naturally "top-down": ethological imperatives interact with evolutionary and developmental constraints under natural selection. We describe how the optimization techniques used to construct NN models capture some key aspects of these dependencies, and thus help explain why brain systems are as they are -- because when a challenging ecologically-relevant goal is shared by a NN and the brain, it places tight constraints on the possible mechanisms exhibited in both kinds of systems. By combining two familiar modes of explanation -- one based on bottom-up mechanism (whose relation to neural network models we address in a companion paper) and the other on top-down constraints, these models illuminate brain function.

Pruning neural networks without any data by iteratively conserving\n synaptic flow

arXiv (Cornell University) · 2020 · 256 citations

• Computer Science
• Computer Science
• Artificial Intelligence

Pruning the parameters of deep neural networks has generated intense interest\ndue to potential savings in time, memory and energy both during training and at\ntest time. Recent works have identified, through an expensive sequence of\ntraining and pruning cycles, the existence of winning lottery tickets or sparse\ntrainable subnetworks at initialization. This raises a foundational question:\ncan we identify highly sparse trainable subnetworks at initialization, without\never training, or indeed without ever looking at the data? We provide an\naffirmative answer to this question through theory driven algorithm design. We\nfirst mathematically formulate and experimentally verify a conservation law\nthat explains why existing gradient-based pruning algorithms at initialization\nsuffer from layer-collapse, the premature pruning of an entire layer rendering\na network untrainable. This theory also elucidates how layer-collapse can be\nentirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow\nPruning (SynFlow). This algorithm can be interpreted as preserving the total\nflow of synaptic strengths through the network at initialization subject to a\nsparsity constraint. Notably, this algorithm makes no reference to the training\ndata and consistently competes with or outperforms existing state-of-the-art\npruning algorithms at initialization over a range of models (VGG and ResNet),\ndatasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to\n99.99 percent). Thus our data-agnostic pruning algorithm challenges the\nexisting paradigm that, at initialization, data must be used to quantify which\nsynapses are important.\n

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

arXiv (Cornell University) · 2020 · 129 citations

• Computer Science
• Computer Science
• Artificial Intelligence

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.