About

Steven Zucker is the David & Lucile Packard Professor of Biomedical Engineering & Computer Science at Yale University. His research focuses on computational vision, integrating insights from neurophysiology, mathematics, and computation to develop an abstract theory of vision. Zucker's work addresses the complexity of human visual processing, which involves a significant portion of the primate brain dedicated to visual information processing. He employs differential geometry to create methods for curve detection, shading and texture analysis, stereo, color, and shape description, aiming to characterize the function of billions of neurons in algorithmic terms. His interdisciplinary approach combines computation, neuroscience, and mathematics to advance understanding in the field of vision systems.

Research topics

• Computer Science
• Artificial Intelligence
• Mathematics
• Sociology
• Geometry
• Political Science
• Combinatorics
• Communication
• Computer vision
• Algorithm
• Psychology
• Cognitive psychology

Selected publications

• How 'Neural' is a Neural Foundation Model?

  PubMed · 2026-01-29

  articleSenior author

  module achieves high fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, this study provides a window into the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons' joint temporal response patterns. Our findings suggest design changes that could bring neural foundation models into closer alignment with biological systems: introducing recurrence in early encoder stages, and constraining features in the readout module.

  Publisher

• Orientation fields predict human perception of 3D shape from shading

  Proceedings of the National Academy of Sciences · 2025-07-10 · 2 citations

  articleOpen access

  How the brain recovers the three-dimensional structure of surfaces and objects from 2D retinal images remains mysterious. Shading patterns provide one of the most powerful-yet least understood-visual depth cues. Most theories assume the brain infers surface normals from luminance values. However, this seems unlikely as visual neurons are broadly insensitive to luminance. To identify alternative cues, we measured responses of model orientation-selective cell populations to images of shaded objects. We found a surprising statistical relationship between image orientations and surface curvature properties, suggesting a way to estimate shape from shading. We find that the orientation-based cues not only predict striking illusions of shape perception when lighting varies, but also the impressive robustness of shape perception when large image modifications are introduced to directly pit luminance and image orientation cues against one another. The findings resolve the longstanding question of which image measurements drive shape from shading perception.

  PublisherOA PDFDOI

• Individuation of 3D Perceptual Units from Neurogeometry of Binocular Cells

  SIAM Journal on Imaging Sciences · 2025-12-04

  articleOpen accessSenior author

  International audience

  PublisherOA PDFDOI

• Population encoding of stimulus features along the visual hierarchy

  Proceedings of the National Academy of Sciences · 2024-01-16 · 13 citations

  articleOpen accessSenior authorCorresponding

  The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.

  PublisherOA PDFDOI

• Functional organization and natural scene responses across mouse visual cortical areas revealed with encoding manifolds

  bioRxiv (Cold Spring Harbor Laboratory) · 2024-10-26 · 1 citations

  preprintOpen accessSenior author

  Abstract A challenge in sensory neuroscience is understanding how populations of neurons operate in concert to represent diverse stimuli. To meet this challenge, we have created “encoding manifolds” that reveal the overall responses of brain areas to diverse stimuli and organize individual neurons according to their selectivity and response dynamics. Here we use encoding manifolds to compare the population-level encoding of primary visual cortex (VISp) with that of five higher visual areas (VISam, VISal, VISpm, VISlm, and VISrl), using data from the Allen Institute Visual Coding–Neuropixels dataset from the mouse. We show that the topology of the encoding manifold for VISp and for higher visual areas is continuous, with smooth coordinates along which stimulus selectivity and response dynamics are organized with layer and cell-type specificity. Surprisingly, the manifolds revealed novel relationships between how natural scenes are encoded relative to static gratings—a relationship conserved across visual areas. Namely, neurons preferring natural scenes preferred either low or high spatial frequency gratings, but not intermediate ones. Analyzing responses by cortical layer reveals a preference for gratings concentrated in layer 6, whereas preferences for natural scenes tended to be higher in layers 2/3 and 4. The results demonstrate how machine learning approaches can be used to organize and visualize the structure of sensory coding, thereby revealing novel relationships within and across brain areas and sensory stimuli. Significance Statement Manifolds have become commonplace for analyzing and visualizing neural responses. However, prior work has focused on building manifolds that organize diverse stimuli in neural response coordinates. Here, we demonstrate the utility of an alternative approach: building manifolds to represent neurons in stimulus/response coordinates, which we term ‘encoding manifolds.’ This approach has several advantages, such as being able to directly visualize and compare how different brain areas encode diverse stimulus ensembles. This approach reveals novel relationships between layer-specific responses and the encoding of natural versus artificial stimuli.

  PublisherDOI

• Individuation of 3D perceptual units from neurogeometry of binocular cells

  arXiv (Cornell University) · 2024-10-03

  preprintOpen accessSenior author

  We model the functional architecture of the early stages of three-dimensional vision by extending the neurogeometric sub-Riemannian model for stereo-vision introduced in \cite{BCSZ23}. A new framework for correspondence is introduced that integrates a neural-based algorithm to achieve stereo correspondence locally while, simultaneously, organizing the corresponding points into global perceptual units. The result is an effective scene segmentation. We achieve this using harmonic analysis on the sub-Riemannian structure and show, in a comparison against Riemannian distance, that the sub-Riemannian metric is central to the solution.

  PublisherOA PDFDOI

• Learning dynamic representations of the functional connectome in neurobiological networks

  PubMed · 2024-02-21

  preprintOpen accessSenior author

  The static synaptic connectivity of neuronal circuits stands in direct contrast to the dynamics of their function. As in changing community interactions, different neurons can participate actively in various combinations to effect behaviors at different times. We introduce an unsupervised approach to learn the dynamic affinities between neurons in live, behaving animals, and to reveal which communities form among neurons at different times. The inference occurs in two major steps. First, pairwise non-linear affinities between neuronal traces from brain-wide calcium activity are organized by non-negative tensor factorization (NTF). Each factor specifies which groups of neurons are most likely interacting for an inferred interval in time, and for which animals. Finally, a generative model that allows for weighted community detection is applied to the functional motifs produced by NTF to reveal a dynamic functional connectome. Since time codes the different experimental variables (e.g., application of chemical stimuli), this provides an atlas of neural motifs active during separate stages of an experiment (e.g., stimulus application or spontaneous behaviors). Results from our analysis are experimentally validated, confirming that our method is able to robustly predict causal interactions between neurons to generate behavior.

  PublisherOA PDFDOI

• A separability-based approach to quantifying generalization: which layer is best?

  arXiv (Cornell University) · 2024-05-02

  preprintOpen accessSenior author

  Generalization to unseen data remains poorly understood for deep learning classification and foundation models, especially in the open set scenario. How can one assess the ability of networks to adapt to new or extended versions of their input space in the spirit of few-shot learning, out-of-distribution generalization, domain adaptation, and category discovery? Which layers of a network are likely to generalize best? We provide a new method for evaluating the capacity of networks to represent a sampled domain, regardless of whether the network has been trained on all classes in that domain. Our approach is the following: after fine-tuning state-of-the-art pre-trained models for visual classification on a particular domain, we assess their performance on data from related but distinct variations in that domain. Generalization power is quantified as a function of the latent embeddings of unseen data from intermediate layers for both unsupervised and supervised settings. Working throughout all stages of the network, we find that (i) high classification accuracy does not imply high generalizability; and (ii) deeper layers in a model do not always generalize the best, which has implications for pruning. Since the trends observed across datasets are largely consistent, we conclude that our approach reveals (a function of) the intrinsic capacity of the different layers of a model to generalize. Our code is available at https://github.com/dyballa/generalization

  PublisherOA PDFDOI

• Orientation fields predict perception of 3D shape from shading

  2024-06-27 · 1 citations

  preprintOpen access

  How the brain recovers the 3D structure of surfaces and objects from 2D retinal images remains mysterious. Shading patterns provide one of the most powerful—yet least understood—visual depth cues. Most theories assume the brain infers surface normals from luminance values. However, this seems unlikely as visual neurons are broadly insensitive to luminance. To identify alternative cues, we measured responses of model orientation-selective cell populations to images of shaded objects. We found a surprising statistical relationship between image orientations and surface curvatures, suggesting a novel way to estimate shape from shading. We find that the orientation-based cues not only predict striking novel illusions of shape perception when lighting varies, but also the impressive robustness of shape perception when large image modifications are introduced to directly pit luminance and image orientation cues against one another. The findings resolve the longstanding question of which image measurements drive shape from shading perception.

  PublisherOA PDFDOI

• Good continuation in 3D: the neurogeometry of stereo vision

  Frontiers in Computer Science · 2024-01-08 · 4 citations

  preprintOpen accessSenior author

  Classical good continuation for image curves is based on 2 D position and orientation. It is supported by the columnar organization of cortex, by psychophysical experiments, and by rich models of (differential) geometry. Here, we extend good continuation to stereo by introducing a neurogeometric model to abstract cortical organization. Our model clarifies which aspects of the projected scene geometry are relevant to neural connections. The model utilizes parameterizations that integrate spatial and orientation disparities, and provides insight into the psychophysics of stereo by yielding a well-defined 3 D association field. In sum, the model illustrates how good continuation in the (3D) world generalizes good continuation in the (2D) plane.

  PublisherOA PDFDOI

Recent grants

• US-German Collaboration: Towards a Neural Theory of 3D Shape Perception

  NSF · $460k · 2011–2015

• CRCNS Research Proposal: Collaborative Research: New Dimensions of Visual Cortical Organization

  NSF · $625k · 2018–2024

Frequent coauthors

• Benjamin Kunsberg

  39 shared

• Allen Tannenbaum

  33 shared

• Kaleem Siddiqi

  McGill University

  33 shared

• Benjamin B. Kimia

  24 shared

• Ohad Ben‐Shahar

  22 shared

• Lee Iverson

  19 shared

• Daniel Holtmann-Rice

  16 shared

• Matthew Lawlor

  Yale University

  15 shared

Labs

• Zucker Computational Vision GroupPI

  description of your site