About

Charless Fowlkes is a Professor at UC Irvine and a Chancellor's Fellow in the Department of Computer Science. His research focuses on computational vision, including understanding the information processing capabilities of the human visual system and developing machine vision systems. He is also interested in applying computer vision techniques to automate the analysis of biological data and in developing algorithmic tools for understanding morphology and spatial aspects of gene expression. Dr. Fowlkes earned his Ph.D. from UC Berkeley in 2005. His work encompasses areas such as artificial intelligence, machine learning, natural language processing, biomedical informatics and computational biology, and computer graphics and vision.

Research topics

• Artificial Intelligence
• Computer Science
• Machine Learning
• Natural Language Processing
• Mathematics
• Engineering
• Physics
• Optics
• Biology
• Botany
• Human–computer interaction
• Computer vision
• Programming language

Selected publications

• CriSp: Leveraging Tread Depth Maps for Enhanced Crime-Scene Shoeprint Matching

  arXiv (Cornell University) · 2024-04-25

  preprintOpen accessSenior author

  Shoeprints are a common type of evidence found at crime scenes and are used regularly in forensic investigations. However, existing methods cannot effectively employ deep learning techniques to match noisy and occluded crime-scene shoeprints to a shoe database due to a lack of training data. Moreover, all existing methods match crime-scene shoeprints to clean reference prints, yet our analysis shows matching to more informative tread depth maps yields better retrieval results. The matching task is further complicated by the necessity to identify similarities only in corresponding regions (heels, toes, etc) of prints and shoe treads. To overcome these challenges, we leverage shoe tread images from online retailers and utilize an off-the-shelf predictor to estimate depth maps and clean prints. Our method, named CriSp, matches crime-scene shoeprints to tread depth maps by training on this data. CriSp incorporates data augmentation to simulate crime-scene shoeprints, an encoder to learn spatially-aware features, and a masking module to ensure only visible regions of crime-scene prints affect retrieval results. To validate our approach, we introduce two validation sets by reprocessing existing datasets of crime-scene shoeprints and establish a benchmarking protocol for comparison. On this benchmark, CriSp significantly outperforms state-of-the-art methods in both automated shoeprint matching and image retrieval tailored to this task.

  PublisherOA PDFDOI

• Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions

  arXiv (Cornell University) · 2024-10-13

  preprintOpen access

  Recent advancements in Model-Based Reinforcement Learning (MBRL) have made it a powerful tool for visual control tasks. Despite improved data efficiency, it remains challenging to train MBRL agents with generalizable perception. Training in the presence of visual distractions is particularly difficult due to the high variation they introduce to representation learning. Building on DREAMER, a popular MBRL method, we propose a simple yet effective auxiliary task to facilitate representation learning in distracting environments. Under the assumption that task-relevant components of image observations are straightforward to identify with prior knowledge in a given task, we use a segmentation mask on image observations to only reconstruct task-relevant components. In doing so, we greatly reduce the complexity of representation learning by removing the need to encode task-irrelevant objects in the latent representation. Our method, Segmentation Dreamer (SD), can be used either with ground-truth masks easily accessible in simulation or by leveraging potentially imperfect segmentation foundation models. The latter is further improved by selectively applying the reconstruction loss to avoid providing misleading learning signals due to mask prediction errors. In modified DeepMind Control suite (DMC) and Meta-World tasks with added visual distractions, SD achieves significantly better sample efficiency and greater final performance than prior work. We find that SD is especially helpful in sparse reward tasks otherwise unsolvable by prior work, enabling the training of visually robust agents without the need for extensive reward engineering.

  PublisherOA PDFDOI

• CriSp: Leveraging Tread Depth Maps for Enhanced Crime-Scene Shoeprint Matching

  Lecture notes in computer science · 2024-11-20 · 1 citations

  book-chapterSenior author
  PublisherDOI

• Instance Tracking in 3D Scenes from Egocentric Videos

  2024-06-16 · 5 citations

  articleSenior author

  Egocentric sensors such as AR/VR devices capture human-object interactions and offer the potential to provide task-assistance by recalling 3D locations of objects of interest in the surrounding environment. This capability requires instance tracking in real-world 3D scenes from egocentric videos (IT3DEgo). We explore this problem by first introducing a new benchmark dataset, consisting of RGB and depth videos, per-frame camera pose, and instance-level annotations in both 2D camera and 3D world coordinates. We present an evaluation protocol which evaluates tracking performance in 3D coordinates with two settings for enrolling instances to track: (1) single-view online enrollment where an instance is specified on-the-fly based on the human wearer's interactions. and (2) multi-view pre-enrollment where images of an instance to be tracked are stored in memory ahead of time. To address IT3DEgo, we first repurpose methods from relevant areas, e.g., single object tracking (SOT) - running SOT methods to track instances in 2D frames and lifting them to 3D using camera pose and depth. We also present a simple method that leverages pre-trained segmentation and detection models to generate proposals from RGB frames and match proposals with enrolled instance images. Our experiments show that our method (with no finetuning) significantly outperforms SOT-based approaches in the egocentric setting. We conclude by arguing that the problem of egocentric instance tracking is made easier by leveraging camera pose and using a 3D allocentric (world) coordinate representation. Dataset and open-source code: https://github.com/IT3DEgo/IT3DEgo.

  PublisherDOI

• Guided Recommendation for Model Fine-Tuning

  2023-06-01 · 10 citations

  article

  Model selection is essential for reducing the search cost of the best pre-trained model over a large-scale model zoo for a downstream task. After analyzing recent hand-designed model selection criteria with 400+ ImageNet pre-trained models and 40 downstream tasks, we find that they can fail due to invalid assumptions and intrinsic limitations. The prior knowledge on model capacity and dataset also can not be easily integrated into the existing criteria. To address these issues, we propose to convert model selection as a recommendation problem and to learn from the past training history. Specifically, we characterize the meta information of datasets and models as features, and use their transfer learning performance as the guided score. With thousands of historical training jobs, a recommendation system can be learned to predict the model selection score given the features of the dataset and the model as input. Our approach enables integrating existing model selection scores as additional features and scales with more historical data. We evaluate the prediction accuracy with 22 pre-trained models over 40 downstream tasks. With extensive evaluations, we show that the learned approach can outperform prior hand-designed model selection methods significantly when relevant training history is available.

  PublisherDOI

• Geometric Pose Affordance: Monocular 3D Human Pose Estimation with Scene Constraints

  Lecture notes in computer science · 2023-01-01 · 14 citations

  book-chapterSenior author
  PublisherDOI

• Instance Tracking in 3D Scenes from Egocentric Videos

  arXiv (Cornell University) · 2023-12-07 · 1 citations

  preprintOpen accessSenior author

  Egocentric sensors such as AR/VR devices capture human-object interactions and offer the potential to provide task-assistance by recalling 3D locations of objects of interest in the surrounding environment. This capability requires instance tracking in real-world 3D scenes from egocentric videos (IT3DEgo). We explore this problem by first introducing a new benchmark dataset, consisting of RGB and depth videos, per-frame camera pose, and instance-level annotations in both 2D camera and 3D world coordinates. We present an evaluation protocol which evaluates tracking performance in 3D coordinates with two settings for enrolling instances to track: (1) single-view online enrollment where an instance is specified on-the-fly based on the human wearer's interactions. and (2) multi-view pre-enrollment where images of an instance to be tracked are stored in memory ahead of time. To address IT3DEgo, we first re-purpose methods from relevant areas, e.g., single object tracking (SOT) -- running SOT methods to track instances in 2D frames and lifting them to 3D using camera pose and depth. We also present a simple method that leverages pretrained segmentation and detection models to generate proposals from RGB frames and match proposals with enrolled instance images. Our experiments show that our method (with no finetuning) significantly outperforms SOT-based approaches in the egocentric setting. We conclude by arguing that the problem of egocentric instance tracking is made easier by leveraging camera pose and using a 3D allocentric (world) coordinate representation.

  PublisherOA PDFDOI

• Your representations are in the network: composable and parallel adaptation for large scale models

  arXiv (Cornell University) · 2023-03-07 · 2 citations

  preprintOpen access

  We propose InCA, a lightweight method for transfer learning that cross-attends to any activation layer of a pre-trained model. During training, InCA uses a single forward pass to extract multiple activations, which are passed to external cross-attention adapters, trained anew and combined or selected for downstream tasks. We show that, even when selecting a single top-scoring adapter, InCA achieves performance comparable to full fine-tuning, at a cost comparable to fine-tuning just the last layer. For example, with a cross-attention probe 1.3% the size of a pre-trained ViT-L/16 model, we achieve performance within 0.2% of the full fine-tuning paragon at a computational training cost of 51% of the baseline, on average across 11 downstream classification. Unlike other forms of efficient adaptation, InCA does not require backpropagating through the pre-trained model, thus leaving its execution unaltered at both training and inference. The versatility of InCA is best illustrated in fine-grained tasks, which may require accessing information absent in the last layer but accessible in intermediate layer activations. Since the backbone is fixed, InCA allows parallel ensembling as well as parallel execution of multiple tasks. InCA achieves state-of-the-art performance in the ImageNet-to-Sketch multi-task benchmark.

  PublisherOA PDFDOI

• Creating a Forensic Database of Shoeprints from Online Shoe-Tread Photos

  2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · 2023-01-01 · 5 citations

  articleSenior author

  Shoe-tread impressions are one of the most common types of evidence left at crime scenes. However, the utility of such evidence is limited by the lack of databases of footwear prints that cover the large and growing number of distinct shoe models. Moreover, the database is preferred to contain the 3D shape, or depth, of shoe-tread photos so as to allow for extracting shoeprints to match a query (crime-scene) print. We propose to address this gap by leveraging shoe-tread photos collected by online retailers. The core challenge is to predict depth maps for these photos. As they do not have ground-truth 3D shapes allowing for training depth predictors, we exploit synthetic data that does. We develop a method, termed ShoeRinsics, that learns to predict depth from fully supervised synthetic data and unsupervised retail image data. In particular, we find domain adaptation and intrinsic image decomposition techniques effectively mitigate the synthetic-real domain gap and yield significantly better depth predictions. To validate our method, we introduce 2 validation sets consisting of shoe-tread image and print pairs and define a benchmarking protocol to quantify the quality of predicted depth. On this benchmark, ShoeRinsics outperforms existing methods of depth prediction and synthetic-to-real domain adaptation.

  PublisherDOI

• GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding

  2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · 2023-01-01 · 11 citations

  articleSenior author

  Reference-guided image inpainting restores image pixels by leveraging the content from another single reference image. The primary challenge is how to precisely place the pixels from the reference image into the hole region. Therefore, understanding the 3D geometry that relates pixels between two views is a crucial step towards building a better model. Given the complexity of handling various types of reference images, we focus on the scenario where the images are captured by freely moving the same camera around. Compared to the previous work, we propose a principled approach that does not make heuristic assumptions about the planarity of the scene. We lever-age a monocular depth estimate and predict relative pose between cameras, then align the reference image to the target by a differentiable 3D reprojection and a joint optimization of relative pose and depth map scale and offset. Our approach achieves state-of-the-art performance on both RealEstate10K and MannequinChallenge dataset with large baselines, complex geometry and extreme camera motions. We experimentally verify our approach is also better at handling large holes.

  PublisherDOI

Recent grants

• Comprehensive Structural and Functional Mapping of the Mammalian Cardiac Nervous System

  NIH · $21.4M · 2016–2023

• Collaborative Research: Biological Shape Spaces, Transforming Shape into Knowledge

  NSF · $284k · 2010–2014

• Collaborative Research: ABI Innovation: Breaking through the taxonomic barrier of the fossil pollen record using bioimage informatics

  NSF · $254k · 2013–2017

• CAREER: Combinatorial Inference and Learning for Fusing Recognition and Perceptual Grouping

  NSF · $508k · 2013–2019

• RI: Small: Building Strong Geometric Priors for Total Scene Understanding

  NSF · $377k · 2016–2020

Frequent coauthors

• Shu Kong

  48 shared

• Jitendra Malik

  37 shared

• Deva Ramanan

  Carnegie Mellon University

  21 shared

• Angela H. DePace

  Harvard University

  21 shared

• Surangi W. Punyasena

  18 shared

• Julian Yarkony

  16 shared

• Meghan D. J. Bragdon

  Boston University

  15 shared

• Daeyun Shin

  University of California, Irvine

  13 shared