About

Professor Frank Dellaert’s research focuses on large-scale inference for autonomous robot systems, on land, air, and in water. He pioneered the use of several probabilistic methods in both computer vision and robotics. With Dieter Fox and Sebastian Thrun, he introduced the Monte Carlo localization method for estimating and tracking the pose of robots, which is now a standard and popular tool in mobile robotics. More recently, he has investigated 3D reconstruction in large-scale environments by taking a graph-theoretic view and introduced factor graphs into the mainstream language of the robotics community. Professor Dellaert has published more than 200 technical articles, as well as several book chapters. He has received an NSF CAREER award and research grants from NSF, DARPA, ARL, ARO, Intel, Microsoft, Samsung, and others. During a leave from 2014 to 2018, he spent time at his alma mater in Belgium, worked as Chief Scientist at Skydio, and held positions at Facebook’s Reality Labs and Google AI, as well as CTO at Verdant Robotics.

Research topics

• Artificial Intelligence
• Computer Science
• Algorithm
• Mathematical optimization
• Mathematics

Selected publications

• CMC-Opt: Constraint Manifold with Corners for Inequality-Constrained Optimization

  arXiv (Cornell University) · 2026-05-20

  preprintOpen accessSenior author

  We introduce a manifold-based framework for addressing optimization problems with equality and inequality constraints found in robotics. Our approach transforms the original problem into an unconstrained optimization problem directly on the constrained state space. To achieve this, we introduce ``constraint manifolds with corners" to represent the state space satisfying mixed nonlinear equality and inequality constraints. We further extend manifold optimization algorithms to operate on this new topological structure. We demonstrate the power and robustness of our framework in the context of a large-scale kinodynamic planning problem, successfully generating dynamically feasible trajectories where standard methods fail.

  PublisherDOI

• CMC-Opt: Constraint Manifold with Corners for Inequality-Constrained Optimization

  ArXiv.org · 2026-05-20

  articleOpen accessSenior author

  We introduce a manifold-based framework for addressing optimization problems with equality and inequality constraints found in robotics. Our approach transforms the original problem into an unconstrained optimization problem directly on the constrained state space. To achieve this, we introduce ``constraint manifolds with corners" to represent the state space satisfying mixed nonlinear equality and inequality constraints. We further extend manifold optimization algorithms to operate on this new topological structure. We demonstrate the power and robustness of our framework in the context of a large-scale kinodynamic planning problem, successfully generating dynamically feasible trajectories where standard methods fail.

  PublisherOA PDF

• Smoothing Out the Edges: Continuous-Time Estimation with Gaussian Process Motion Priors on Factor Graphs

  arXiv (Cornell University) · 2026-05-09

  preprintOpen access

  Continuous-time state estimation is gaining in popularity due to its abilities to provide smooth solutions, handle asynchronous sensors, and interpolate between data points. While there are two main paradigms, parametric (e.g., temporal basis functions, splines) and nonparametric (Gaussian processes), the latter has seen less adoption despite its technical advantages and relative ease of implementation. In this article, we seek to rectify this situation by providing a new simplified explanation of GP continuous-time estimation rooted in the language of factor graphs, which have become the de facto estimation paradigm in much of robotics. To simplify onboarding, we also provide three working examples implemented in the popular GTSAM estimation framework.

  PublisherDOI

• Variable Elimination in Hybrid Factor Graphs for Discrete-Continuous Inference & Estimation

  arXiv (Cornell University) · 2026-01-02

  articleOpen accessSenior author

  Many problems in robotics involve both continuous and discrete components, and modeling them together for estimation tasks has been a long standing and difficult problem. Hybrid Factor Graphs give us a mathematical framework to model these types of problems, however existing approaches for solving them are based on approximations. In this work, we propose a new framework for hybrid factor graphs along with a novel variable elimination algorithm to produce a hybrid Bayes network, which can be used for exact Maximum A Posteriori estimation and marginalization over both sets of variables. Our approach first develops a novel hybrid Gaussian factor which can connect to both discrete and continuous variables, and a hybrid conditional which can represent multiple continuous hypotheses conditioned on the discrete variables. Using these representations, we derive the process of hybrid variable elimination under the Conditional Linear Gaussian scheme, giving us exact posteriors as a hybrid Bayes network. To bound the number of discrete hypotheses, we use a tree-structured representation of the factors coupled with a simple pruning and probabilistic assignment scheme, which allows for tractable inference. We demonstrate the applicability of our framework on a large scale SLAM dataset and a real world pose graph optimization problem, both with ambiguous measurements which require discrete choices to be made for the most likely measurements. Our demonstrated results showcase the accuracy, generality, and simplicity of our hybrid factor graph framework.

  PublisherOA PDF

• Variable Elimination in Hybrid Factor Graphs for Discrete-Continuous Inference & Estimation

  arXiv (Cornell University) · 2026-01-02

  preprintOpen accessSenior author

  Many problems in robotics involve both continuous and discrete components, and modeling them together for estimation tasks has been a long standing and difficult problem. Hybrid Factor Graphs give us a mathematical framework to model these types of problems, however existing approaches for solving them are based on approximations. In this work, we propose a new framework for hybrid factor graphs along with a novel variable elimination algorithm to produce a hybrid Bayes network, which can be used for exact Maximum A Posteriori estimation and marginalization over both sets of variables. Our approach first develops a novel hybrid Gaussian factor which can connect to both discrete and continuous variables, and a hybrid conditional which can represent multiple continuous hypotheses conditioned on the discrete variables. Using these representations, we derive the process of hybrid variable elimination under the Conditional Linear Gaussian scheme, giving us exact posteriors as a hybrid Bayes network. To bound the number of discrete hypotheses, we use a tree-structured representation of the factors coupled with a simple pruning and probabilistic assignment scheme, which allows for tractable inference. We demonstrate the applicability of our framework on a large scale SLAM dataset and a real world pose graph optimization problem, both with ambiguous measurements which require discrete choices to be made for the most likely measurements. Our demonstrated results showcase the accuracy, generality, and simplicity of our hybrid factor graph framework.

  PublisherDOI

• Smoothing Out the Edges: Continuous-Time Estimation with Gaussian Process Motion Priors on Factor Graphs

  ArXiv.org · 2026-05-09

  articleOpen access

  Continuous-time state estimation is gaining in popularity due to its abilities to provide smooth solutions, handle asynchronous sensors, and interpolate between data points. While there are two main paradigms, parametric (e.g., temporal basis functions, splines) and nonparametric (Gaussian processes), the latter has seen less adoption despite its technical advantages and relative ease of implementation. In this article, we seek to rectify this situation by providing a new simplified explanation of GP continuous-time estimation rooted in the language of factor graphs, which have become the de facto estimation paradigm in much of robotics. To simplify onboarding, we also provide three working examples implemented in the popular GTSAM estimation framework.

  PublisherOA PDF

• ObjectTrack: 6DoF Object Tracking Through UWB-IMU Fusion

  2025-09-15

  article

  This paper presents a UWB-IMU fusion approach to obtain location and orientation of an object in 6 degrees of freedom at the room level, without use of optical motion capture systems. When tested with different human movement patterns such as walking, running, jumping, and swirling on a wheeled chair, we obtain less than 10cm of 3D localization error and under 5° of orientation error at the 90<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> percentile. We expect our system, called ObjectTrack, to enable spatial audio and interaction for VR/AR applications, enable precision tracking of objects, and for localization of robotic motion systems. ObjectTrack significantly reduces the cost barrier by about 50× compared to popular motion capture systems.

  PublisherDOI

• Autonomous Vehicle Trajectory Planning for Minimum Position Uncertainty

  2025-04-28

  articleSenior author

  Position estimation accuracy often depends on the path that is traveled, particularly in the absence of measurements to globally referenced sources such as GPS. For autonomous vehicles operating in an open world environment, the vehicle trajectories may be planned to minimize the expected position estimation uncertainty. Our approach uses automatic differentiation of a scalar valued function of the expected final covariance with respect to control point locations that define the path with B-splines. A gradient descent or quasi-Newton method is then used to determine path parameters that optimize the objective. For a collaborative navigation scenario, this method is a significant improvement compared to a derivative-free approach.

  PublisherDOI

• SALVe: Semantic Alignment Verification for Floorplan Reconstruction from Sparse Panoramas

  arXiv (Cornell University) · 2024-06-27

  preprintOpen access

  We propose a new system for automatic 2D floorplan reconstruction that is enabled by SALVe, our novel pairwise learned alignment verifier. The inputs to our system are sparsely located 360$^\circ$ panoramas, whose semantic features (windows, doors, and openings) are inferred and used to hypothesize pairwise room adjacency or overlap. SALVe initializes a pose graph, which is subsequently optimized using GTSAM. Once the room poses are computed, room layouts are inferred using HorizonNet, and the floorplan is constructed by stitching the most confident layout boundaries. We validate our system qualitatively and quantitatively as well as through ablation studies, showing that it outperforms state-of-the-art SfM systems in completeness by over 200%, without sacrificing accuracy. Our results point to the significance of our work: poses of 81% of panoramas are localized in the first 2 connected components (CCs), and 89% in the first 3 CCs. Code and models are publicly available at https://github.com/zillow/salve.

  PublisherOA PDFDOI

• Generalizing Trajectory Retiming to Quadratic Objective Functions

  2024-05-13 · 1 citations

  article

  Trajectory retiming is the task of computing a feasible time parameterization to traverse a path. It is commonly used in the decoupled approach to trajectory optimization whereby a path is first found, then a retiming algorithm computes a speed profile that satisfies kino-dynamic and other constraints. While trajectory retiming is most often formulated with the minimum-time objective (i.e. traverse the path as fast as possible), it is not always the most desirable objective, particularly when we seek to balance multiple objectives or when bang-bang control is unsuitable. In this paper, we present a novel algorithm based on factor graph variable elimination that can solve for the global optimum of the retiming problem with quadratic objectives as well (e.g. minimize control effort or match a nominal speed by minimizing squared error), which may extend to arbitrary objectives with iteration. Our work extends prior works, which find only solutions on the boundary of the feasible region, while maintaining the same linear time complexity from a single forward-backward pass. We experimentally demonstrate that (1) we achieve better real-world robot performance by using quadratic objectives in place of the minimum-time objective, and (2) our implementation is comparable or faster than state-of-the-art retiming algorithms.

  PublisherDOI

Recent grants

• Unlocking the Urban Photographic Record Through 4D Scene Understanding and Modeling

  NSF · $206k · 2005–2010

• RI: Collaborative Research: Bion-Inspired Navigation

  NSF · $211k · 2007–2011

• CAREER: Markov Chain Monte Carlo Methods for Large Scale Correspondence Problems in Computer Vision and Robotics

  NSF · $422k · 2005–2011

Frequent coauthors

• Luca Carlone

  39 shared

• Michael Kaess

  32 shared

• Vadim Indelman

  23 shared

• Ananth Ranganathan

  20 shared

• Richard Roberts

  17 shared

• Tucker Balch

  16 shared

• Sebastian Thrun

  15 shared

• Gerry Chen

  Georgia Institute of Technology

  15 shared

Education

• Ph.D., Computer Science

  Carnegie Mellon University

  2001

Awards & honors

• NSF CAREER Award (2005)