About

Joyce Chai is a Professor in the Department of Electrical Engineering and Computer Science at the University of Michigan. She holds a Ph.D. in Computer Science from Duke University. Her research interests span natural language processing, embodied AI, situated dialogue agents, multimodal large language models, and human-AI collaboration. Her current work explores the intersection between language, perception, and action to enable situated communication with embodied agents. She is fascinated by how experience with the physical world and social interaction shape language learning and use, and she is dedicated to developing sensorimotor-grounded, pragmatically-rich, and cognitively-motivated human language technology. Professor Chai is a member of the Michigan AI Lab and directs the Situated Language and Embodied Dialogue (SLED) research group. She is also affiliated with the Michigan Robotics Institute and served as the Associate Director of the Michigan Institute for Data and AI in Society (MIDAS) from 2023 to 2025. Her research emphasizes grounded language processing to facilitate communication with robots and embodied AI agents. She has contributed to the field through her leadership roles, including serving on the executive board of NAACL and as Program Co-Chair for conferences such as ACL 2020. Recognized for her impactful work, she is a recipient of the NSF Career Award and has received multiple paper awards, including the Best Long Paper Award at ACL 2010 and Outstanding Paper Awards at EMNLP 2021 and ACL 2023. She is a Fellow of ACL.

Research topics

• Computer Science
• Natural Language Processing
• Artificial Intelligence
• Psychology
• Cognitive science
• History
• Human–computer interaction
• Linguistics

Selected publications

• Fast Spatial Memory with Elastic Test-Time Training

  arXiv (Cornell University) · 2026-04-08

  articleOpen access

  Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.

  PublisherOA PDF

• Fast Spatial Memory with Elastic Test-Time Training

  arXiv (Cornell University) · 2026-04-08

  preprintOpen access

  Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.

  PublisherDOI

• Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass

  ArXiv.org · 2025-01-23

  preprintOpen access

  Multi-view 3D reconstruction remains a core challenge in computer vision, particularly in applications requiring accurate and scalable representations across diverse perspectives. Current leading methods such as DUSt3R employ a fundamentally pairwise approach, processing images in pairs and necessitating costly global alignment procedures to reconstruct from multiple views. In this work, we propose Fast 3D Reconstruction (Fast3R), a novel multi-view generalization to DUSt3R that achieves efficient and scalable 3D reconstruction by processing many views in parallel. Fast3R's Transformer-based architecture forwards N images in a single forward pass, bypassing the need for iterative alignment. Through extensive experiments on camera pose estimation and 3D reconstruction, Fast3R demonstrates state-of-the-art performance, with significant improvements in inference speed and reduced error accumulation. These results establish Fast3R as a robust alternative for multi-view applications, offering enhanced scalability without compromising reconstruction accuracy.

  PublisherOA PDFDOI

• SAB3R: Semantic-Augmented Backbone in 3D Reconstruction

  ArXiv.org · 2025-06-02

  preprintOpen access

  We introduce a new task, Map and Locate, which unifies the traditionally distinct objectives of open-vocabulary segmentation - detecting and segmenting object instances based on natural language queries - and 3D reconstruction, the process of estimating a scene's 3D structure from visual inputs. Specifically, Map and Locate involves generating a point cloud from an unposed video and segmenting object instances based on open-vocabulary queries. This task serves as a critical step toward real-world embodied AI applications and introduces a practical task that bridges reconstruction, recognition and reorganization. To tackle this task, we introduce a simple yet effective baseline, which we denote as SAB3R. Our approach builds upon MASt3R, a recent breakthrough in 3D computer vision, and incorporates a lightweight distillation strategy. This method transfers dense, per-pixel semantic features from 2D vision backbones (eg, CLIP and DINOv2) to enhance MASt3R's capabilities. Without introducing any auxiliary frozen networks, our model generates per-pixel semantic features and constructs cohesive point maps in a single forward pass. Compared to separately deploying MASt3R and CLIP, our unified model, SAB3R, achieves superior performance on the Map and Locate benchmark. Furthermore, we evaluate SAB3R on both 2D semantic segmentation and 3D tasks to comprehensively validate its effectiveness.

  PublisherOA PDFDOI

• Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation

  ArXiv.org · 2025-04-22

  preprintOpen accessSenior author

  Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative communication (Grice, 1975). However, current evaluations of vision-language models (VLMs) often overlook the pragmatic dimension, reducing REG to a region-based captioning task and neglecting Gricean maxims. In this work, we revisit REG from a pragmatic perspective, introducing a new dataset (RefOI) of 1.5k images annotated with both written and spoken referring expressions. Through a systematic evaluation of state-of-the-art VLMs, we identify three key failures of pragmatic competence: (1) failure to uniquely identify the referent, (2) inclusion of excessive or irrelevant information, and (3) misalignment with human pragmatic preference, such as the underuse of minimal spatial cues. We also show that standard automatic evaluations fail to capture these pragmatic violations, reinforcing superficial cues rather than genuine referential success. Our findings call for a renewed focus on pragmatically informed models and evaluation frameworks that align with real human communication.

  PublisherOA PDFDOI

• Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models

  ArXiv.org · 2025-05-16

  preprintOpen accessSenior author

  Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also precisely timed. We identify two core capabilities necessary for such settings -- $\textit{perceptual updating}$ and $\textit{contingency awareness}$ -- and propose a new benchmark task, $\textbf{Temporally-Grounded Language Generation (TGLG)}$, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, $\textbf{TRACE}$, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present $\textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}$, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest -- highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Code and data available $\href{https://github.com/yukw777/tglg}{here}$.

  PublisherOA PDFDOI

• Spoken language interaction with robots: Recommendations for future research

  UNC Libraries · 2025-04-18

  articleOpen access
  PublisherDOI

• Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations

  2025-01-01 · 2 citations

  articleOpen accessSenior author

  Ziqiao Ma, Zekun Wang, Joyce Chai. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

  PublisherOA PDFDOI

• Proactive Assistant Dialogue Generation from Streaming Egocentric Videos

  ArXiv.org · 2025-06-06

  preprintOpen access

  Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/

  PublisherOA PDFDOI

• 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination

  2025-06-10 · 5 citations

  articleSenior author

  The integration of language and 3D perception is crucial for embodied agents and robots that comprehend and interact with the physical world. While large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, their adaptation to 3D environments (3D-LLMs) remains in its early stages. A primary challenge is a lack of large-scale datasets with dense grounding between language and 3D scenes. We introduce 3D-GRAND, a pioneering large-scale dataset comprising 40,087 household scenes paired with 6.2 million densely-grounded scene-language instructions. Our results show that instruction tuning with 3D-GRAND significantly enhances grounding capabilities and reduces hallucinations in 3D-LLMs. As part of our contributions, we propose a comprehensive benchmark 3D-POPE to systematically evaluate hallucination in 3D-LLMs, enabling fair comparisons of models. Our experiments highlight a scaling effect between dataset size and 3D-LLM performance, emphasizing the importance of large-scale 3D-text datasets for embodied AI research. Our results demonstrate early signals for effective sim-to-real transfer, indicating that models trained on large synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and 3D-POPE, we aim to equip the embodied AI community with resources and insights to lead to more reliable and better-grounded 3D-LLMs.

  PublisherDOI

Recent grants

• II-NEW: Towards an Infrastructure for Research on Multimodal Language Processing in Situated Human Robot Dialogue

  NSF · $229k · 2010–2014

• EAGER: Shared Gaze in Collaborative Referring

  NSF · $100k · 2010–2012

• RI: Small: Extending Verb Semantics with Causality towards Physical World

  NSF · $493k · 2016–2020

• NRI-Small: Contextually Grounded Collaborative Discourse for Mediating Shared Basis in Situated Human Robot Dialogue

  NSF · $981k · 2012–2017

• CAREER: Learning and Optimization for Robust Multimodal Interpretation in Conversation Systems

  NSF · $524k · 2004–2010

Frequent coauthors

• Ziqiao Ma

  24 shared

• Yichi Zhang

  20 shared

• Shane Storks

  20 shared

• Changsong Liu

  17 shared

• Lanbo She

  Microsoft (United States)

  15 shared

• Jiayi Pan

  14 shared

• Rong Jin

  14 shared

• Tyler Baldwin

  IBM (United States)

  13 shared

Awards & honors

• NSF Career Award
• Best Long Paper Award at ACL 2010
• Outstanding Paper Awards at EMNLP 2021
• Outstanding Paper Awards at ACL 2023
• Fellow of ACL