"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

2026-04-13 · 1 citations

Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal care items, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues—such as blur, misframing, and rotation—affect the accuracy of VLM-generated captions and whether the resulting captions meet BLV people’s information needs. Based on a survey of 86 BLV participants, we develop an annotated dataset of 1,859 product images from BLV people to systematically evaluate how image quality issues affect VLM-generated captions. While the best VLM achieves 98% accuracy on images with no quality issues, accuracy drops to 75% overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people’s experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.

A Framework for Variational Inference and Data Assimilation of Soil Biogeochemical Models Using Normalizing Flows

Journal of Advances in Modeling Earth Systems · 2025-08-01

Abstract Soil biogeochemical models (SBMs) represent soil variables and their responses to global change. Data assimilation approaches help determine whether SBMs accurately represent soil processes consistent with soil pool and flux measurements. Bayesian inference is commonly used in data assimilation procedures that estimate posterior parameter distributions with Markov chain Monte Carlo (MCMC) methods. The ability to account for data and parameter uncertainty is a strength of MCMC inference, but the computational inefficiency of MCMC methods remains a barrier to their wider application, especially with large data sets. Given the limitations of MCMC approaches, we developed an alternative variational inference framework that uses a method called normalizing flows from the field of machine learning. Normalizing flows rely on deep learning to map probability distributions and approximate SBMs that have been discretized into state space models. As a test of our method, we fit approximated SBMs to synthetic data sourced from known data‐generating processes to identify discrepancies between the inference results and true parameter values. Our approach compares favorably with established MCMC methods and could be a viable alternative for SBM data assimilation that reduces computational time and resource needs. However, our method has some limitations, including challenges assimilating data with irregular measurement intervals, underestimation of posterior parameter uncertainty, and limited goodness‐of‐fit metrics for comparison to MCMC inference methods. Many of these limitations could be overcome with additional algorithm development based on the approaches we report here.

Learning to be Smooth: An End-to-End Differentiable Particle Smoother

ArXiv.org · 2025-02-14 · 1 citations

For challenging state estimation problems arising in domains like vision and robotics, particle-based representations attractively enable temporal reasoning about multiple posterior modes. Particle smoothers offer the potential for more accurate offline data analysis by propagating information both forward and backward in time, but have classically required human-engineered dynamics and observation models. Extending recent advances in discriminative training of particle filters, we develop a framework for low-variance propagation of gradients across long time sequences when training particle smoothers. Our "two-filter'' smoother integrates particle streams that are propagated forward and backward in time, while incorporating stratification and importance weights in the resampling step to provide low-variance gradient estimates for neural network dynamics and observation models. The resulting mixture density particle smoother is substantially more accurate than state-of-the-art particle filters, as well as search-based baselines, for city-scale global vehicle localization from real-world videos and maps.

Bayesian temporal biclustering with applications to multi-subject neuroscience studies

arXiv (Cornell University) · 2024-06-24

We consider the problem of analyzing multivariate time series collected on multiple subjects, with the goal of identifying groups of subjects exhibiting similar trends in their recorded measurements over time as well as time-varying groups of associated measurements. To this end, we propose a Bayesian model for temporal biclustering featuring nested partitions, where a time-invariant partition of subjects induces a time-varying partition of measurements. Our approach allows for data-driven determination of the number of subject and measurement clusters as well as estimation of the number and location of changepoints in measurement partitions. To efficiently perform model fitting and posterior estimation with Markov Chain Monte Carlo, we derive a blocked update of measurements' cluster-assignment sequences. We illustrate the performance of our model in two applications to functional magnetic resonance imaging data and to an electroencephalogram dataset. The results indicate that the proposed model can combine information from potentially many subjects to discover a set of interpretable, dynamic patterns. Experiments on simulated data compare the estimation performance of the proposed model against ground-truth values and other statistical methods, showing that it performs well at identifying ground-truth subject and measurement clusters even when no subject or time dependence is present.

Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes

arXiv (Cornell University) · 2024 · 1 citations

• Computer Science
• Artificial Intelligence
• Computer Science

Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods. Such generative models may be inaccurate or unavailable for high-dimensional observations like images. We instead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders. While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities. Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness. On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, while also showing much greater stability across multiple training runs.

Learning to be Smooth: An End-to-End Differentiable Particle Smoother

2024-01-01

Predicting Patient Outcomes from Time Series with Missing Data Via a Semi-Supervised Hidden Markov Model

SSRN Electronic Journal · 2024-01-01

VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference

arXiv (Cornell University) · 2024-11-28 · 1 citations

Diffusion probabilistic models learn to remove noise added during training, generating novel data (e.g., images) from Gaussian noise through sequential denoising. However, conditioning the generative process on corrupted or masked images is challenging. While various methods have been proposed for inpainting masked images with diffusion priors, they often fail to produce samples from the true conditional distribution, especially for large masked regions. Many baselines also cannot be applied to latent diffusion models which generate high-quality images with much lower computational cost. We propose a hierarchical variational inference algorithm that optimizes a non-Gaussian Markov approximation of the true diffusion posterior. Our VIPaint method outperforms existing approaches to inpainting, producing diverse high-quality imputations even for state-of-the-art text-conditioned latent diffusion models, and is also effective for other inverse problems like deblurring and superresolution.

Unbiased Learning of Deep Generative Models with Structured Discrete Representations

arXiv (Cornell University) · 2023

• Computer Science
• Artificial Intelligence
• Computer Science

By composing graphical models with deep learning architectures, we learn generative models with the strengths of both frameworks. The structured variational autoencoder (SVAE) inherits structure and interpretability from graphical models, and flexible likelihoods for high-dimensional data from deep learning, but poses substantial optimization challenges. We propose novel algorithms for learning SVAEs, and are the first to demonstrate the SVAE's ability to handle multimodal uncertainty when data is missing by incorporating discrete latent variables. Our memory-efficient implicit differentiation scheme makes the SVAE tractable to learn via gradient descent, while demonstrating robustness to incomplete optimization. To more rapidly learn accurate graphical model parameters, we derive a method for computing natural gradients without manual derivations, which avoids biases found in prior work. These optimization innovations enable the first comparisons of the SVAE to state-of-the-art time series models, where the SVAE performs competitively while learning interpretable and structured discrete data representations.

Unbiased learning of deep generative models with structured discrete representations

2023-01-01