Scaling Gaussian Processes with Derivative Information Using Variational Inference

arXiv (Cornell University) · 2021 · 9 citations

• Computer Science
• Artificial Intelligence
• Machine Learning

Gaussian processes with derivative information are useful in many settings where derivative information is available, including numerous Bayesian optimization and regression tasks that arise in the natural sciences. Incorporating derivative observations, however, comes with a dominating $O(N^3D^3)$ computational cost when training on $N$ points in $D$ input dimensions. This is intractable for even moderately sized problems. While recent work has addressed this intractability in the low-$D$ setting, the high-$N$, high-$D$ setting is still unexplored and of great value, particularly as machine learning problems increasingly become high dimensional. In this paper, we introduce methods to achieve fully scalable Gaussian process regression with derivatives using variational inference. Analogous to the use of inducing values to sparsify the labels of a training set, we introduce the concept of inducing directional derivatives to sparsify the partial derivative information of a training set. This enables us to construct a variational posterior that incorporates derivative information but whose size depends neither on the full dataset size $N$ nor the full dimensionality $D$. We demonstrate the full scalability of our approach on a variety of tasks, ranging from a high dimensional stellarator fusion regression task to training graph convolutional neural networks on Pubmed using Bayesian optimization. Surprisingly, we find that our approach can improve regression performance even in settings where only label data is available.

Determining subpopulation methylation profiles from bisulfite sequencing data of heterogeneous samples using DXM

Nucleic Acids Research · 2021 · 15 citations

• Biology
• Genetics
• Computational biology

Epigenetic changes, such as aberrant DNA methylation, contribute to cancer clonal expansion and disease progression. However, identifying subpopulation-level changes in a heterogeneous sample remains challenging. Thus, we have developed a computational approach, DXM, to deconvolve the methylation profiles of major allelic subpopulations from the bisulfite sequencing data of a heterogeneous sample. DXM does not require prior knowledge of the number of subpopulations or types of cells to expect. We benchmark DXM's performance and demonstrate improvement over existing methods. We further experimentally validate DXM predicted allelic subpopulation-methylation profiles in four Diffuse Large B-Cell Lymphomas (DLBCLs). Lastly, as proof-of-concept, we apply DXM to a cohort of 31 DLBCLs and relate allelic subpopulation methylation profiles to relapse. We thus demonstrate that DXM can robustly find allelic subpopulation methylation profiles that may contribute to disease progression using bisulfite sequencing data of any heterogeneous sample.

Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization

arXiv (Cornell University) · 2020 · 11 citations

• Computer Science
• Mathematics
• Algorithm

Matrix square roots and their inverses arise frequently in machine learning, e.g., when sampling from high-dimensional Gaussians $\mathcal{N}(\mathbf 0, \mathbf K)$ or whitening a vector $\mathbf b$ against covariance matrix $\mathbf K$. While existing methods typically require $O(N^3)$ computation, we introduce a highly-efficient quadratic-time algorithm for computing $\mathbf K^{1/2} \mathbf b$, $\mathbf K^{-1/2} \mathbf b$, and their derivatives through matrix-vector multiplication (MVMs). Our method combines Krylov subspace methods with a rational approximation and typically achieves $4$ decimal places of accuracy with fewer than $100$ MVMs. Moreover, the backward pass requires little additional computation. We demonstrate our method's applicability on matrices as large as $50,\!000 \times 50,\!000$ - well beyond traditional methods - with little approximation error. Applying this increased scalability to variational Gaussian processes, Bayesian optimization, and Gibbs sampling results in more powerful models with higher accuracy.