11 research outputs found
SPPL: Probabilistic Programming with Fast Exact Symbolic Inference
We present the Sum-Product Probabilistic Language (SPPL), a new probabilistic
programming language that automatically delivers exact solutions to a broad
range of probabilistic inference queries. SPPL translates probabilistic
programs into sum-product expressions, a new symbolic representation and
associated semantic domain that extends standard sum-product networks to
support mixed-type distributions, numeric transformations, logical formulas,
and pointwise and set-valued constraints. We formalize SPPL via a novel
translation strategy from probabilistic programs to sum-product expressions and
give sound exact algorithms for conditioning on and computing probabilities of
events. SPPL imposes a collection of restrictions on probabilistic programs to
ensure they can be translated into sum-product expressions, which allow the
system to leverage new techniques for improving the scalability of translation
and inference by automatically exploiting probabilistic structure. We implement
a prototype of SPPL with a modular architecture and evaluate it on benchmarks
the system targets, showing that it obtains up to 3500x speedups over
state-of-the-art symbolic systems on tasks such as verifying the fairness of
decision tree classifiers, smoothing hidden Markov models, conditioning
transformed random variables, and computing rare event probabilities
Optimal Approximate Sampling from Discrete Probability Distributions
This paper addresses a fundamental problem in random variate generation:
given access to a random source that emits a stream of independent fair bits,
what is the most accurate and entropy-efficient algorithm for sampling from a
discrete probability distribution , where the probabilities
of the output distribution of the sampling
algorithm must be specified using at most bits of precision? We present a
theoretical framework for formulating this problem and provide new techniques
for finding sampling algorithms that are optimal both statistically (in the
sense of sampling accuracy) and information-theoretically (in the sense of
entropy consumption). We leverage these results to build a system that, for a
broad family of measures of statistical accuracy, delivers a sampling algorithm
whose expected entropy usage is minimal among those that induce the same
distribution (i.e., is "entropy-optimal") and whose output distribution
is a closest approximation to the target
distribution among all entropy-optimal sampling algorithms
that operate within the specified -bit precision. This optimal approximate
sampler is also a closer approximation than any (possibly entropy-suboptimal)
sampler that consumes a bounded amount of entropy with the specified precision,
a class which includes floating-point implementations of inversion sampling and
related methods found in many software libraries. We evaluate the accuracy,
entropy consumption, precision requirements, and wall-clock runtime of our
optimal approximate sampling algorithms on a broad set of distributions,
demonstrating the ways that they are superior to existing approximate samplers
and establishing that they often consume significantly fewer resources than are
needed by exact samplers
From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought
How does language inform our downstream thinking? In particular, how do
humans make meaning from language -- and how can we leverage a theory of
linguistic meaning to build machines that think in more human-like ways? In
this paper, we propose \textit{rational meaning construction}, a computational
framework for language-informed thinking that combines neural models of
language with probabilistic models for rational inference. We frame linguistic
meaning as a context-sensitive mapping from natural language into a
\textit{probabilistic language of thought} (PLoT) -- a general-purpose symbolic
substrate for probabilistic, generative world modeling. Our architecture
integrates two powerful computational tools that have not previously come
together: we model thinking with \textit{probabilistic programs}, an expressive
representation for flexible commonsense reasoning; and we model meaning
construction with \textit{large language models} (LLMs), which support
broad-coverage translation from natural language utterances to code expressions
in a probabilistic programming language. We illustrate our framework in action
through examples covering four core domains from cognitive science:
probabilistic reasoning, logical and relational reasoning, visual and physical
reasoning, and social reasoning about agents and their plans. In each, we show
that LLMs can generate context-sensitive translations that capture
pragmatically-appropriate linguistic meanings, while Bayesian inference with
the generated programs supports coherent and robust commonsense reasoning. We
extend our framework to integrate cognitively-motivated symbolic modules to
provide a unified commonsense thinking interface from language. Finally, we
explore how language can drive the construction of world models themselves
PROBABILISTIC MODEL DISCOVERY RELATIONAL LEARNING AND SCALABLE INFERENCE
Department of Computer Science and EngineeringThis thesis studies interesting problems in compositionality for machine learning models under some settings including relational learning, scalability and deep models. Compositionality is the terminology describing the process of building small objects to complex ones. Bringing this concept into machine learning is important because it appears in many aspects from infinitesimal atomic to planetary structures. In this thesis, machine learning models center around Gaussian process of which covariance function is compositionally constructed. The proposed approach builds methods that can explore compositional model space automatically and efficiently as well as strives to address the interpretability for obtained models.
The aforementioned problems are both important and challenging. Considering multivariate or relational learning is de facto in time series analysis for many domains. However, the existing methods of compositional learning are inapplicable to extend to such a setting since the explosion in model space makes it infeasible to use. Learning compositional structures is already a time-consuming task. Although there are existing approximation methods, they do not work well for compositional covariances. This makes it even harder to propose a scalable approach without sacrificing model performances. Finally, analyzing hierarchical deep Gaussian processes is notoriously difficult especially when incorporating different covariance functions. Previous work focuses on a single case of covariance function and is difficult to generalize for many other cases.
The goal of this thesis is to propose solutions to the given problems. The first contribution of this thesis is a general framework for modeling multiple time series which provides descriptive relations between time series. Second, this thesis presents efficient probabilistic approaches to address the model search problem which previously is done by exhaustive enumerating evaluation. Furthermore, a scalable inference for Gaussian process is proposed, providing accurate approximation with guarantees of error bounds. Last but not least, to address the existing issues in deep Gaussian process, this thesis presents a unified theoretical framework to explain the pathology in deep Gasssian processes with better error bounds for various kernels compared to existing work and rates of convergence.ope
Computer Aided Verification
This open access two-volume set LNCS 11561 and 11562 constitutes the refereed proceedings of the 31st International Conference on Computer Aided Verification, CAV 2019, held in New York City, USA, in July 2019. The 52 full papers presented together with 13 tool papers and 2 case studies, were carefully reviewed and selected from 258 submissions. The papers were organized in the following topical sections: Part I: automata and timed systems; security and hyperproperties; synthesis; model checking; cyber-physical systems and machine learning; probabilistic systems, runtime techniques; dynamical, hybrid, and reactive systems; Part II: logics, decision procedures; and solvers; numerical programs; verification; distributed systems and networks; verification and invariants; and concurrency
Bayesian synthesis of probabilistic programs for automatic data modeling
We present new techniques for automatically constructing probabilistic programs for data analysis, interpretation, and prediction. These techniques work with probabilistic domain-specific data modeling languages that capture key properties of a broad class of data generating processes, using Bayesian inference to synthesize probabilistic programs in these modeling languages given observed data. We provide a precise formulation of Bayesian synthesis for automatic data modeling that identifies sufficient conditions for the resulting synthesis procedure to be sound. We also derive a general class of synthesis algorithms for domain-specific languages specified by probabilistic context-free grammars and establish the soundness of our approach for these languages. We apply the techniques to automatically synthesize probabilistic programs for time series data and multivariate tabular data. We show how to analyze the structure of the synthesized programs to compute, for key qualitative properties of interest, the probability that the underlying data generating process exhibits each of these properties. Second, we translate probabilistic programs in the domain-specific language into probabilistic programs in Venture, a general-purpose probabilistic programming system. The translated Venture programs are then executed to obtain predictions of new time series data and new multivariate data records. Experimental results show that our techniques can accurately infer qualitative structure in multiple real-world data sets and outperform standard data analysis methods in forecasting and predicting new data.</jats:p