852 research outputs found
Running Probabilistic Programs Backwards
Many probabilistic programming languages allow programs to be run under
constraints in order to carry out Bayesian inference. Running programs under
constraints could enable other uses such as rare event simulation and
probabilistic verification---except that all such probabilistic languages are
necessarily limited because they are defined or implemented in terms of an
impoverished theory of probability. Measure-theoretic probability provides a
more general foundation, but its generality makes finding computational content
difficult.
We develop a measure-theoretic semantics for a first-order probabilistic
language with recursion, which interprets programs as functions that compute
preimages. Preimage functions are generally uncomputable, so we derive an
abstract semantics. We implement the abstract semantics and use the
implementation to carry out Bayesian inference, stochastic ray tracing (a rare
event simulation), and probabilistic verification of floating-point error
bounds.Comment: 26 pages, ESOP 2015 (to appear
Exact Recursive Probabilistic Programming
Recursive calls over recursive data are widely useful for generating
probability distributions, and probabilistic programming allows computations
over these distributions to be expressed in a modular and intuitive way. Exact
inference is also useful, but unfortunately, existing probabilistic programming
languages do not perform exact inference on recursive calls over recursive
data, forcing programmers to code many applications manually. We introduce a
probabilistic language in which a wide variety of recursion can be expressed
naturally, and inference carried out exactly. For instance, probabilistic
pushdown automata and their generalizations are easy to express, and
polynomial-time parsing algorithms for them are derived automatically. We
eliminate recursive data types using program transformations related to
defunctionalization and refunctionalization. These transformations are assured
correct by a linear type system, and a successful choice of transformations, if
there is one, is guaranteed to be found by a greedy algorithm
Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages
We propose an efficient nonparametric strategy for learning a message
operator in expectation propagation (EP), which takes as input the set of
incoming messages to a factor node, and produces an outgoing message as output.
This learned operator replaces the multivariate integral required in classical
EP, which may not have an analytic expression. We use kernel-based regression,
which is trained on a set of probability distributions representing the
incoming messages, and the associated outgoing messages. The kernel approach
has two main advantages: first, it is fast, as it is implemented using a novel
two-layer random feature representation of the input message distributions;
second, it has principled uncertainty estimates, and can be cheaply updated
online, meaning it can request and incorporate new training data when it
encounters inputs on which it is uncertain. In experiments, our approach is
able to solve learning problems where a single message operator is required for
multiple, substantially different data sets (logistic regression for a variety
of classification problems), where it is essential to accurately assess
uncertainty and to efficiently and robustly update the message operator.Comment: accepted to UAI 2015. Correct typos. Add more content to the
appendix. Main results unchange
Automatic differentiation in machine learning: a survey
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD) is a technique for calculating derivatives of numeric functions expressed as computer programs efficiently and accurately, used in fields such as computational fluid dynamics, nuclear engineering, and atmospheric sciences. Despite its advantages and use in other fields, machine learning practitioners have been little influenced by AD and make scant use of available tools. We survey the intersection of AD and machine learning, cover applications where AD has the potential to make a big impact, and report on some recent developments in the adoption of this technique. We aim to dispel some misconceptions that we contend have impeded the use of AD within the machine learning community
Recommended from our members
Mathematical Logic: Proof theory, Constructive Mathematics
The workshop “Mathematical Logic: Proof Theory, Constructive Mathematics” was centered around proof-theoretic aspects of current mathematics, constructive mathematics and logical aspects of computational complexit
Automatic differentiation in machine learning: a survey
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD) is a technique for calculating derivatives of numeric functions expressed as computer programs efficiently and accurately, used in fields such as computational fluid dynamics, nuclear engineering, and atmospheric sciences. Despite its advantages and use in other fields, machine learning practitioners have been little influenced by AD and make scant use of available tools. We survey the intersection of AD and machine learning, cover applications where AD has the potential to make a big impact, and report on some recent developments in the adoption of this technique. We aim to dispel some misconceptions that we contend have impeded the use of AD within the machine learning community
- …