410 research outputs found
Policy Iteration for Factored MDPs
Many large MDPs can be represented compactly using a dynamic Bayesian
network. Although the structure of the value function does not retain the
structure of the process, recent work has shown that value functions in
factored MDPs can often be approximated well using a decomposed value function:
a linear combination of restricted basis functions, each of which refers
only to a small subset of variables. An approximate value function for a
particular policy can be computed using approximate dynamic programming, but
this approach (and others) can only produce an approximation relative to a
distance metric which is weighted by the stationary distribution of the current
policy. This type of weighted projection is ill-suited to policy improvement.
We present a new approach to value determination, that uses a simple
closed-form computation to directly compute a least-squares decomposed
approximation to the value function for any weights. We then use this
value determination algorithm as a subroutine in a policy iteration process. We
show that, under reasonable restrictions, the policies induced by a factored
value function are compactly represented, and can be manipulated efficiently in
a policy iteration process. We also present a method for computing error bounds
for decomposed value functions using a variable-elimination algorithm for
function optimization. The complexity of all of our algorithms depends on the
factorization of system dynamics and of the approximate value function.Comment: Appears in Proceedings of the Sixteenth Conference on Uncertainty in
Artificial Intelligence (UAI2000
Object-Oriented Bayesian Networks
Bayesian networks provide a modeling language and associated inference
algorithm for stochastic domains. They have been successfully applied in a
variety of medium-scale applications. However, when faced with a large complex
domain, the task of modeling using Bayesian networks begins to resemble the
task of programming using logical circuits. In this paper, we describe an
object-oriented Bayesian network (OOBN) language, which allows complex domains
to be described in terms of inter-related objects. We use a Bayesian network
fragment to describe the probabilistic relations between the attributes of an
object. These attributes can themselves be objects, providing a natural
framework for encoding part-of hierarchies. Classes are used to provide a
reusable probabilistic model which can be applied to multiple similar objects.
Classes also support inheritance of model fragments from a class to a subclass,
allowing the common aspects of related classes to be defined only once. Our
language has clear declarative semantics: an OOBN can be interpreted as a
stochastic functional program, so that it uniquely specifies a probabilistic
model. We provide an inference algorithm for OOBNs, and show that much of the
structural information encoded by an OOBN--particularly the encapsulation of
variables within an object and the reuse of model fragments in different
contexts--can also be used to speed up the inference process.Comment: Appears in Proceedings of the Thirteenth Conference on Uncertainty in
Artificial Intelligence (UAI1997
Tractable Inference for Complex Stochastic Processes
The monitoring and control of any dynamic system depends crucially on the
ability to reason about its current status and its future trajectory. In the
case of a stochastic system, these tasks typically involve the use of a belief
state- a probability distribution over the state of the process at a given
point in time. Unfortunately, the state spaces of complex processes are very
large, making an explicit representation of a belief state intractable. Even in
dynamic Bayesian networks (DBNs), where the process itself can be represented
compactly, the representation of the belief state is intractable. We
investigate the idea of maintaining a compact approximation to the true belief
state, and analyze the conditions under which the errors due to the
approximations taken over the lifetime of the process do not accumulate to make
our answers completely irrelevant. We show that the error in a belief state
contracts exponentially as the process evolves. Thus, even with multiple
approximations, the error in our process remains bounded indefinitely. We show
how the additional structure of a DBN can be used to design our approximation
scheme, improving its performance significantly. We demonstrate the
applicability of our ideas in the context of a monitoring task, showing that
orders of magnitude faster inference can be achieved with only a small
degradation in accuracy.Comment: Appears in Proceedings of the Fourteenth Conference on Uncertainty in
Artificial Intelligence (UAI1998
Probabilistic Models for Agents' Beliefs and Decisions
Many applications of intelligent systems require reasoning about the mental
states of agents in the domain. We may want to reason about an agent's beliefs,
including beliefs about other agents; we may also want to reason about an
agent's preferences, and how his beliefs and preferences relate to his
behavior. We define a probabilistic epistemic logic (PEL) in which belief
statements are given a formal semantics, and provide an algorithm for asserting
and querying PEL formulas in Bayesian networks. We then show how to reason
about an agent's behavior by modeling his decision process as an influence
diagram and assuming that he behaves rationally. PEL can then be used for
reasoning from an agent's observed actions to conclusions about other aspects
of the domain, including unobserved domain variables and the agent's mental
states.Comment: Appears in Proceedings of the Sixteenth Conference on Uncertainty in
Artificial Intelligence (UAI2000
Being Bayesian about Network Structure
In many domains, we are interested in analyzing the structure of the
underlying distribution, e.g., whether one variable is a direct parent of the
other. Bayesian model-selection attempts to find the MAP model and use its
structure to answer these questions. However, when the amount of available data
is modest, there might be many models that have non-negligible posterior. Thus,
we want compute the Bayesian posterior of a feature, i.e., the total posterior
probability of all models that contain it. In this paper, we propose a new
approach for this task. We first show how to efficiently compute a sum over the
exponential number of networks that are consistent with a fixed ordering over
network variables. This allows us to compute, for a given ordering, both the
marginal probability of the data and the posterior of a feature. We then use
this result as the basis for an algorithm that approximates the Bayesian
posterior of a feature. Our approach uses a Markov Chain Monte Carlo (MCMC)
method, but over orderings rather than over network structures. The space of
orderings is much smaller and more regular than the space of structures, and
has a smoother posterior `landscape'. We present empirical results on synthetic
and real-life datasets that compare our approach to full model averaging (when
possible), to MCMC over network structures, and to a non-Bayesian bootstrap
approach.Comment: Appears in Proceedings of the Sixteenth Conference on Uncertainty in
Artificial Intelligence (UAI2000
Utilities as Random Variables: Density Estimation and Structure Discovery
Decision theory does not traditionally include uncertainty over utility
functions. We argue that the a person's utility value for a given outcome can
be treated as we treat other domain attributes: as a random variable with a
density function over its possible values. We show that we can apply
statistical density estimation techniques to learn such a density function from
a database of partially elicited utility functions. In particular, we define a
Bayesian learning framework for this problem, assuming the distribution over
utilities is a mixture of Gaussians, where the mixture components represent
statistically coherent subpopulations. We can also extend our techniques to the
problem of discovering generalized additivity structure in the utility
functions in the population. We define a Bayesian model selection criterion for
utility function structure and a search procedure over structures. The
factorization of the utilities in the learned model, and the generalization
obtained from density estimation, allows us to provide robust estimates of
utilities using a significantly smaller number of utility elicitation
questions. We experiment with our technique on synthetic utility data and on a
real database of utility functions in the domain of prenatal diagnosis.Comment: Appears in Proceedings of the Sixteenth Conference on Uncertainty in
Artificial Intelligence (UAI2000
Nonuniform Dynamic Discretization in Hybrid Networks
We consider probabilistic inference in general hybrid networks, which include
continuous and discrete variables in an arbitrary topology. We reexamine the
question of variable discretization in a hybrid network aiming at minimizing
the information loss induced by the discretization. We show that a nonuniform
partition across all variables as opposed to uniform partition of each variable
separately reduces the size of the data structures needed to represent a
continuous function. We also provide a simple but efficient procedure for
nonuniform partition. To represent a nonuniform discretization in the computer
memory, we introduce a new data structure, which we call a Binary Split
Partition (BSP) tree. We show that BSP trees can be an exponential factor
smaller than the data structures in the standard uniform discretization in
multiple dimensions and show how the BSP trees can be used in the standard join
tree algorithm. We show that the accuracy of the inference process can be
significantly improved by adjusting discretization with evidence. We construct
an iterative anytime algorithm that gradually improves the quality of the
discretization and the accuracy of the answer on a query. We provide empirical
evidence that the algorithm converges.Comment: Appears in Proceedings of the Thirteenth Conference on Uncertainty in
Artificial Intelligence (UAI1997
Exact Inference in Networks with Discrete Children of Continuous Parents
Many real life domains contain a mixture of discrete and continuous variables
and can be modeled as hybrid Bayesian Networks. Animportant subclass of hybrid
BNs are conditional linear Gaussian (CLG) networks, where the conditional
distribution of the continuous variables given an assignment to the discrete
variables is a multivariate Gaussian. Lauritzen's extension to the clique tree
algorithm can be used for exact inference in CLG networks. However, many
domains also include discrete variables that depend on continuous ones, and CLG
networks do not allow such dependencies to berepresented. No exact inference
algorithm has been proposed for these enhanced CLG networks. In this paper, we
generalize Lauritzen's algorithm, providing the first "exact" inference
algorithm for augmented CLG networks - networks where continuous nodes are
conditional linear Gaussians but that also allow discrete children ofcontinuous
parents. Our algorithm is exact in the sense that it computes the exact
distributions over the discrete nodes, and the exact first and second moments
of the continuous ones, up to the accuracy obtained by numerical integration
used within thealgorithm. When the discrete children are modeled with softmax
CPDs (as is the case in many real world domains) the approximation of the
continuous distributions using the first two moments is particularly accurate.
Our algorithm is simple to implement and often comparable in its complexity to
Lauritzen's algorithm. We show empirically that it achieves substantially
higher accuracy than previous approximate algorithms.Comment: Appears in Proceedings of the Seventeenth Conference on Uncertainty
in Artificial Intelligence (UAI2001
Reasoning at the Right Time Granularity
Most real-world dynamic systems are composed of different components that
often evolve at very different rates. In traditional temporal graphical models,
such as dynamic Bayesian networks, time is modeled at a fixed granularity,
generally selected based on the rate at which the fastest component evolves.
Inference must then be performed at this fastest granularity, potentially at
significant computational cost. Continuous Time Bayesian Networks (CTBNs) avoid
time-slicing in the representation by modeling the system as evolving
continuously over time. The expectation-propagation (EP) inference algorithm of
Nodelman et al. (2005) can then vary the inference granularity over time, but
the granularity is uniform across all parts of the system, and must be selected
in advance. In this paper, we provide a new EP algorithm that utilizes a
general cluster graph architecture where clusters contain distributions that
can overlap in both space (set of variables) and time. This architecture allows
different parts of the system to be modeled at very different time
granularities, according to their current rate of evolution. We also provide an
information-theoretic criterion for dynamically re-partitioning the clusters
during inference to tune the level of approximation to the current rate of
evolution. This avoids the need to hand-select the appropriate granularity, and
allows the granularity to adapt as information is transmitted across the
network. We present experiments demonstrating that this approach can result in
significant computational savings.Comment: Appears in Proceedings of the Twenty-Third Conference on Uncertainty
in Artificial Intelligence (UAI2007
- …