141 research outputs found
Robustness and Conditional Independence Ideals
We study notions of robustness of Markov kernels and probability distribution
of a system that is described by input random variables and one output
random variable. Markov kernels can be expanded in a series of potentials that
allow to describe the system's behaviour after knockouts. Robustness imposes
structural constraints on these potentials. Robustness of probability
distributions is defined via conditional independence statements. These
statements can be studied algebraically. The corresponding conditional
independence ideals are related to binary edge ideals. The set of robust
probability distributions lies on an algebraic variety. We compute a Gr\"obner
basis of this ideal and study the irreducible decomposition of the variety.
These algebraic results allow to parametrize the set of all robust probability
distributions.Comment: 16 page
Adaptive Dynamics for Interacting Markovian Processes
Dynamics of information flow in adaptively interacting stochastic processes
is studied. We give an extended form of game dynamics for Markovian processes
and study its behavior to observe information flow through the system. Examples
of the adaptive dynamics for two stochastic processes interacting through
matching pennies game interaction are exhibited along with underlying causal
structure
Quantifying Morphological Computation
The field of embodied intelligence emphasises the importance of the
morphology and environment with respect to the behaviour of a cognitive system.
The contribution of the morphology to the behaviour, commonly known as
morphological computation, is well-recognised in this community. We believe
that the field would benefit from a formalisation of this concept as we would
like to ask how much the morphology and the environment contribute to an
embodied agent's behaviour, or how an embodied agent can maximise the
exploitation of its morphology within its environment. In this work we derive
two concepts of measuring morphological computation, and we discuss their
relation to the Information Bottleneck Method. The first concepts asks how much
the world contributes to the overall behaviour and the second concept asks how
much the agent's action contributes to a behaviour. Various measures are
derived from the concepts and validated in two experiments which highlight
their strengths and weaknesses
Information-theoretic inference of common ancestors
A directed acyclic graph (DAG) partially represents the conditional
independence structure among observations of a system if the local Markov
condition holds, that is, if every variable is independent of its
non-descendants given its parents. In general, there is a whole class of DAGs
that represents a given set of conditional independence relations. We are
interested in properties of this class that can be derived from observations of
a subsystem only. To this end, we prove an information theoretic inequality
that allows for the inference of common ancestors of observed parts in any DAG
representing some unknown larger system. More explicitly, we show that a large
amount of dependence in terms of mutual information among the observations
implies the existence of a common ancestor that distributes this information.
Within the causal interpretation of DAGs our result can be seen as a
quantitative extension of Reichenbach's Principle of Common Cause to more than
two variables. Our conclusions are valid also for non-probabilistic
observations such as binary strings, since we state the proof for an
axiomatized notion of mutual information that includes the stochastic as well
as the algorithmic version.Comment: 18 pages, 4 figure
Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines
We improve recently published results about resources of Restricted Boltzmann
Machines (RBM) and Deep Belief Networks (DBN) required to make them Universal
Approximators. We show that any distribution p on the set of binary vectors of
length n can be arbitrarily well approximated by an RBM with k-1 hidden units,
where k is the minimal number of pairs of binary vectors differing in only one
entry such that their union contains the support set of p. In important cases
this number is half of the cardinality of the support set of p. We construct a
DBN with 2^n/2(n-b), b ~ log(n), hidden layers of width n that is capable of
approximating any distribution on {0,1}^n arbitrarily well. This confirms a
conjecture presented by Le Roux and Bengio 2010
- …