280 research outputs found
Unique Information via Dependency Constraints
The partial information decomposition (PID) is perhaps the leading proposal
for resolving information shared between a set of sources and a target into
redundant, synergistic, and unique constituents. Unfortunately, the PID
framework has been hindered by a lack of a generally agreed-upon, multivariate
method of quantifying the constituents. Here, we take a step toward rectifying
this by developing a decomposition based on a new method that quantifies unique
information. We first develop a broadly applicable method---the dependency
decomposition---that delineates how statistical dependencies influence the
structure of a joint distribution. The dependency decomposition then allows us
to define a measure of the information about a target that can be uniquely
attributed to a particular source as the least amount which the source-target
statistical dependency can influence the information shared between the sources
and the target. The result is the first measure that satisfies the core axioms
of the PID framework while not satisfying the Blackwell relation, which depends
on a particular interpretation of how the variables are related. This makes a
key step forward to a practical PID.Comment: 15 pages, 7 figures, 2 tables, 3 appendices;
http://csc.ucdavis.edu/~cmg/compmech/pubs/idep.ht
Secret Sharing and Shared Information
Secret sharing is a cryptographic discipline in which the goal is to
distribute information about a secret over a set of participants in such a way
that only specific authorized combinations of participants together can
reconstruct the secret. Thus, secret sharing schemes are systems of variables
in which it is very clearly specified which subsets have information about the
secret. As such, they provide perfect model systems for information
decompositions. However, following this intuition too far leads to an
information decomposition with negative partial information terms, which are
difficult to interpret. One possible explanation is that the partial
information lattice proposed by Williams and Beer is incomplete and has to be
extended to incorporate terms corresponding to higher order redundancy. These
results put bounds on information decompositions that follow the partial
information framework, and they hint at where the partial information lattice
needs to be improved.Comment: 9 pages, 1 figure. The material was presented at a Workshop on
information decompositions at FIAS, Frankfurt, in 12/2016. The revision
includes changes in the definition of combinations of secret sharing schemes.
Section 3 and Appendix now discusses in how far existing measures satisfy the
proposed properties. The concluding section is considerably revise
Measuring multivariate redundant information with pointwise common change in surprisal
The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables Xi . It can be thought of as quantifying overlapping information content or similarities in the representation of S between the Xi . We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples
From Babel to Boole: The Logical Organization of Information Decompositions
The conventional approach to the general Partial Information Decomposition
(PID) problem has been redundancy-based: specifying a measure of redundant
information between collections of source variables induces a PID via
Moebius-Inversion over the so called redundancy lattice. Despite the prevalence
of this method, there has been ongoing interest in examining the problem
through the lens of different base-concepts of information, such as synergy,
unique information, or union information. Yet, a comprehensive understanding of
the logical organization of these different based-concepts and their associated
PIDs remains elusive. In this work, we apply the mereological formulation of
PID that we introduced in a recent paper to shed light on this problem. Within
the mereological approach base-concepts can be expressed in terms of conditions
phrased in formal logic on the specific parthood relations between the PID
components and the different mutual information terms. We set forth a general
pattern of these logical conditions of which all PID base-concepts in the
literature are special cases and that also reveals novel base-concepts, in
particular a concept we call ``vulnerable information''.Comment: 20 pages, 8 figure
A Bivariate Measure of Redundant Information
We define a measure of redundant information based on projections in the
space of probability distributions. Redundant information between random
variables is information that is shared between those variables. But in
contrast to mutual information, redundant information denotes information that
is shared about the outcome of a third variable. Formalizing this concept, and
being able to measure it, is required for the non-negative decomposition of
mutual information into redundant and synergistic information. Previous
attempts to formalize redundant or synergistic information struggle to capture
some desired properties. We introduce a new formalism for redundant information
and prove that it satisfies all the properties necessary outlined in earlier
work, as well as an additional criterion that we propose to be necessary to
capture redundancy. We also demonstrate the behaviour of this new measure for
several examples, compare it to previous measures and apply it to the
decomposition of transfer entropy.Comment: 16 pages, 15 figures, 1 table, added citation to Griffith et al 2012,
Maurer et al 199
Partial Information Decomposition via Deficiency for Multivariate Gaussians
Bivariate partial information decompositions (PIDs) characterize how the
information in a "message" random variable is decomposed between two
"constituent" random variables in terms of unique, redundant and synergistic
information components. These components are a function of the joint
distribution of the three variables, and are typically defined using an
optimization over the space of all possible joint distributions. This makes it
computationally challenging to compute PIDs in practice and restricts their use
to low-dimensional random vectors. To ease this burden, we consider the case of
jointly Gaussian random vectors in this paper. This case was previously
examined by Barrett (2015), who showed that certain operationally
well-motivated PIDs reduce to a closed form expression for scalar messages.
Here, we show that Barrett's result does not extend to vector messages in
general, and characterize the set of multivariate Gaussian distributions that
reduce to closed-form. Then, for all other multivariate Gaussian distributions,
we propose a convex optimization framework for approximately computing a
specific PID definition based on the statistical concept of deficiency. Using
simplifying assumptions specific to the Gaussian case, we provide an efficient
algorithm to approximately compute the bivariate PID for multivariate Gaussian
variables with tens or even hundreds of dimensions. We also theoretically and
empirically justify the goodness of this approximation.Comment: Presented at ISIT 2022. This version has been updated to reflect the
final conference publication, including appendices. It also corrects
technical errors in Remark 1 and Appendix C, adds a new experiment, and has a
substantially improved presentation as well as additional detail in the
appendix, compared to the previous arxiv versio
A New Framework for Decomposing Multivariate Information
What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much-criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. This thesis presents a new framework for information decomposition that is based upon the decomposition of pointwise mutual information rather than mutual information. The framework is derived in two separate ways. The first of these derivations is based upon a modified version of the original axiomatic approach taken by Williams and Beer. However, to overcome the difficulty associated with signed pointwise mutual information, the decomposition is applied separately to the unsigned entropic components of pointwise mutual information which are referred to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Based upon an operational interpretation of redundancy, measures of redundant specificity and redundant ambiguity are defined which enables one to evaluate the partial information atoms separately for each lattice. These separate atoms can then be recombined to yield the sought-after multivariate information decomposition. This framework is applied to canonical examples from the literature and the results and various properties of the decomposition are discussed. In particular, the pointwise decomposition using specificity and ambiguity is shown to satisfy a chain rule over target variables, which provides new insights into the so-called two-bit-copy example. The second approach begins by considering the distinct ways in which two marginal observers can share their information with the non-observing individual third party. Several novel measures of information content are introduced, namely the union, intersection and unique information contents. Next, the algebraic structure of these new measures of shared marginal information is explored, and it is shown that the structure of shared marginal information is that of a distributive lattice. Furthermore, by using the fundamental theorem of distributive lattices, it is shown that these new measures are isomorphic to a ring of sets. Finally, by combining this structure together with the semi-lattice of joint information, the redundancy lattice form partial information decomposition is found to be embedded within this larger algebraic structure. However, since this structure considers information contents, it is actually equivalent to the specificity lattice from the first derivation of pointwise partial information decomposition. The thesis then closes with a discussion about whether or not one should combine the information contents from the specificity and ambiguity lattices
- …