299 research outputs found
Dependence of dissipation on the initial distribution over states
We analyze how the amount of work dissipated by a fixed nonequilibrium
process depends on the initial distribution over states. Specifically, we
compare the amount of dissipation when the process is used with some specified
initial distribution to the minimal amount of dissipation possible for any
initial distribution. We show that the difference between those two amounts of
dissipation is given by a simple information-theoretic function that depends
only on the initial and final state distributions. Crucially, this difference
is independent of the details of the process relating those distributions. We
then consider how dissipation depends on the initial distribution for a
'computer', i.e., a nonequilibrium process whose dynamics over coarse-grained
macrostates implement some desired input-output map. We show that our results
still apply when stated in terms of distributions over the computer's
coarse-grained macrostates. This can be viewed as a novel thermodynamic cost of
computation, reflecting changes in the distribution over inputs rather than the
logical dynamics of the computation
Estimating Mixture Entropy with Pairwise Distances
Mixture distributions arise in many parametric and non-parametric settings --
for example, in Gaussian mixture models and in non-parametric estimation. It is
often necessary to compute the entropy of a mixture, but, in most cases, this
quantity has no closed-form expression, making some form of approximation
necessary. We propose a family of estimators based on a pairwise distance
function between mixture components, and show that this estimator class has
many attractive properties. For many distributions of interest, the proposed
estimators are efficient to compute, differentiable in the mixture parameters,
and become exact when the mixture components are clustered. We prove this
family includes lower and upper bounds on the mixture entropy. The Chernoff
-divergence gives a lower bound when chosen as the distance function,
with the Bhattacharyya distance providing the tightest lower bound for
components that are symmetric and members of a location family. The
Kullback-Leibler divergence gives an upper bound when used as the distance
function. We provide closed-form expressions of these bounds for mixtures of
Gaussians, and discuss their applications to the estimation of mutual
information. We then demonstrate that our bounds are significantly tighter than
well-known existing bounds using numeric simulations. This estimator class is
very useful in optimization problems involving maximization/minimization of
entropy and mutual information, such as MaxEnt and rate distortion problems.Comment: Corrects several errata in published version, in particular in
Section V (bounds on mutual information
Generalized Zurek's bound on the cost of an individual classical or quantum computation
We consider the minimal thermodynamic cost of an individual computation,
where a single input is mapped into a single output . In prior work,
Zurek proposed that this cost was given by , the conditional
Kolmogorov complexity of given (up to an additive constant which does
not depend on or ). However, this result was derived from an informal
argument, applied only to deterministic computations, and had an arbitrary
dependence on the choice of protocol (via the additive constant). Here we use
stochastic thermodynamics to derive a generalized version of Zurek's bound from
a rigorous Hamiltonian formulation. Our bound applies to all quantum and
classical processes, whether noisy or deterministic, and it explicitly captures
the dependence on the protocol. We show that is a minimal cost of
mapping to that must be paid using some combination of heat, noise, and
protocol complexity, implying a tradeoff between these three resources. Our
result is a kind of ``algorithmic fluctuation theorem'' with implications for
the relationship between the Second Law and the Physical Church-Turing thesis
Evaluation of linear classifiers on articles containing pharmacokinetic evidence of drug-drug interactions
Background. Drug-drug interaction (DDI) is a major cause of morbidity and
mortality. [...] Biomedical literature mining can aid DDI research by
extracting relevant DDI signals from either the published literature or large
clinical databases. However, though drug interaction is an ideal area for
translational research, the inclusion of literature mining methodologies in DDI
workflows is still very preliminary. One area that can benefit from literature
mining is the automatic identification of a large number of potential DDIs,
whose pharmacological mechanisms and clinical significance can then be studied
via in vitro pharmacology and in populo pharmaco-epidemiology. Experiments. We
implemented a set of classifiers for identifying published articles relevant to
experimental pharmacokinetic DDI evidence. These documents are important for
identifying causal mechanisms behind putative drug-drug interactions, an
important step in the extraction of large numbers of potential DDIs. We
evaluate performance of several linear classifiers on PubMed abstracts, under
different feature transformation and dimensionality reduction methods. In
addition, we investigate the performance benefits of including various
publicly-available named entity recognition features, as well as a set of
internally-developed pharmacokinetic dictionaries. Results. We found that
several classifiers performed well in distinguishing relevant and irrelevant
abstracts. We found that the combination of unigram and bigram textual features
gave better performance than unigram features alone, and also that
normalization transforms that adjusted for feature frequency and document
length improved classification. For some classifiers, such as linear
discriminant analysis (LDA), proper dimensionality reduction had a large impact
on performance. Finally, the inclusion of NER features and dictionaries was
found not to help classification.Comment: Pacific Symposium on Biocomputing, 201
Caveats for information bottleneck in deterministic scenarios
Information bottleneck (IB) is a method for extracting information from one
random variable that is relevant for predicting another random variable
. To do so, IB identifies an intermediate "bottleneck" variable that has
low mutual information and high mutual information . The "IB
curve" characterizes the set of bottleneck variables that achieve maximal
for a given , and is typically explored by maximizing the "IB
Lagrangian", . In some cases, is a deterministic
function of , including many classification problems in supervised learning
where the output class is a deterministic function of the input . We
demonstrate three caveats when using IB in any situation where is a
deterministic function of : (1) the IB curve cannot be recovered by
maximizing the IB Lagrangian for different values of ; (2) there are
"uninteresting" trivial solutions at all points of the IB curve; and (3) for
multi-layer classifiers that achieve low prediction error, different layers
cannot exhibit a strict trade-off between compression and prediction, contrary
to a recent proposal. We also show that when is a small perturbation away
from being a deterministic function of , these three caveats arise in an
approximate way. To address problem (1), we propose a functional that, unlike
the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the
three caveats on the MNIST dataset
Nonlinear Information Bottleneck
Information bottleneck (IB) is a technique for extracting information in one
random variable that is relevant for predicting another random variable
. IB works by encoding in a compressed "bottleneck" random variable
from which can be accurately decoded. However, finding the optimal
bottleneck variable involves a difficult optimization problem, which until
recently has been considered for only two limited cases: discrete and
with small state spaces, and continuous and with a Gaussian joint
distribution (in which case optimal encoding and decoding maps are linear). We
propose a method for performing IB on arbitrarily-distributed discrete and/or
continuous and , while allowing for nonlinear encoding and decoding
maps. Our approach relies on a novel non-parametric upper bound for mutual
information. We describe how to implement our method using neural networks. We
then show that it achieves better performance than the recently-proposed
"variational IB" method on several real-world datasets
Modularity and the spread of perturbations in complex dynamical systems
We propose a method to decompose dynamical systems based on the idea that
modules constrain the spread of perturbations. We find partitions of system
variables that maximize 'perturbation modularity', defined as the
autocovariance of coarse-grained perturbed trajectories. The measure
effectively separates the fast intramodular from the slow intermodular dynamics
of perturbation spreading (in this respect, it is a generalization of the
'Markov stability' method of network community detection). Our approach
captures variation of modular organization across different system states, time
scales, and in response to different kinds of perturbations: aspects of
modularity which are all relevant to real-world dynamical systems. It offers a
principled alternative to detecting communities in networks of statistical
dependencies between system variables (e.g., 'relevance networks' or
'functional networks'). Using coupled logistic maps, we demonstrate that the
method uncovers hierarchical modular organization planted in a system's
coupling matrix. Additionally, in homogeneously-coupled map lattices, it
identifies the presence of self-organized modularity that depends on the
initial state, dynamical parameters, and type of perturbations. Our approach
offers a powerful tool for exploring the modular organization of complex
dynamical systems
The Revival of the Russian Literary Avantgarde
This study is devoted to the authors who initiated the revival of the Russian avantgarde tradition, which had been brutally suppressed by the Soviet authorities in the mid- 1930s. The revival of this tradition took place in the literary underground, where writers who endeavored to fulfil this challenging task largely remained until the collapse of the Soviet regime. Most of them emerged from obscurity only at the beginning of the 1990s, which explains why their dramatic and fascinating history has been so little examined by scholars. Although the situation has changed significantly in the last decade, during which some insightful studies have appeared in both Russia and the West, the subject obviously requires more thorough and systematic exploration. This book aims to narrow important gaps in the scholarship on the Russian literary avantgarde during its least investigated period
- …