16 research outputs found
Successive Refinement of Abstract Sources
In successive refinement of information, the decoder refines its
representation of the source progressively as it receives more encoded bits.
The rate-distortion region of successive refinement describes the minimum rates
required to attain the target distortions at each decoding stage. In this
paper, we derive a parametric characterization of the rate-distortion region
for successive refinement of abstract sources. Our characterization extends
Csiszar's result to successive refinement, and generalizes a result by Tuncel
and Rose, applicable for finite alphabet sources, to abstract sources. This
characterization spawns a family of outer bounds to the rate-distortion region.
It also enables an iterative algorithm for computing the rate-distortion
region, which generalizes Blahut's algorithm to successive refinement. Finally,
it leads a new nonasymptotic converse bound. In all the scenarios where the
dispersion is known, this bound is second-order optimal.
In our proof technique, we avoid Karush-Kuhn-Tucker conditions of optimality,
and we use basic tools of probability theory. We leverage the Donsker-Varadhan
lemma for the minimization of relative entropy on abstract probability spaces.Comment: Extended version of a paper presented at ISIT 201
Probabilistic Deterministic Finite Automata and Recurrent Networks, Revisited
Reservoir computers (RCs) and recurrent neural networks (RNNs) can mimic any
finite-state automaton in theory, and some workers demonstrated that this can
hold in practice. We test the capability of generalized linear models, RCs, and
Long Short-Term Memory (LSTM) RNN architectures to predict the stochastic
processes generated by a large suite of probabilistic deterministic
finite-state automata (PDFA). PDFAs provide an excellent performance benchmark
in that they can be systematically enumerated, the randomness and correlation
structure of their generated processes are exactly known, and their optimal
memory-limited predictors are easily computed. Unsurprisingly, LSTMs outperform
RCs, which outperform generalized linear models. Surprisingly, each of these
methods can fall short of the maximal predictive accuracy by as much as 50%
after training and, when optimized, tend to fall short of the maximal
predictive accuracy by ~5%, even though previously available methods achieve
maximal predictive accuracy with orders-of-magnitude less data. Thus, despite
the representational universality of RCs and RNNs, using them can engender a
surprising predictive gap for simple stimuli. One concludes that there is an
important and underappreciated role for methods that infer "causal states" or
"predictive state representations"
Parameters estimation for spatio-temporal maximum entropy distributions: application to neural spike trains
We propose a numerical method to learn Maximum Entropy (MaxEnt) distributions
with spatio-temporal constraints from experimental spike trains. This is an
extension of two papers [10] and [4] who proposed the estimation of parameters
where only spatial constraints were taken into account. The extension we
propose allows to properly handle memory effects in spike statistics, for large
sized neural networks.Comment: 34 pages, 33 figure
Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems
In Rate Distortion (RD) problems one seeks reduced representations of a
source that meet a target distortion constraint. Such optimal representations
undergo topological transitions at some critical rate values, when their
cardinality or dimensionality change. We study the convergence time of the
Arimoto-Blahut alternating projection algorithms, used to solve such problems,
near those critical points, both for the Rate Distortion and Information
Bottleneck settings. We argue that they suffer from Critical Slowing Down -- a
diverging number of iterations for convergence -- near the critical points.
This phenomenon can have theoretical and practical implications for both
Machine Learning and Data Compression problems.Comment: 9 pages, 2 figures, ISIT 2021 submissio
The rate-distortion function for successive refinement of abstract sources
In successive refinement of information, the decoder refines its representation of the source progressively as it receives more encoded bits. The rate-distortion region of successive refinement describes the minimum rates required to attain the target distortions at each decoding stage. In this paper, we derive a parametric characterization of the rate-distortion region for successive refinement of abstract sources. Our characterization extends Csiszar's result [1] to successive refinement, and generalizes a result by Tuncel and Rose [2], applicable for finite alphabet sources, to abstract sources. The new characterization leads to a family of outer bounds to the rate-distortion region. It also enables new nonasymptotic converse bounds
The Generalized Multiplicative Gradient Method and Its Convergence Rate Analysis
Multiplicative gradient method is a classical and effective method for
solving the positron emission tomography (PET) problem. In this work, we
propose a generalization of this method on a broad class of problems, which
includes the PET problem as a special case. We show that this generalized
method converges with rate .Comment: 20 page
A Constrained BA Algorithm for Rate-Distortion and Distortion-Rate Functions
The Blahut-Arimoto (BA) algorithm has played a fundamental role in the
numerical computation of rate-distortion (RD) functions. This algorithm
possesses a desirable monotonic convergence property by alternatively
minimizing its Lagrangian with a fixed multiplier. In this paper, we propose a
novel modification of the BA algorithm, wherein the multiplier is updated
through a one-dimensional root-finding step using a monotonic univariate
function, efficiently implemented by Newton's method in each iteration.
Consequently, the modified algorithm directly computes the RD function for a
given target distortion, without exploring the entire RD curve as in the
original BA algorithm. Moreover, this modification presents a versatile
framework, applicable to a wide range of problems, including the computation of
distortion-rate (DR) functions. Theoretical analysis shows that the outputs of
the modified algorithms still converge to the solutions of the RD and DR
functions with rate , where is the number of iterations.
Additionally, these algorithms provide -approximation solutions
with
arithmetic operations, where are the sizes of source and reproduced
alphabets respectively. Numerical experiments demonstrate that the modified
algorithms exhibit significant acceleration compared with the original BA
algorithms and showcase commendable performance across classical source
distributions such as discretized Gaussian, Laplacian and uniform sources.Comment: Version_
Computing the Rate-Distortion Function of Gray-Wyner System
In this paper, the rate-distortion theory of Gray-Wyner lossy source coding
system is investigated. An iterative algorithm is proposed to compute
rate-distortion function for general successive source. For the case of jointly
Gaussian distributed sources, the Lagrangian analysis of scalable source coding
in [1] is generalized to the Gray-Wyner instance. Upon the existing
single-letter characterization of the rate-distortion region, we compute and
determine an analytical expression of the rate-distortion function under
quadratic distortion constraints. According to the rate-distortion function,
another approach, different from Viswanatha et al. used, is provided to compute
Wyner's Common Information. The convergence of proposed iterative algorithm, RD
function with different parameters and the projection plane of RD region are
also shown via numerical simulations at last.Comment: This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessibl