22 research outputs found
Causal Dependence Tree Approximations of Joint Distributions for Multiple Random Processes
We investigate approximating joint distributions of random processes with
causal dependence tree distributions. Such distributions are particularly
useful in providing parsimonious representation when there exists causal
dynamics among processes. By extending the results by Chow and Liu on
dependence tree approximations, we show that the best causal dependence tree
approximation is the one which maximizes the sum of directed informations on
its edges, where best is defined in terms of minimizing the KL-divergence
between the original and the approximate distribution. Moreover, we describe a
low-complexity algorithm to efficiently pick this approximate distribution.Comment: 9 pages, 15 figure
Capacity of Continuous Channels with Memory via Directed Information Neural Estimator
Calculating the capacity (with or without feedback) of channels with memory
and continuous alphabets is a challenging task. It requires optimizing the
directed information (DI) rate over all channel input distributions. The
objective is a multi-letter expression, whose analytic solution is only known
for a few specific cases. When no analytic solution is present or the channel
model is unknown, there is no unified framework for calculating or even
approximating capacity. This work proposes a novel capacity estimation
algorithm that treats the channel as a `black-box', both when feedback is or is
not present. The algorithm has two main ingredients: (i) a neural distribution
transformer (NDT) model that shapes a noise variable into the channel input
distribution, which we are able to sample, and (ii) the DI neural estimator
(DINE) that estimates the communication rate of the current NDT model. These
models are trained by an alternating maximization procedure to both estimate
the channel capacity and obtain an NDT for the optimal input distribution. The
method is demonstrated on the moving average additive Gaussian noise channel,
where it is shown that both the capacity and feedback capacity are estimated
without knowledge of the channel transition kernel. The proposed estimation
framework opens the door to a myriad of capacity approximation results for
continuous alphabet channels that were inaccessible until now