9 research outputs found
In-Network Learning: Distributed Training and Inference in Networks
It is widely perceived that leveraging the success of modern machine learning
techniques to mobile devices and wireless networks has the potential of
enabling important new services. This, however, poses significant challenges,
essentially due to that both data and processing power are highly distributed
in a wireless network. In this paper, we develop a learning algorithm and an
architecture that make use of multiple data streams and processing units, not
only during the training phase but also during the inference phase. In
particular, the analysis reveals how inference propagates and fuses across a
network. We study the design criterion of our proposed method and its bandwidth
requirements. Also, we discuss implementation aspects using neural networks in
typical wireless radio access; and provide experiments that illustrate benefits
over state-of-the-art techniques.Comment: Submitted to the IEEE Journal on Selected Areas in Communications
(JSAC) Series on Machine Learning for Communications and Networks. arXiv
admin note: substantial text overlap with arXiv:2104.1492
Bottleneck Problems: Information and Estimation-Theoretic View
Information bottleneck (IB) and privacy funnel (PF) are two closely related
optimization problems which have found applications in machine learning, design
of privacy algorithms, capacity problems (e.g., Mrs. Gerber's Lemma), strong
data processing inequalities, among others. In this work, we first investigate
the functional properties of IB and PF through a unified theoretical framework.
We then connect them to three information-theoretic coding problems, namely
hypothesis testing against independence, noisy source coding and dependence
dilution. Leveraging these connections, we prove a new cardinality bound for
the auxiliary variable in IB, making its computation more tractable for
discrete random variables.
In the second part, we introduce a general family of optimization problems,
termed as \textit{bottleneck problems}, by replacing mutual information in IB
and PF with other notions of mutual information, namely -information and
Arimoto's mutual information. We then argue that, unlike IB and PF, these
problems lead to easily interpretable guarantee in a variety of inference tasks
with statistical constraints on accuracy and privacy. Although the underlying
optimization problems are non-convex, we develop a technique to evaluate
bottleneck problems in closed form by equivalently expressing them in terms of
lower convex or upper concave envelope of certain functions. By applying this
technique to binary case, we derive closed form expressions for several
bottleneck problems
Distributed Information Bottleneck Method for Discrete and Gaussian Sources
Submitted to the 2018 International Zurich Seminar on Information and Communication (IZS)International audienceWe study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another signal. The model generalizes Tishby's centralized information bottleneck method to the setting of multiple distributed encoders. We establish single-letter characterizations of the information-rate region of this problem for both i) a class of discrete memoryless sources and ii) memoryless vector Gaussian sources. Furthermore, assuming a sum constraint on rate or complexity, for both models we develop Blahut-Arimoto type iterative algorithms that allow to compute optimal information-rate trade-offs, by iterating over a set of self-consistent equations
Distributed Information Bottleneck Method for Discrete and Gaussian Sources
Submitted to the 2018 International Zurich Seminar on Information and Communication (IZS)International audienceWe study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another signal. The model generalizes Tishby's centralized information bottleneck method to the setting of multiple distributed encoders. We establish single-letter characterizations of the information-rate region of this problem for both i) a class of discrete memoryless sources and ii) memoryless vector Gaussian sources. Furthermore, assuming a sum constraint on rate or complexity, for both models we develop Blahut-Arimoto type iterative algorithms that allow to compute optimal information-rate trade-offs, by iterating over a set of self-consistent equations