Coding against a Limited-view Adversary: The Effect of Causality and Feedback

We consider the problem of communication over a multi-path network in the presence of a causal adversary. The limited-view causal adversary is able to eavesdrop on a subset of links and also jam on a potentially overlapping subset of links based on the current and past information. To ensure that the communication takes place reliably and secretly, resilient network codes with necessary redundancy are needed. We study two adversarial models - additive and overwrite jamming and we optionally assume passive feedback from decoder to encoder, i.e., the encoder sees everything that the decoder sees. The problem assumes transmissions are in the large alphabet regime. For both jamming models, we find the capacity under four scenarios - reliability without feedback, reliability and secrecy without feedback, reliability with passive feedback, reliability and secrecy with passive feedback. We observe that, in comparison to the non-causal setting, the capacity with a causal adversary is strictly increased for a wide variety of parameter settings and present our intuition through several examples.Comment: 15 page

Community Detection in the Multi-View Stochastic Block Model

This paper considers the problem of community detection on multiple potentially correlated graphs from an information-theoretical perspective. We first put forth a random graph model, called the multi-view stochastic block model (MVSBM), designed to generate correlated graphs on the same set of nodes (with cardinality $n$). The $n$ nodes are partitioned into two disjoint communities of equal size. The presence or absence of edges in the graphs for each pair of nodes depends on whether the two nodes belong to the same community or not. The objective for the learner is to recover the hidden communities with observed graphs. Our technical contributions are two-fold: (i) We establish an information-theoretic upper bound (Theorem~1) showing that exact recovery of community is achievable when the model parameters of MVSBM exceed a certain threshold. (ii) Conversely, we derive an information-theoretic lower bound (Theorem~2) showing that when the model parameters of MVSBM fall below the aforementioned threshold, then for any estimator, the expected number of misclassified nodes will always be greater than one. Our results for the MVSBM recover several prior results for community detection in the standard SBM as well as in multiple independent SBMs as special cases.Comment: Submitted to IEEE for possible publicatio

Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems

As one of the central tasks in machine learning, regression finds lots of applications in different fields. An existing common practice for solving regression problems is the mean square error (MSE) minimization approach or its regularized variants which require prior knowledge about the models. Recently, Yi et al., proposed a mutual information based supervised learning framework where they introduced a label entropy regularization which does not require any prior knowledge. When applied to classification tasks and solved via a stochastic gradient descent (SGD) optimization algorithm, their approach achieved significant improvement over the commonly used cross entropy loss and its variants. However, they did not provide a theoretical convergence analysis of the SGD algorithm for the proposed formulation. Besides, applying the framework to regression tasks is nontrivial due to the potentially infinite support set of the label. In this paper, we investigate the regression under the mutual information based supervised learning framework. We first argue that the MSE minimization approach is equivalent to a conditional entropy learning problem, and then propose a mutual information learning formulation for solving regression problems by using a reparameterization technique. For the proposed formulation, we give the convergence analysis of the SGD algorithm for solving it in practice. Finally, we consider a multi-output regression data model where we derive the generalization performance lower bound in terms of the mutual information associated with the underlying data distribution. The result shows that the high dimensionality can be a bless instead of a curse, which is controlled by a threshold. We hope our work will serve as a good starting point for further research on the mutual information based regression.Comment: 28 pages, 2 figures, presubmitted to AISTATS2023 for reviewin