Justification of Logarithmic Loss via the Benefit of Side Information

We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference on a random variable under this loss function. When such a measure satisfies a natural data processing property, and the random variable of interest has alphabet size greater than two, we show that it is uniquely characterized by the mutual information, and the corresponding loss function coincides with logarithmic loss. In doing so, our work provides a new characterization of mutual information, and justifies its use as a measure of relevance. When the alphabet is binary, we characterize the only admissible forms the measure of relevance can assume while obeying the specified data processing property. Our results naturally extend to measuring causal influence between stochastic processes, where we unify different causal-inference measures in the literature as instantiations of directed information

Online Learning of k-CNF Boolean Functions

This paper revisits the problem of learning a k-CNF Boolean function from examples in the context of online learning under the logarithmic loss. In doing so, we give a Bayesian interpretation to one of Valiant's celebrated PAC learning algorithms, which we then build upon to derive two efficient, online, probabilistic, supervised learning algorithms for predicting the output of an unknown k-CNF Boolean function. We analyze the loss of our methods, and show that the cumulative log-loss can be upper bounded, ignoring logarithmic factors, by a polynomial function of the size of each example.Comment: 20 LaTeX pages. 2 Algorithms. Some Theorem

On the Information Bottleneck Problems: An Information Theoretic Perspective

International Zurich Seminar on Information and Communication (IZS), February 26 – 28, 202

Data Offloading in Load Coupled Networks: A Utility Maximization Framework

We provide a general framework for the problem of data offloading in a heterogeneous wireless network, where some demand of cellular users is served by a complementary network. The complementary network is either a small-cell network that shares the same resources as the cellular network, or a WiFi network that uses orthogonal resources. For a given demand served in a cellular network, the load, or the level of resource usage, of each cell depends in a non-linear manner on the load of other cells due to the mutual coupling of interference seen by one another. With load coupling, we optimize the demand to be served in the cellular or the complementary networks, so as to maximize a utility function. We consider three representative utility functions that balance, to varying degrees, the revenue from serving the users vs the user fairness. We establish conditions for which the optimization problem has a feasible solution and is convex, and hence tractable to numerical computations. Finally, we propose a strategy with theoretical justification to constrain the load to some maximum value, as required for practical implementation. Numerical studies are conducted for both under-loaded and over-loaded networks.Comment: 12 pages, accepted for publication in IEEE Transactions on Wireless Communication