106 research outputs found
Ensemble Estimation of Information Divergence
Recent work has focused on the problem of nonparametric estimation of information divergence functionals between two continuous random variables. Many existing approaches require either restrictive assumptions about the density support set or difficult calculations at the support set boundary which must be known a priori. The mean squared error (MSE) convergence rate of a leave-one-out kernel density plug-in divergence functional estimator for general bounded density support sets is derived where knowledge of the support boundary, and therefore, the boundary correction is not required. The theory of optimally weighted ensemble estimation is generalized to derive a divergence estimator that achieves the parametric rate when the densities are sufficiently smooth. Guidelines for the tuning parameter selection and the asymptotic distribution of this estimator are provided. Based on the theory, an empirical estimator of Rényi-α divergence is proposed that greatly outperforms the standard kernel density plug-in estimator in terms of mean squared error, especially in high dimensions. The estimator is shown to be robust to the choice of tuning parameters. We show extensive simulation results that verify the theoretical results of our paper. Finally, we apply the proposed estimator to estimate the bounds on the Bayes error rate of a cell classification problem
Divergence Measures
Data science, information theory, probability theory, statistical learning and other related disciplines greatly benefit from non-negative measures of dissimilarity between pairs of probability measures. These are known as divergence measures, and exploring their mathematical foundations and diverse applications is of significant interest. The present Special Issue, entitled “Divergence Measures: Mathematical Foundations and Applications in Information-Theoretic and Statistical Problems”, includes eight original contributions, and it is focused on the study of the mathematical properties and applications of classical and generalized divergence measures from an information-theoretic perspective. It mainly deals with two key generalizations of the relative entropy: namely, the R_ényi divergence and the important class of f -divergences. It is our hope that the readers will find interest in this Special Issue, which will stimulate further research in the study of the mathematical foundations and applications of divergence measures
Geometric Inhomogeneous Random Graphs for Algorithm Engineering
The design and analysis of graph algorithms is heavily based on the worst case.
In practice, however, many algorithms perform much better than the worst case would suggest.
Furthermore, various problems can be tackled more efficiently if one assumes the input to be, in a sense, realistic.
The field of network science, which studies the structure and emergence of real-world networks, identifies locality and heterogeneity as two frequently occurring properties.
A popular model that captures these properties are geometric inhomogeneous random graphs (GIRGs), which is a generalization of hyperbolic random graphs (HRGs).
Aside from their importance to network science, GIRGs can be an immensely valuable tool in algorithm engineering.
Since they convincingly mimic real-world networks, guarantees about quality and performance of an algorithm on instances of the model can be transferred to real-world applications.
They have model parameters to control the amount of heterogeneity and locality, which allows to evaluate those properties in isolation while keeping the rest fixed.
Moreover, they can be efficiently generated which allows for experimental analysis.
While realistic instances are often rare, generated instances are readily available.
Furthermore, the underlying geometry of GIRGs helps to visualize the network, e.g.,~for debugging or to improve understanding of its structure.
The aim of this work is to demonstrate the capabilities of geometric inhomogeneous random graphs in algorithm engineering and establish them as routine tools to replace previous models like the Erd\H{o}s-R{\\u27e}nyi model, where each edge exists with equal probability.
We utilize geometric inhomogeneous random graphs to design, evaluate, and optimize efficient algorithms for realistic inputs.
In detail, we provide the currently fastest sequential generator for GIRGs and HRGs and describe algorithms for maximum flow, directed spanning arborescence, cluster editing, and hitting set.
For all four problems, our implementations beat the state-of-the-art on realistic inputs.
On top of providing crucial benchmark instances, GIRGs allow us to obtain valuable insights.
Most notably, our efficient generator allows us to
experimentally show sublinear running time of our flow algorithm,
investigate the solution structure of cluster editing,
complement our benchmark set of arborescence instances with a density for which there are no real-world networks available,
and generate networks with adjustable locality and heterogeneity to reveal the effects of these properties on our algorithms
CRS-FL: Conditional Random Sampling for Communication-Efficient and Privacy-Preserving Federated Learning
Federated Learning (FL), a privacy-oriented distributed ML paradigm, is being
gaining great interest in Internet of Things because of its capability to
protect participants data privacy. Studies have been conducted to address
challenges existing in standard FL, including communication efficiency and
privacy-preserving. But they cannot achieve the goal of making a tradeoff
between communication efficiency and model accuracy while guaranteeing privacy.
This paper proposes a Conditional Random Sampling (CRS) method and implements
it into the standard FL settings (CRS-FL) to tackle the above-mentioned
challenges. CRS explores a stochastic coefficient based on Poisson sampling to
achieve a higher probability of obtaining zero-gradient unbiasedly, and then
decreases the communication overhead effectively without model accuracy
degradation. Moreover, we dig out the relaxation Local Differential Privacy
(LDP) guarantee conditions of CRS theoretically. Extensive experiment results
indicate that (1) in communication efficiency, CRS-FL performs better than the
existing methods in metric accuracy per transmission byte without model
accuracy reduction in more than 7% sampling ratio (# sampling size / # model
size); (2) in privacy-preserving, CRS-FL achieves no accuracy reduction
compared with LDP baselines while holding the efficiency, even exceeding them
in model accuracy under more sampling ratio conditions
NetShaper: A Differentially Private Network Side-Channel Mitigation System
The widespread adoption of encryption in network protocols has significantly
improved the overall security of many Internet applications. However, these
protocols cannot prevent network side-channel leaks -- leaks of sensitive
information through the sizes and timing of network packets. We present
NetShaper, a system that mitigates such leaks based on the principle of traffic
shaping. NetShaper's traffic shaping provides differential privacy guarantees
while adapting to the prevailing workload and congestion condition, and allows
configuring a tradeoff between privacy guarantees, bandwidth and latency
overheads. Furthermore, NetShaper provides a modular and portable tunnel
endpoint design that can support diverse applications. We present a
middlebox-based implementation of NetShaper and demonstrate its applicability
in a video streaming and a web service application
Compression with Exact Error Distribution for Federated Learning
Compression schemes have been extensively used in Federated Learning (FL) to
reduce the communication cost of distributed learning. While most approaches
rely on a bounded variance assumption of the noise produced by the compressor,
this paper investigates the use of compression and aggregation schemes that
produce a specific error distribution, e.g., Gaussian or Laplace, on the
aggregated data. We present and analyze different aggregation schemes based on
layered quantizers achieving exact error distribution. We provide different
methods to leverage the proposed compression schemes to obtain
compression-for-free in differential privacy applications. Our general
compression methods can recover and improve standard FL schemes with Gaussian
perturbations such as Langevin dynamics and randomized smoothing
- …