5,354 research outputs found
Differentially private partitioned variational inference
Learning a privacy-preserving model from sensitive data which are distributed
across multiple devices is an increasingly important problem. The problem is
often formulated in the federated learning context, with the aim of learning a
single global model while keeping the data distributed. Moreover, Bayesian
learning is a popular approach for modelling, since it naturally supports
reliable uncertainty estimates. However, Bayesian learning is generally
intractable even with centralised non-private data and so approximation
techniques such as variational inference are a necessity. Variational inference
has recently been extended to the non-private federated learning setting via
the partitioned variational inference algorithm. For privacy protection, the
current gold standard is called differential privacy. Differential privacy
guarantees privacy in a strong, mathematically clearly defined sense.
In this paper, we present differentially private partitioned variational
inference, the first general framework for learning a variational approximation
to a Bayesian posterior distribution in the federated learning setting while
minimising the number of communication rounds and providing differential
privacy guarantees for data subjects.
We propose three alternative implementations in the general framework, one
based on perturbing local optimisation runs done by individual parties, and two
based on perturbing updates to the global model (one using a version of
federated averaging, the second one adding virtual parties to the protocol),
and compare their properties both theoretically and empirically.Comment: Published in TMLR 04/2023: https://openreview.net/forum?id=55Bcghgic
Data Analytics with Differential Privacy
Differential privacy is the state-of-the-art definition for privacy,
guaranteeing that any analysis performed on a sensitive dataset leaks no
information about the individuals whose data are contained therein. In this
thesis, we develop differentially private algorithms to analyze distributed and
streaming data. In the distributed model, we consider the particular problem of
learning -- in a distributed fashion -- a global model of the data, that can
subsequently be used for arbitrary analyses. We build upon PrivBayes, a
differentially private method that approximates the high-dimensional
distribution of a centralized dataset as a product of low-order distributions,
utilizing a Bayesian Network model. We examine three novel approaches to
learning a global Bayesian Network from distributed data, while offering the
differential privacy guarantee to all local datasets. Our work includes a
detailed theoretical analysis of the distributed, differentially private
entropy estimator which we use in one of our algorithms, as well as a detailed
experimental evaluation, using both synthetic and real-world data. In the
streaming model, we focus on the problem of estimating the density of a stream
of users, which expresses the fraction of all users that actually appear in the
stream. We offer one of the strongest privacy guarantees for the streaming
model, user-level pan-privacy, which ensures that the privacy of any user is
protected, even against an adversary that observes the internal state of the
algorithm. We provide a detailed analysis of an existing, sampling-based
algorithm for the problem and propose two novel modifications that
significantly improve it, both theoretically and experimentally, by optimally
using all the allocated "privacy budget."Comment: Diploma Thesis, School of Electrical and Computer Engineering,
Technical University of Crete, Chania, Greece, 201
Fast Differentially Private Matrix Factorization
Differentially private collaborative filtering is a challenging task, both in
terms of accuracy and speed. We present a simple algorithm that is provably
differentially private, while offering good performance, using a novel
connection of differential privacy to Bayesian posterior sampling via
Stochastic Gradient Langevin Dynamics. Due to its simplicity the algorithm
lends itself to efficient implementation. By careful systems design and by
exploiting the power law behavior of the data to maximize CPU cache bandwidth
we are able to generate 1024 dimensional models at a rate of 8.5 million
recommendations per second on a single PC
- …