2,537 research outputs found
Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks
We present a novel active learning algorithm, termed as iterative surrogate
model optimization (ISMO), for robust and efficient numerical approximation of
PDE constrained optimization problems. This algorithm is based on deep neural
networks and its key feature is the iterative selection of training data
through a feedback loop between deep neural networks and any underlying
standard optimization algorithm. Under suitable hypotheses, we show that the
resulting optimizers converge exponentially fast (and with exponentially
decaying variance), with respect to increasing number of training samples.
Numerical examples for optimal control, parameter identification and shape
optimization problems for PDEs are provided to validate the proposed theory and
to illustrate that ISMO significantly outperforms a standard deep neural
network based surrogate optimization algorithm
Decision-making with gaussian processes: sampling strategies and monte carlo methods
We study Gaussian processes and their application to decision-making in the real world. We begin by reviewing the foundations of Bayesian decision theory and show how these ideas give rise to methods such as Bayesian optimization. We investigate practical techniques for carrying out these strategies, with an emphasis on estimating and maximizing acquisition functions. Finally, we introduce pathwise approaches to conditioning Gaussian processes and demonstrate key benefits for representing random variables in this manner.Open Acces
(Amplified) Banded Matrix Factorization: A unified approach to private training
Matrix factorization (MF) mechanisms for differential privacy (DP) have
substantially improved the state-of-the-art in privacy-utility-computation
tradeoffs for ML applications in a variety of scenarios, but in both the
centralized and federated settings there remain instances where either MF
cannot be easily applied, or other algorithms provide better tradeoffs
(typically, as becomes small). In this work, we show how MF can
subsume prior state-of-the-art algorithms in both federated and centralized
training settings, across all privacy budgets. The key technique throughout is
the construction of MF mechanisms with banded matrices (lower-triangular
matrices with at most nonzero bands including the main diagonal). For
cross-device federated learning (FL), this enables multiple-participations with
a relaxed device participation schema compatible with practical FL
infrastructure (as demonstrated by a production deployment). In the centralized
setting, we prove that banded matrices enjoy the same privacy amplification
results as the ubiquitous DP-SGD algorithm, but can provide strictly better
performance in most scenarios -- this lets us always at least match DP-SGD, and
often outperform it.Comment: 34 pages, 13 figure
MVG Mechanism: Differential Privacy under Matrix-Valued Query
Differential privacy mechanism design has traditionally been tailored for a
scalar-valued query function. Although many mechanisms such as the Laplace and
Gaussian mechanisms can be extended to a matrix-valued query function by adding
i.i.d. noise to each element of the matrix, this method is often suboptimal as
it forfeits an opportunity to exploit the structural characteristics typically
associated with matrix analysis. To address this challenge, we propose a novel
differential privacy mechanism called the Matrix-Variate Gaussian (MVG)
mechanism, which adds a matrix-valued noise drawn from a matrix-variate
Gaussian distribution, and we rigorously prove that the MVG mechanism preserves
-differential privacy. Furthermore, we introduce the concept
of directional noise made possible by the design of the MVG mechanism.
Directional noise allows the impact of the noise on the utility of the
matrix-valued query function to be moderated. Finally, we experimentally
demonstrate the performance of our mechanism using three matrix-valued queries
on three privacy-sensitive datasets. We find that the MVG mechanism notably
outperforms four previous state-of-the-art approaches, and provides comparable
utility to the non-private baseline.Comment: Appeared in CCS'1
The Geometry of Differential Privacy: the Sparse and Approximate Cases
In this work, we study trade-offs between accuracy and privacy in the context
of linear queries over histograms. This is a rich class of queries that
includes contingency tables and range queries, and has been a focus of a long
line of work. For a set of linear queries over a database , we
seek to find the differentially private mechanism that has the minimum mean
squared error. For pure differential privacy, an approximation to
the optimal mechanism is known. Our first contribution is to give an approximation guarantee for the case of (\eps,\delta)-differential
privacy. Our mechanism is simple, efficient and adds correlated Gaussian noise
to the answers. We prove its approximation guarantee relative to the hereditary
discrepancy lower bound of Muthukrishnan and Nikolov, using tools from convex
geometry.
We next consider this question in the case when the number of queries exceeds
the number of individuals in the database, i.e. when . It is known that better mechanisms exist in this setting. Our second
main contribution is to give an (\eps,\delta)-differentially private
mechanism which is optimal up to a \polylog(d,N) factor for any given query
set and any given upper bound on . This approximation is
achieved by coupling the Gaussian noise addition approach with a linear
regression step. We give an analogous result for the \eps-differential
privacy setting. We also improve on the mean squared error upper bound for
answering counting queries on a database of size by Blum, Ligett, and Roth,
and match the lower bound implied by the work of Dinur and Nissim up to
logarithmic factors.
The connection between hereditary discrepancy and the privacy mechanism
enables us to derive the first polylogarithmic approximation to the hereditary
discrepancy of a matrix
Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning
We introduce new differentially private (DP) mechanisms for gradient-based
machine learning (ML) with multiple passes (epochs) over a dataset,
substantially improving the achievable privacy-utility-computation tradeoffs.
We formalize the problem of DP mechanisms for adaptive streams with multiple
participations and introduce a non-trivial extension of online matrix
factorization DP mechanisms to our setting. This includes establishing the
necessary theory for sensitivity calculations and efficient computation of
optimal matrices. For some applications like SGD steps, applying
these optimal techniques becomes computationally expensive. We thus design an
efficient Fourier-transform-based mechanism with only a minor utility loss.
Extensive empirical evaluation on both example-level DP for image
classification and user-level DP for language modeling demonstrate substantial
improvements over all previous methods, including the widely-used DP-SGD .
Though our primary application is to ML, our main DP results are applicable to
arbitrary linear queries and hence may have much broader applicability.Comment: 9 pages main-text, 3 figures. 40 pages with 13 figures tota
- …