5,043 research outputs found
Differentially Private Empirical Risk Minimization with Sparsity-Inducing Norms
Differential privacy is concerned about the prediction quality while
measuring the privacy impact on individuals whose information is contained in
the data. We consider differentially private risk minimization problems with
regularizers that induce structured sparsity. These regularizers are known to
be convex but they are often non-differentiable. We analyze the standard
differentially private algorithms, such as output perturbation, Frank-Wolfe and
objective perturbation. Output perturbation is a differentially private
algorithm that is known to perform well for minimizing risks that are strongly
convex. Previous works have derived excess risk bounds that are independent of
the dimensionality. In this paper, we assume a particular class of convex but
non-smooth regularizers that induce structured sparsity and loss functions for
generalized linear models. We also consider differentially private Frank-Wolfe
algorithms to optimize the dual of the risk minimization problem. We derive
excess risk bounds for both these algorithms. Both the bounds depend on the
Gaussian width of the unit ball of the dual norm. We also show that objective
perturbation of the risk minimization problems is equivalent to the output
perturbation of a dual optimization problem. This is the first work that
analyzes the dual optimization problems of risk minimization problems in the
context of differential privacy
Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning
In this paper, we focus on developing a novel mechanism to preserve
differential privacy in deep neural networks, such that: (1) The privacy budget
consumption is totally independent of the number of training steps; (2) It has
the ability to adaptively inject noise into features based on the contribution
of each to the output; and (3) It could be applied in a variety of different
deep neural networks. To achieve this, we figure out a way to perturb affine
transformations of neurons, and loss functions used in deep neural networks. In
addition, our mechanism intentionally adds "more noise" into features which are
"less relevant" to the model output, and vice-versa. Our theoretical analysis
further derives the sensitivities and error bounds of our mechanism. Rigorous
experiments conducted on MNIST and CIFAR-10 datasets show that our mechanism is
highly effective and outperforms existing solutions.Comment: IEEE ICDM 2017 - regular pape
Efficient Private ERM for Smooth Objectives
In this paper, we consider efficient differentially private empirical risk
minimization from the viewpoint of optimization algorithms. For strongly convex
and smooth objectives, we prove that gradient descent with output perturbation
not only achieves nearly optimal utility, but also significantly improves the
running time of previous state-of-the-art private optimization algorithms, for
both -DP and -DP. For non-convex but smooth
objectives, we propose an RRPSGD (Random Round Private Stochastic Gradient
Descent) algorithm, which provably converges to a stationary point with privacy
guarantee. Besides the expected utility bounds, we also provide guarantees in
high probability form. Experiments demonstrate that our algorithm consistently
outperforms existing method in both utility and running time
- …