20 research outputs found

    Hypocoercivity properties of adaptive Langevin dynamics

    Get PDF
    International audienceAdaptive Langevin dynamics is a method for sampling the Boltzmann-Gibbs distribution at prescribed temperature in cases where the potential gradient is subject to stochastic perturbation of unknown magnitude. The method replaces the friction in underdamped Langevin dynamics with a dynamical variable, updated according to a negative feedback loop control law as in the Nose-Hoover thermostat. Using a hypocoercivity analysis we show that the law of Adaptive Langevin dynamics converges exponentially rapidly to the stationary distribution, with a rate that can be quantified in terms of the key parameters of the dynamics. This allows us in particular to obtain a central limit theorem with respect to the time averages computed along a stochastic path. Our theoretical findings are illustrated by numerical simulations involving classification of the MNIST data set of handwritten digits using Bayesian logistic regression

    Affine Invariant Covariance Estimation for Heavy-Tailed Distributions

    Get PDF
    In this work we provide an estimator for the covariance matrix of a heavy-tailed multivariate distributionWe prove that the proposed estimator S^\widehat{\mathbf{S}} admits an \textit{affine-invariant} bound of the form (1ε)SS^(1+ε)S(1-\varepsilon) \mathbf{S} \preccurlyeq \widehat{\mathbf{S}} \preccurlyeq (1+\varepsilon) \mathbf{S}in high probability, where S\mathbf{S} is the unknown covariance matrix, and \preccurlyeq is the positive semidefinite order on symmetric matrices. The result only requires the existence of fourth-order moments, and allows for ε=O(κ4dlog(d/δ)/n)\varepsilon = O(\sqrt{\kappa^4 d\log(d/\delta)/n}) where κ4\kappa^4 is a measure of kurtosis of the distribution, dd is the dimensionality of the space, nn is the sample size, and 1δ1-\delta is the desired confidence level. More generally, we can allow for regularization with level λ\lambda, then dd gets replaced with the degrees of freedom number. Denoting cond(S)\text{cond}(\mathbf{S}) the condition number of S\mathbf{S}, the computational cost of the novel estimator is O(d2n+d3log(cond(S)))O(d^2 n + d^3\log(\text{cond}(\mathbf{S}))), which is comparable to the cost of the sample covariance estimator in the statistically interesing regime ndn \ge d. We consider applications of our estimator to eigenvalue estimation with relative error, and to ridge regression with heavy-tailed random design

    S-GBDT: Frugal Differentially Private Gradient Boosting Decision Trees

    Full text link
    Privacy-preserving learning of gradient boosting decision trees (GBDT) has the potential for strong utility-privacy tradeoffs for tabular data, such as census data or medical meta data: classical GBDT learners can extract non-linear patterns from small sized datasets. The state-of-the-art notion for provable privacy-properties is differential privacy, which requires that the impact of single data points is limited and deniable. We introduce a novel differentially private GBDT learner and utilize four main techniques to improve the utility-privacy tradeoff. (1) We use an improved noise scaling approach with tighter accounting of privacy leakage of a decision tree leaf compared to prior work, resulting in noise that in expectation scales with O(1/n)O(1/n), for nn data points. (2) We integrate individual R\'enyi filters to our method to learn from data points that have been underutilized during an iterative training process, which -- potentially of independent interest -- results in a natural yet effective insight to learning streams of non-i.i.d. data. (3) We incorporate the concept of random decision tree splits to concentrate privacy budget on learning leaves. (4) We deploy subsampling for privacy amplification. Our evaluation shows for the Abalone dataset (<4k<4k training data points) a R2R^2-score of 0.390.39 for ε=0.15\varepsilon=0.15, which the closest prior work only achieved for ε=10.0\varepsilon=10.0. On the Adult dataset (50k50k training data points) we achieve test error of 18.7%18.7\,\% for ε=0.07\varepsilon=0.07 which the closest prior work only achieved for ε=1.0\varepsilon=1.0. For the Abalone dataset for ε=0.54\varepsilon=0.54 we achieve R2R^2-score of 0.470.47 which is very close to the R2R^2-score of 0.540.54 for the nonprivate version of GBDT. For the Adult dataset for ε=0.54\varepsilon=0.54 we achieve test error 17.1%17.1\,\% which is very close to the test error 13.7%13.7\,\% of the nonprivate version of GBDT.Comment: The first two authors equally contributed to this wor
    corecore