Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent

Each round in Differential Private Stochastic Gradient Descent (DPSGD) transmits a sum of clipped gradients obfuscated with Gaussian noise to a central server which uses this to update a global model which often represents a deep neural network. Since the clipped gradients are computed separately, which we call Individual Clipping (IC), deep neural networks like resnet-18 cannot use Batch Normalization Layers (BNL) which is a crucial component in deep neural networks for achieving a high accuracy. To utilize BNL, we introduce Batch Clipping (BC) where, instead of clipping single gradients as in the orginal DPSGD, we average and clip batches of gradients. Moreover, the model entries of different layers have different sensitivities to the added Gaussian noise. Therefore, Adaptive Layerwise Clipping methods (ALC), where each layer has its own adaptively finetuned clipping constant, have been introduced and studied, but so far without rigorous DP proofs. In this paper, we propose {\em a new ALC and provide rigorous DP proofs for both BC and ALC}. Experiments show that our modified DPSGD with BC and ALC for CIFAR-$10$ with resnet-$18$ converges while DPSGD with IC and ALC does not.Comment: 20 pages, 18 Figure

Catàleg de les molses d'Andorra

El catàleg és una recopilació de totes les dades publicades que fan referència a les molses d'Andorra. S'han revisat tots els exemplars d'herbari assequibles corresponents a les espècies publicades. Aquest treball representa l'estat actual del coneixement de la brioflora (molses) d'Andorra.The catalogue is a compilation of all the data published relating to the species of mosses collected in Andorra. All the available specimens published have been revised. This work represents the current knowledge of the Andorran bryoflora (mosses)

Generalizing DP-SGD with Shuffling and Batch Clipping

Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling. Our DP analysis follows the $f$-DP approach and introduces a new proof technique which allows us to derive simple closed form expressions and to also analyse group privacy. In particular, for $E$ epochs work and groups of size $g$, we show a $\sqrt{g E}$ DP dependency for batch clipping with shuffling.Comment: Update disclaimer

Prc1E and Kif4A control microtubule organization within and between large Xenopus egg asters

© The Author(s), 2018. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Molecular Biology of the Cell 29 (2018): 304-316, doi:10.1091/mbc.E17-09-0540.The cleavage furrow in Xenopus zygotes is positioned by two large microtubule asters that grow out from the poles of the first mitotic spindle. Where these asters meet at the midplane, they assemble a disk-shaped interaction zone consisting of anti-parallel microtubule bundles coated with chromosome passenger complex (CPC) and centralspindlin that instructs the cleavage furrow. Here we investigate the mechanism that keeps the two asters separate and forms a distinct boundary between them, focusing on the conserved cytokinesis midzone proteins Prc1 and Kif4A. Prc1E, the egg orthologue of Prc1, and Kif4A were recruited to anti-parallel bundles at interaction zones between asters in Xenopus egg extracts. Prc1E was required for Kif4A recruitment but not vice versa. Microtubule plus-end growth slowed and terminated preferentially within interaction zones, resulting in a block to interpenetration that depended on both Prc1E and Kif4A. Unexpectedly, Prc1E and Kif4A were also required for radial order of large asters growing in isolation, apparently to compensate for the direction-randomizing influence of nucleation away from centrosomes. We propose that Prc1E and Kif4, together with catastrophe factors, promote “anti-parallel pruning” that enforces radial organization within asters and generates boundaries to microtubule growth between asters

Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes

Hogwild! implements asynchronous Stochastic Gradient Descent (SGD) where multiple threads in parallel access a common repository containing training data, perform SGD iterations and update shared state that represents a jointly learned (global) model. We consider big data analysis where training data is distributed among local data sets in a heterogeneous way -- and we wish to move SGD computations to local compute nodes where local data resides. The results of these local SGD computations are aggregated by a central "aggregator" which mimics Hogwild!. We show how local compute nodes can start choosing small mini-batch sizes which increase to larger ones in order to reduce communication cost (round interaction with the aggregator). We improve state-of-the-art literature and show $O(\sqrt{K}$) communication rounds for heterogeneous data for strongly convex problems, where $K$ is the total number of gradient computations across all local compute nodes. For our scheme, we prove a \textit{tight} and novel non-trivial convergence analysis for strongly convex problems for {\em heterogeneous} data which does not use the bounded gradient assumption as seen in many existing publications. The tightness is a consequence of our proofs for lower and upper bounds of the convergence rate, which show a constant factor difference. We show experimental results for plain convex and non-convex problems for biased (i.e., heterogeneous) and unbiased local data sets.Comment: arXiv admin note: substantial text overlap with arXiv:2007.09208 AISTATS 202

Traditional Float Charges: are They Suited to Stationary Antimony-free Lead Acid Batteries?

International audienc