Generalizing DP-SGD with shuffling and batch clipping

Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling. Our DP analysis follows the f -DP approach and introduces a new proof technique based on a slightly stronger adversarial model which allows us to derive simple closed form expressions and to also analyse group privacy. In particular, for E epochs work and groups of size g, we show a√gE DP dependency for batch clipping with shuffling

Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent

Each round in Differential Private Stochastic Gradient Descent (DPSGD) transmits a sum of clipped gradients obfuscated with Gaussian noise to a central server which uses this to update a global model which often represents a deep neural network. Since the clipped gradients are computed separately, which we call Individual Clipping (IC), deep neural networks like resnet-18 cannot use Batch Normalization Layers (BNL) which is a crucial component in deep neural networks for achieving a high accuracy. To utilize BNL, we introduce Batch Clipping (BC) where, instead of clipping single gradients as in the orginal DPSGD, we average and clip batches of gradients. Moreover, the model entries of different layers have different sensitivities to the added Gaussian noise. Therefore, Adaptive Layerwise Clipping methods (ALC), where each layer has its own adaptively finetuned clipping constant, have been introduced and studied, but so far without rigorous DP proofs. In this paper, we propose {\\em a new ALC and provide rigorous DP proofs for both BC and ALC}. Experiments show that our modified DPSGD with BC and ALC for CIFAR-10 with resnet-18 converges while DPSGD with IC and ALC does not

On the Tightness of the Moment Accountant for DP-SGD

In order to provide differential privacy, Gaussian noise with standard deviation σ is added to local SGD updates after performing a clipping operation in Differential Private SGD (DP-SGD). By non-trivially improving the moment account method we prove a closed form (ϵ, δ)-DP guarantee: DP-SGD is (ϵ ≤ 1/2, δ = 1/N )-DP if σ = p2(ϵ + ln(1/δ))/ϵ with T at least ≈ 2k2/ϵ and (2/e)2k2 − 1/2 ≥ ln(N ), where T is the total number of rounds, and K = kN is the total number of gradient computations where k measures K in number of epochs of size N of the local data set. We prove that our expression is close to tight in that if T is more than a constant factor ≈ 8 smaller than the lower bound ≈ 2k2/ϵ, then the (ϵ, δ)-DP guarantee is violated. Choosing the smallest possible value T ≈ 2k2/ϵ not only leads to a close to tight DP guarantee, but also minimizes the total number of communicated updates and this means that the least amount of noise is aggregated into the global model and in addition accuracy is optimized as confirmed by simulations

Ficolin-2 Levels and FCN2 Haplotypes Influence Hepatitis B Infection Outcome in Vietnamese Patients

Human Ficolin-2 (L-ficolins) encoded by FCN2 gene is a soluble serum protein that plays an important role in innate immunity and is mainly expressed in the liver. Ficolin-2 serum levels and FCN2 single nucleotide polymorphisms were associated to several infectious diseases. We initially screened the complete FCN2 gene in 48 healthy individuals of Vietnamese ethnicity. We genotyped a Vietnamese cohort comprising of 423 clinically classified hepatitis B virus patients and 303 controls for functional single nucleotide polymorphisms in the promoter region (-986G>A, -602G>A, -4A>G) and in exon 8 (+6424G>T) by real-time PCR and investigated the contribution of FCN2 genotypes and haplotypes to serum Ficolin-2 levels, viral load and liver enzyme levels. Haplotypes differed significantly between patients and controls (P = 0.002) and the haplotype AGGG was found frequently in controls in comparison to patients with hepatitis B virus and hepatocellular carcinoma (P = 0.0002 and P<0.0001) conferring a protective effect. Ficolin-2 levels differed significantly between patients and controls (p<0.0001). Patients with acute hepatitis B had higher serum Ficolin-2 levels compared to other patient groups and controls.The viral load was observed to be significantly distributed among the haplotypes (P = 0.04) and the AAAG haplotype contributed to higher Ficolin-2 levels and to viral load. Four novel single nucleotide polymorphisms in introns (-941G>T, -310G>A, +2363G>A, +4882G>A) and one synonymous mutation in exon 8 (+6485G>T) was observed. Strong linkage was found between the variant -986G>A and -4A>G. The very first study on Vietnamese cohort associates both Ficolin-2 serum levels and FCN2 haplotypes to hepatitis B virus infection and subsequent disease progression

Efficacy against pneumococcal carriage and the immunogenicity of reduced-dose (0 + 1 and 1 + 1) PCV10 and PCV13 schedules in Ho Chi Minh City, Viet Nam: a parallel, single-blind, randomised controlled trial

Background Interest in reduced-dose pneumococcal conjugate vaccine (PCV) schedules is growing, but data on their ability to provide direct and indirect protection are scarce. We evaluated 1 + 1 (at 2 months and 12 months) and 0 + 1 (at 12 months) schedules of PCV10 or PCV13 in a predominately unvaccinated population. Methods In this parallel, single-blind, randomised controlled trial, healthy infants aged 2 months were recruited from birth records in three districts in Ho Chi Minh City, Vietnam, and assigned (4:4:4:4:9) to one of five groups: PCV10 at 12 months of age (0 + 1 PCV10), PCV13 at 12 months of age (0 + 1 PCV13), PCV10 at 2 months and 12 months of age (1 + 1 PCV10), PCV13 at 2 months and 12 months of age (1 + 1 PCV13), and unvaccinated control. Outcome assessors were masked to group allocation, and the infants' caregivers and those administering vaccines were not. Nasopharyngeal swabs collected at 6 months, 12 months, 18 months, and 24 months were analysed for pneumococcal carriage. Blood samples collected from a subset of participants (200 per group) at various timepoints were analysed by ELISA and opsonophagocytic assay. The primary outcome was the efficacy of each schedule against vaccine-type carriage at 24 months, analysed by intention to treat for all those with a nasopharyngeal swab available. This trial is registered at ClinicalTrials.gov, NCT03098628. Findings 2501 infants were enrolled between March 8, 2017, and July 24, 2018 and randomly assigned to study groups (400 to 0 + 1 PCV10, 400 to 0 + 1 PCV13, 402 to 1 + 1 PCV10, 401 to 1 + 1 PCV13, and 898 to control). Analysis of the primary endpoint included 341 participants for 0 + 1 PCV10, 356 0 + 1 PCV13, 358 1 + 1 PCV10, 350 1 + 1 PCV13, and 758 control. At 24 months, a 1 + 1 PCV10 schedule reduced PCV10-type carriage by 58% (95% CI 25 to 77), a 1 + 1 PCV13 schedule reduced PCV13-type carriage by 65% (42 to 79), a 0 + 1 PCV10 schedule reduced PCV10-type carriage by 53% (17 to 73), and a 0 + 1 PCV13 schedule non-significantly reduced PCV13-type carriage by 25% (–7 to 48) compared with the unvaccinated control group. Reactogenicity and serious adverse events were similar across groups. Interpretation A 1 + 1 PCV schedule greatly reduces vaccine-type carriage and is likely to generate substantial herd protection and provide some degree of individual protection during the first year of life. Such a schedule is suitable for mature PCV programmes or for introduction in conjunction with a comprehensive catch-up campaign, and potentially could be most effective given as a mixed regimen (PCV10 then PCV13). A 0 + 1 PCV schedule has some effect on carriage along with a reasonable immune response and could be considered for use in humanitarian crises or remote settings. Funding Bill & Melinda Gates Foundation. Translation For the Vietnamese translation of the abstract see Supplementary Materials section