STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

Distributed parallel stochastic gradient descent algorithms are workhorses for large scale machine learning tasks. Among them, local stochastic gradient descent (Local SGD) has attracted significant attention due to its low communication complexity. Previous studies prove that the communication complexity of Local SGD with a fixed or an adaptive communication period is in the order of $O (N^{\frac{3}{2}} T^{\frac{1}{2}})$ and $O (N^{\frac{3}{4}} T^{\frac{3}{4}})$ when the data distributions on clients are identical (IID) or otherwise (Non-IID), where $N$ is the number of clients and $T$ is the number of iterations. In this paper, to accelerate the convergence by reducing the communication complexity, we propose \textit{ST}agewise \textit{L}ocal \textit{SGD} (STL-SGD), which increases the communication period gradually along with decreasing learning rate. We prove that STL-SGD can keep the same convergence rate and linear speedup as mini-batch SGD. In addition, as the benefit of increasing the communication period, when the objective is strongly convex or satisfies the Polyak-\L ojasiewicz condition, the communication complexity of STL-SGD is $O (N \log{T})$ and $O (N^{\frac{1}{2}} T^{\frac{1}{2}})$ for the IID case and the Non-IID case respectively, achieving significant improvements over Local SGD. Experiments on both convex and non-convex problems demonstrate the superior performance of STL-SGD.Comment: Accepted by AAAI202

Differentially Private Learning with Per-Sample Adaptive Clipping

Privacy in AI remains a topic that draws attention from researchers and the general public in recent years. As one way to implement privacy-preserving AI, differentially private learning is a framework that enables AI models to use differential privacy (DP). To achieve DP in the learning process, existing algorithms typically limit the magnitude of gradients with a constant clipping, which requires carefully tuned due to its significant impact on model performance. As a solution to this issue, latest works NSGD and Auto-S innovatively propose to use normalization instead of clipping to avoid hyperparameter tuning. However, normalization-based approaches like NSGD and Auto-S rely on a monotonic weight function, which imposes excessive weight on small gradient samples and introduces extra deviation to the update. In this paper, we propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function, which guarantees privacy without the typical hyperparameter tuning process of using a constant clipping while significantly reducing the deviation between the update and true batch-averaged gradient. We provide a rigorous theoretical convergence analysis and show that with convergence rate at the same order, the proposed algorithm achieves a lower non-vanishing bound, which is maintained over training iterations, compared with NSGD/Auto-S. In addition, through extensive experimental evaluation, we show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.Comment: To appear in AAAI 2023, Revised acknowledgments and citation

Development of polymorphic EST-SSR markers and characterization of the autotetraploid genome of sainfoin (Onobrychis viciifolia)

Background Sainfoin (Onobrychis viciifolia) is a highly nutritious, tannin-containing, and tetraploid forage legume. Due to the lack of detailed transcriptomic and genomic information on this species, genetic and breeding projects for sainfoin improvement have been significantly hindered. Methods In this study, a total of 24,630,711 clean reads were generated from 14 different sainfoin tissues using Illumina paired-end sequencing technology and deposited in the NCBI SRA database (SRX3763386). From these clean reads, 77,764 unigene sequences were obtained and 6,752 EST-SSRs were identified using de novo assembly. A total of 2,469 primer pairs were designed, and 200 primer pairs were randomly selected to analyze the polymorphism in five sainfoin wild accessions. Results Further analysis of 40 sainfoin individuals from the five wild populations using 61 EST-SSR loci showed that the number of alleles per locus ranged from 4 to 15, and the expected heterozygosity varied from 0.55 to 0.91. Additionally, by counting the EST-SSR band number and sequencing the three or four bands in one sainfoin individual, sainfoin was confirmed to be autotetraploid. This finding provides a high level of information about this plant. Discussion Through this study, 61 EST-SSR markers were successfully developed and shown to be useful for genetic studies and investigations of population genetic structures and variabilities among different sainfoin accessions

Enhancer trapping and annotation in Zebrafish mediated with Sleeping Beauty, piggyBac and Tol2 transposons

Although transposon-mediated enhancer trapping (ET) is successfully applied in diverse models, the efficiency of various transposon systems varies significantly, and little information is available regarding efficiency of enhancer trapping by various transposons in zebrafish. Most potential enhancers (Ens) still lack evidence of actual En activity. Here, we compared the differences in ET efficiency between sleeping beauty (SB), piggyBac (PB) and Tol2 transposons. Tol2 represented the highest germline transfer efficiencies at 55.56% (NF0 = 165), followed by SB (38.36%, NF0 = 151) and PB (32.65%, NF0 = 149). ET lines generated by the Tol2 transposon tended to produce offspring with a single expression pattern per line, while PB and SB tended to generate embryos with multiple expression patterns. In our tests, 10 putative Ens (En1⁻10) were identified by splinkerette PCR and comparative genomic analysis. Combining the GFP expression profiles and mRNA expression patterns revealed that En1 and En2 may be involved in regulation of the expression of dlx1a and dlx2a, while En6 may be involved in regulation of the expression of line TK4 transgene and rps26, and En7 may be involved in the regulation of the expression of wnt1 and wnt10b. Most identified Ens were found to be transcribed in zebrafish embryos, and their regulatory function may involve eRNAs

Design an Indoor Air Quality Controller Based on LPC2478

Indoor air quality is very important to our lives, because we spend most of our time indoor. In order to improve the air quality of indoor, this paper designs an indoor environment quality monitoring and controlling system based on ARM microcontroller LPC2478. It will do a real-time monitoring work for detecting the indoor environmental factors and comprehensively evaluate its air quality level. While the indoor air quality status is "poor", this intelligent system will automatically start the heat exchange ventilator for indoor environmental quality improvement. The results compared to traditional natural ventilation method show the better performance of proposed system

Depth Annotations: Designing Depth of a Single Image for Depth-based Effects

We present a novel pipeline to generate a depth map from a single image that can be used as input for a variety of artistic depth-based effects. In such a context, the depth maps do not have to be perfect but are rather designed with respect to a desired result. Consequently, our solution centers around user interaction and relies on a scribble-based depth editing. The annotations can be sparse, as the depth map is generated by a diffusion process, which is guided by image features. We support a variety of controls, such as a non-linear depth mapping, a steering mechanism for the diffusion (e.g., directionality, emphasis, or reduction of the influence of image cues), and besides absolute, we also support relative depth indications. In case that a depth estimate is available from an automatic solution, we illustrate how this information can be integrated in form of a depth palette, that allows the user to transfer depth values via a painting metaphor. We demonstrate a variety of artistic 3D results, including wiggle stereoscopy, artistic abstractions, haze, unsharp masking, and depth of field.Comp Graphics & Visualisatio

Differentially Private Learning with Per-Sample Adaptive Clipping

Privacy in AI remains a topic that draws attention from researchers and the general public in recent years. As one way to implement privacy-preserving AI, differentially private learning is a framework that enables AI models to use differential privacy (DP). To achieve DP in the learning process, existing algorithms typically limit the magnitude of gradients with a constant clipping, which requires carefully tuned due to its significant impact on model performance. As a solution to this issue, latest works NSGD and Auto-S innovatively propose to use normalization instead of clipping to avoid hyperparameter tuning. However, normalization-based approaches like NSGD and Auto-S rely on a monotonic weight function, which imposes excessive weight on small gradient samples and introduces extra deviation to the update. In this paper, we propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function, which guarantees privacy without the typical hyperparameter tuning process of using a constant clipping while significantly reducing the deviation between the update and true batch-averaged gradient. We provide a rigorous theoretical convergence analysis and show that with convergence rate at the same order, the proposed algorithm achieves a lower non-vanishing bound, which is maintained over training iterations, compared with NSGD/Auto-S. In addition, through extensive experimental evaluation, we show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks