Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy

Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping. We propose an efficient and scalable implementation of this clipping on convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The improvement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost clipping and existing DP training algorithms. Extensive experiments on vision classification tasks, with large ResNet, VGG, and Vision Transformers, demonstrate that DP training with mixed ghost clipping adds $1\sim 10\%$ memory overhead and $<2\times$ slowdown to the standard non-private training. Specifically, when training VGG19 on CIFAR10, the mixed ghost clipping is $3\times$ faster than state-of-the-art Opacus library with $18\times$ larger maximum batch size. To emphasize the significance of efficient DP training on convolutional layers, we achieve 96.7\% accuracy on CIFAR10 and 83.0\% on CIFAR100 at $\epsilon=1$ using BEiT, while the previous best results are 94.8\% and 67.4\%, respectively. We open-source a privacy engine (\url{https://github.com/JialinMao/private_CNN}) that implements DP training of CNN with a few lines of code.Comment: Accepted to NeurIPS 202

IPO7 promotes pancreatic cancer progression via regulating ERBB pathway

Background: Importin 7 (IPO7) belongs to the Importin β family and is implicated in the progression of diverse human malignancies. This work is performed to probe the role of IPO7 in pancreatic cancer development and its potential downstream mechanisms. Methods: IPO7 expression in PC and paracancerous tissues were measured using Immunohistochemistry (IHC) staining and qRT-PCR. Western blotting was utilized to detect the expression level of IPO7 in PC cells and immortalize the pancreatic ductal epithelial cell line. After constructing the IPO7 overexpression and knockdown models, the effect of IPO7 on the proliferation of PC cells was analyzed by the CCK-8 and EdU assay. The migration and invasion of PC cells were examined by wound healing assay and Transwell experiment. The apoptosis rate of PC cells was analyzed by flow cytometry and TUNEL assay. The Gene Set Enrichment Analysis (GSEA) was used to determine the enrichment pathways of IPO7. The effect of IPO7 on the ERBB2 expression was determined using Western blotting. A xenograft mouse model was applied to investigate the carcinogenic effect of IPO7 in vivo. Results: IPO7 expression was remarkably elevated in the cancer tissues of PC patients. IPO7 overexpression remarkably enhanced PC cell proliferation, migration and invasion and suppressed apoptosis, while knockdown of IPO7 exerted the opposite effect. Mechanistically, IPO7 facilitated the malignant phenotype of PC cells by up-regulating ERBB2 expression. In addition, knockdown of IPO7 inhibited tumor growth and lung metastasis in vivo. Conclusion: IPO7 can act as an oncogenic factor and accelerate PC progression by modulating the ERBB pathway

LLMCad: Fast and Scalable On-device Large Language Model Inference

Generative tasks, such as text generation and question answering, hold a crucial position in the realm of mobile applications. Due to their sensitivity to privacy concerns, there is a growing demand for their execution directly on mobile devices. Currently, the execution of these generative tasks heavily depends on Large Language Models (LLMs). Nevertheless, the limited memory capacity of these devices presents a formidable challenge to the scalability of such models. In our research, we introduce LLMCad, an innovative on-device inference engine specifically designed for efficient generative Natural Language Processing (NLP) tasks. The core idea behind LLMCad revolves around model collaboration: a compact LLM, residing in memory, takes charge of generating the most straightforward tokens, while a high-precision LLM steps in to validate these tokens and rectify any identified errors. LLMCad incorporates three novel techniques: (1) Instead of generating candidate tokens in a sequential manner, LLMCad employs the smaller LLM to construct a token tree, encompassing a wider range of plausible token pathways. Subsequently, the larger LLM can efficiently validate all of these pathways simultaneously. (2) It employs a self-adjusting fallback strategy, swiftly initiating the verification process whenever the smaller LLM generates an erroneous token. (3) To ensure a continuous flow of token generation, LLMCad speculatively generates tokens during the verification process by implementing a compute-IO pipeline. Through an extensive series of experiments, LLMCad showcases an impressive token generation speed, achieving rates up to 9.3x faster than existing inference engines

Apomixis for no bacteria-induced thelytoky in Diglyphus wani (Hymenoptera: Eulophidae)

In Hymenoptera species, the reproductive mode is usually arrhenotoky, where haploid males arise from unfertilized eggs and diploid females from fertilized eggs. In addition, a few species reproduce by thelytoky, where diploid females arise from unfertilized eggs. Diploid females can be derived through various cytological mechanisms in thelytokous Hymenoptera species. Hitherto, these mechanisms were revealed mainly in endosymbiont-induced thelytokous Hymenoptera species. In contrast, thelytokous Hymenoptera species in which a reproductive manipulator has not been verified or several common endosymbionts have been excluded were paid less attention in their cytological mechanisms, for instance, Diglyphus wani (Hymenoptera: Eulophidae). Here, we investigated the cytological mechanism of D. wani using cytological methods and genetic markers. Our observations indicated that the diploid karyotypes of two strains of D. wani consist of four pairs of relatively large metacentric chromosomes and one pair of short submetacentric chromosomes (2n = 10). The arrhenotokous strains could complete normal meiosis, whereas the thelytokous strain lacked meiosis and did not expulse any polar bodies. This reproductive type of lacking meiosis is classified as apomictic thelytoky. Moreover, a total of 636 microsatellite sequences were obtained from thelytokous D. wani, dominated by dinucleotide repeats. Genetic markers results showed all three generations of offspring from thelytokous strain maintained the same genotype as their parents. Our results revealed that D. wani is the first eulophid parasitoid wasp in Hymenoptera whose thelytoky was not induced by bacteria to form an apomictic thelytoky. These findings provide a baseline for future inner molecular genetic studies of ameiotic thelytoky

Parameter design oriented analysis of the current control stability of the weak-grid-tied VSC

This paper studies the dynamic behaviors of weak-grid-tied VSCs with simplified transfer functions, which provides an accurate stability analysis and useful indications for tuning system parameters. A reduced-order multi-input multi-output (MIMO) transfer function that contains four single-input single-output (SISO) transfer functions for the weak-grid-tied VSC is first presented. It is found that the four SISO transfer functions share the same equivalent open-loop transfer function, i.e., the same stability conclusion. The Bode plots of the equivalent open-loop transfer function show that the inner current loop behaves as a band-pass filter whose maximum gain is approximately at the frequency of the PLL's bandwidth. By stability criterion, the harmonic amplification and instability occur when its maximum gain exceeds 0dB caused by high PLL's bandwidth, large grid impedance or high active power. It is also found that the target system is less stable when it works as an inverter than as a rectifier, due to the risk of the local positive feedback in the inverter mode. An effective criterion is further proposed to guide the selection of a proper PLL's bandwidth to ensure the stability of the VSC system. Simulation results validate the correctness of the analysis and the efficacy of the criterion