36 research outputs found

    Computing-In-Memory Neural Network Accelerators for Safety-Critical Systems: Can Small Device Variations Be Disastrous?

    Full text link
    Computing-in-Memory (CiM) architectures based on emerging non-volatile memory (NVM) devices have demonstrated great potential for deep neural network (DNN) acceleration thanks to their high energy efficiency. However, NVM devices suffer from various non-idealities, especially device-to-device variations due to fabrication defects and cycle-to-cycle variations due to the stochastic behavior of devices. As such, the DNN weights actually mapped to NVM devices could deviate significantly from the expected values, leading to large performance degradation. To address this issue, most existing works focus on maximizing average performance under device variations. This objective would work well for general-purpose scenarios. But for safety-critical applications, the worst-case performance must also be considered. Unfortunately, this has been rarely explored in the literature. In this work, we formulate the problem of determining the worst-case performance of CiM DNN accelerators under the impact of device variations. We further propose a method to effectively find the specific combination of device variation in the high-dimensional space that leads to the worst-case performance. We find that even with very small device variations, the accuracy of a DNN can drop drastically, causing concerns when deploying CiM accelerators in safety-critical applications. Finally, we show that surprisingly none of the existing methods used to enhance average DNN performance in CiM accelerators are very effective when extended to enhance the worst-case performance, and further research down the road is needed to address this problem

    Towards the Law of Capacity Gap in Distilling Language Models

    Full text link
    Language model (LM) distillation is a trending area that aims to distil the knowledge resided in a large teacher LM to a small student one. While various methods have been proposed to push the distillation to its limits, it is still a pain distilling LMs when a large capacity gap is exhibited between the teacher and the student LMs. The pain is mainly resulted by the curse of capacity gap, which describes that a larger teacher LM cannot always lead to a better student LM than one distilled from a smaller teacher LM due to the affect of capacity gap increment. That is, there is likely an optimal point yielding the best student LM along the scaling course of the teacher LM. Even worse, the curse of capacity gap can be only partly yet not fully lifted as indicated in previous studies. However, the tale is not ever one-sided. Although a larger teacher LM has better performance than a smaller teacher LM, it is much more resource-demanding especially in the context of recent large LMs (LLMs). Consequently, instead of sticking to lifting the curse, leaving the curse as is should be arguably fine. Even better, in this paper, we reveal that the optimal capacity gap is almost consistent across different student scales and architectures, fortunately turning the curse into the law of capacity gap. The law later guides us to distil a 3B student LM (termed MiniMA) from a 7B teacher LM (adapted LLaMA2-7B). MiniMA is demonstrated to yield a new compute-performance pareto frontier among existing 3B LMs on commonly used benchmarks, and its instruction-tuned version (termed MiniChat) outperforms a wide range of 3B competitors in GPT4 evaluation and could even compete with several 7B chat models.Comment: 22 pages, 8 figures, 12 tables, work in progress. Code and checkpoints are available at https://github.com/GeneZC/MiniM

    Circulating Monocytes Act as a Common Trigger for the Calcification Paradox of Osteoporosis and Carotid Atherosclerosis via TGFB1-SP1 and TNFSF10-NFKB1 Axis

    Get PDF
    BackgroundOsteoporosis often occurs with carotid atherosclerosis and causes contradictory calcification across tissue in the same patient, which is called the ‚Äúcalcification paradox‚ÄĚ. Circulating monocytes may be responsible for this unbalanced ectopic calcification. Here, we aimed to show how CD14+ monocytes contribute to the pathophysiology of coexisting postmenopausal osteoporosis and carotid atherosclerosis.MethodsWe comprehensively analyzed osteoporosis data from the mRNA array dataset GSE56814 and the scRNA-seq dataset GSM4423510. Carotid atherosclerosis data were obtained from the GSE23746 mRNA dataset and GSM4705591 scRNA-seq dataset. First, osteoblast and vascular SMC lineages were annotated based on their functional expression using gene set enrichment analysis and AUCell scoring. Next, pseudotime analysis was applied to draw their differentiated trajectory and identify the key gene expression changes in crossroads. Then, ligand‚Äďreceptor interactions between CD14+ monocytes and osteoblast and vascular smooth muscle cell (SMC) lineages were annotated with iTALK. Finally, we selected calcification paradox-related expression in circulating monocytes with LASSO analysis.ResultsFirst, we found a large proportion of delayed premature osteoblasts in osteoporosis and osteogenic SMCs in atherosclerosis. Second, CD14+ monocytes interacted with the intermediate cells of the premature osteoblast and osteogenic SMC lineage by delivering TGFB1 and TNFSF10. This interaction served as a trigger activating the transcription factors (TF) SP1 and NFKB1 to upregulate the inflammatory response and cell senescence and led to a retarded premature state in the osteoblast lineage and osteogenic transition in the SMC lineage. Then, 76.49% of common monocyte markers were upregulated in the circulating monocytes between the two diseases, which were related to chemotaxis and inflammatory responses. Finally, we identified 7 calcification paradox-related genes on circulating monocytes, which were upregulated in aging cells and downregulated in DNA repair cells, indicating that the aging monocytes contributed to the development of the two diseases.ConclusionsOur work provides a perspective for understanding the triggering roles of CD14+ monocytes in the development of the calcification paradox in osteoporosis- and atherosclerosis-related cells based on combined scRNA and mRNA data. This study provided us with an elucidation of the mechanisms underlying the calcification paradox and could help in developing preventive and therapeutic strategies

    DeepDyve: Dynamic Verification for Deep Neural Networks

    Full text link
    Deep neural networks (DNNs) have become one of the enabling technologies in many safety-critical applications, e.g., autonomous driving and medical image analysis. DNN systems, however, suffer from various kinds of threats, such as adversarial example attacks and fault injection attacks. While there are many defense methods proposed against maliciously crafted inputs, solutions against faults presented in the DNN system itself (e.g., parameters and calculations) are far less explored. In this paper, we develop a novel lightweight fault-tolerant solution for DNN-based systems, namely DeepDyve, which employs pre-trained neural networks that are far simpler and smaller than the original DNN for dynamic verification. The key to enabling such lightweight checking is that the smaller neural network only needs to produce approximate results for the initial task without sacrificing fault coverage much. We develop efficient and effective architecture and task exploration techniques to achieve optimized risk/overhead trade-off in DeepDyve. Experimental results show that DeepDyve can reduce 90% of the risks at around 10% overhead

    FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data

    Full text link
    Despite the widespread adoption of face recognition technology around the world, and its remarkable performance on current benchmarks, there are still several challenges that must be covered in more detail. This paper offers an overview of the Face Recognition Challenge in the Era of Synthetic Data (FRCSyn) organized at WACV 2024. This is the first international challenge aiming to explore the use of synthetic data in face recognition to address existing limitations in the technology. Specifically, the FRCSyn Challenge targets concerns related to data privacy issues, demographic biases, generalization to unseen scenarios, and performance limitations in challenging scenarios, including significant age disparities between enrollment and testing, pose variations, and occlusions. The results achieved in the FRCSyn Challenge, together with the proposed benchmark, contribute significantly to the application of synthetic data to improve face recognition technology.Comment: 10 pages, 1 figure, WACV 2024 Workshop