Boosting Adversarial Robustness via Neural Architecture Search and Design

Adversarial robustness in Deep Neural Networks (DNNs) is a critical and emerging field of research that addresses the vulnerability of DNNs to subtle, intentionally crafted perturbations in their input data. These perturbations, often imperceptible to the human eye, can lead to significant error increment in the network's predictions, while they can be easily derived via adversarial attacks in various data formats, such as image, text, and audio. This susceptibility poses serious security and trustworthy concerns in real-world applications such as autonomous driving, healthcare diagnostics, and cybersecurity. To enhance the trustworthiness of DNNs, lots of research efforts have been put into developing techniques that aim to improve DNNs ability to defend against such adversarial attacks, ensuring that trustworthy results can be provided in real-world scenarios. The main stream of adversarial robustness lies in the adversarial training strategies and regularizations. However, less attention has been paid to the DNN itself. Little is known about the influence of different neural network architectures or designs on adversarial robustness. To fulfill this knowledge gap, we propose to advance adversarial robustness via investigating neural architecture search and design in this thesis

Adversarially Robust Neural Architectures

Deep Neural Network (DNN) are vulnerable to adversarial attack. Existing methods are devoted to developing various robust training strategies or regularizations to update the weights of the neural network. But beyond the weights, the overall structure and information flow in the network are explicitly determined by the neural architecture, which remains unexplored. This paper thus aims to improve the adversarial robustness of the network from the architecture perspective with NAS framework. We explore the relationship among adversarial robustness, Lipschitz constant, and architecture parameters and show that an appropriate constraint on architecture parameters could reduce the Lipschitz constant to further improve the robustness. For NAS framework, all the architecture parameters are equally treated when the discrete architecture is sampled from supernet. However, the importance of architecture parameters could vary from operation to operation or connection to connection, which is not explored and might reduce the confidence of robust architecture sampling. Thus, we propose to sample architecture parameters from trainable multivariate log-normal distributions, with which the Lipschitz constant of entire network can be approximated using a univariate log-normal distribution with mean and variance related to architecture parameters. Compared with adversarially trained neural architectures searched by various NAS algorithms as well as efficient human-designed models, our algorithm empirically achieves the best performance among all the models under various attacks on different datasets.Comment: 9 pages, 3 figures, 5 table

Parameter-Saving Adversarial Training: Reinforcing Multi-Perturbation Robustness via Hypernetworks

Adversarial training serves as one of the most popular and effective methods to defend against adversarial perturbations. However, most defense mechanisms only consider a single type of perturbation while various attack methods might be adopted to perform stronger adversarial attacks against the deployed model in real-world scenarios, e.g., $\ell_2$ or $\ell_\infty$. Defending against various attacks can be a challenging problem since multi-perturbation adversarial training and its variants only achieve suboptimal robustness trade-offs, due to the theoretical limit to multi-perturbation robustness for a single model. Besides, it is impractical to deploy large models in some storage-efficient scenarios. To settle down these drawbacks, in this paper we propose a novel multi-perturbation adversarial training framework, parameter-saving adversarial training (PSAT), to reinforce multi-perturbation robustness with an advantageous side effect of saving parameters, which leverages hypernetworks to train specialized models against a single perturbation and aggregate these specialized models to defend against multiple perturbations. Eventually, we extensively evaluate and compare our proposed method with state-of-the-art single/multi-perturbation robust methods against various latest attack methods on different datasets, showing the robustness superiority and parameter efficiency of our proposed method, e.g., for the CIFAR-10 dataset with ResNet-50 as the backbone, PSAT saves approximately 80\% of parameters with achieving the state-of-the-art robustness trade-off accuracy.Comment: 9 pages, 2 figure

Modelling Skeleton-based Human Dynamics via Retrospection

Human motion prediction is one of the key problems in computer vision and robotic vision and has received increasing attention in recent years. The target is to generate the future continuous, realistic human poses given a seed sequence, which can further assist human motion analysis. However, due to the high-uncertainty, it is difficult and challenging to model human dynamics which not only requires spatial information including complicated joint correlations, but also temporal information including periodic properties. Recently, deep recurrent neural networks (RNNs) have achieved impressive success in forecasting human motion with a sequence-to-sequence architecture. However, forecasting in longer time horizons often leads to implausible human poses or converges to mean poses, because of error accumulation and difficulties in keeping track of longer-term information. Based on these observations, in this study, we propose to retrospect human dynamics with attention. A retrospection module is designed upon RNN to regularly retrospect past frames and correct mistakes in time. This significantly improves the memory of RNN and provides sufficient information for the decoder networks to generate longer-term predictions. Moreover, we present a spatial attention module to explore cooperation among joints in performing a particular motion as well as a temporal attention module to exploit the level of importance among observed frames. Residual connections are also included to guarantee the performance of short-term prediction. We evaluate the proposed algorithm on the largest and most challenging Human 3.6M dataset in the field. Experimental results demonstrate the necessity of investigating motion prediction in a self-audit manner and the effectiveness of the proposed algorithm in both short-term and long-term predictions

Neural Architecture Retrieval

With the increasing number of new neural architecture designs and substantial existing neural architectures, it becomes difficult for the researchers to situate their contributions compared with existing neural architectures or establish the connections between their designs and other relevant ones. To discover similar neural architectures in an efficient and automatic manner, we define a new problem Neural Architecture Retrieval which retrieves a set of existing neural architectures which have similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due to the graph size and motifs. To fulfill this potential, we propose to divide the graph into motifs which are used to rebuild the macro graph to tackle these issues, and introduce multi-level contrastive learning to achieve accurate graph representation learning. Extensive evaluations on both human-designed and synthesized neural architectures demonstrate the superiority of our algorithm. Such a dataset which contains 12k real-world network architectures, as well as their embedding, is built for neural architecture retrieval.Comment: ICLR 202

Qidonghuoxue Decoction Ameliorates Pulmonary Edema in Acute Lung Injury Mice through the Upregulation of Epithelial Sodium Channel and Aquaporin-1

QDHX decoction is an effective traditional Chinese medicine that has been used to treat ALI, a disease characterized by pulmonary edema and inflammation. In this study, the aim is to elucidate the molecular mechanisms of QDHX decoction on improving the alveolar-capillary membrane permeability and alleviating inflammatory response. The BALB/c mice were divided into five groups including the control group, ALI group, ALI + low-dose QDHX decoction, ALI + high-dose QDHX decoction, and ALI + dexamethasone. When the animals were sacrificed, the pathology and wet/dry of lung tissue were tested and confirmed Ali model, the LDH and nucleated cells in BALF, and TNF-α and IL-1β in serum; α-ENaC and AQP-1 in lung tissue were examined. In the results, QDHX decoction downregulated the cytokine such as TNF-α and IL-1β, reduced the nucleated cells, and some biochemical parameters of the BALF. It also ameliorated the ENaC-α and AQP-1 expression induced by LPS in primary epithelial cells. These findings may provide new insights into the application of QDHX decoction for the prevention and treatment of LPS-related ALI

Adversarial Recurrent Time Series Imputation

For the real-world time series analysis, data missing is a ubiquitously existing problem due to anomalies during data collecting and storage. If not treated properly, this problem will seriously hinder the classification, regression or related tasks. Existing methods for time series imputation either impose too strong assumptions on the distribution of missing data, or cannot fully exploit, even simply ignore the informative temporal dependencies and feature correlations across different time steps. In this paper, inspired by the idea of conditional generative adversarial networks, we propose a generative adversarial learning framework for time series imputation under the condition of observed data (as well as the labels, if possible). In our model, we employ a modified bidirectional RNN structure as the generator G, which is aimed at generating the missing values by taking advantage of the temporal and non-temporal information extracted from the observed time series. The discriminator D is designed to distinguish whether each value in a time series is generated or not, so that it can help the generator to make an adjustment towards a more authentic imputation result. For an empirical verification of our model, we conduct imputation and classification experiments on several real-world time series datasets. The experimental results show an eminent improvement compared with state-of-the-art baseline models

A Benchmark Study on Calibration

Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data preprocessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS. The project page can be found at https://www.taolinwei.com/calibration-studyComment: ICLR 2024 poste

Wenshen Yiqi Keli Mitigates the Proliferation and Migration of Cigarette Smoke Extract-Induced Human Airway Smooth Muscle Cells through miR-155/FoxO3a Axis

Some domestic scholars revealed the effectiveness of Wenshen Yiqi Keli (WSYQKL) on chronic obstructive pulmonary disease (COPD). However, the exact mechanism of WSYQKL on COPD is fuzzy and needs further research. We adopted UPLC-Q/TOF-MS to analyze the chemical components of WSYQKL. In in vitro experiments, human airway smooth muscle cells (hASMCs) were intervened with 2.5% cigarette smoke extract (CSE), medicine serum of WSYQKL, miR-155 mimic, and FoxO3a silencing. Cell viability, proliferation, migration, and the expressions of miR-155, PCNA, Ki67, p21, p27, and FoxO3a were examined by cell counting kit-8, EdU staining, Transwell assay, scarification assay, qRT-PCR, immunol cytochemistry, and western blot, respectively. The association between miR-155 and FoxO3a was assessed by database and luciferase reporter gene analysis. We identified 47 kinds of chemical compositions of WSYQKL in ESI+ mode and 42 kinds of components of WSYQKL in ESI− mode. The medicine serum of WSYQKL strongly alleviated the proliferation and migration of hASMCs induced by CSE in a concentration-dependent manner. The medicine serum of WSYQKL enhanced the levels of p21, p27, and FoxO3a and weakened PCNA and Ki67 levels in hASMCs induced by CSE with the increase of concentration. MiR-155 mimic or FoxO3a silencing notably advanced CSE-treated HASMC viability, proliferation, migration, and the levels of PCNA and Ki67 and downregulated the levels of p21, p27, and FoxO3a in CSE-triggered hASMCs, which was reversed by WSYQKL-containing serum. Our results described that WSYQKL alleviated the proliferation and migration of hASMCs induced by CSE by modulating the miR-155/FoxO3a axis

Comparison of Ultrasound-Guided Fine-Needle Cytology Quality in Thyroid Nodules with 22-, 23-, and 25-Gauge Needles

Objective. To compare the cytology quality of ultrasound-guided fine-needle biopsy in thyroid nodules with 22-, 23-, and 25-gauge (G) needles prospectively. Methods. A total of 240 consecutive nodules underwent ultrasound-guided fine-needle aspiration (USG-FNA) and 240 nodules underwent ultrasound-guided fine-needle capillary (USG-FNC) were included in this prospective study from October 2014 to February 2016. Each nodule was sampled using 22 G, 23 G, and 25 G needle according to designed orders, and 1240 smears were finally obtained. Cytology quality was scored by a cytologist blinded to needle selection. Results. In USG-FNA, the average scores and standard deviations were 5.50±2.87 for 25 G needles, 4.82±2.95 for 23 G needles, and 5.19±2.81 for 22 G needles. In USG-FNC, the average scores and standard deviations of each group were 5.12±2.69 for 25 G, 4.60±2.90 for 23 G, and 4.90±2.90 for 22 G needles. The specimen quality scores of 25 G group were significantly higher than that of 23 G group (P0.017 for all). Conclusions. 25 G needles obtained the highest scores of sample quality in thyroid FNA and FNC comparing with 22 G and 23 G needles. 25 G needle should be first choice of thyroid FNA and FNC in routine work