136 research outputs found
Backdoor Learning for NLP: Recent Advances, Challenges, and Future Research Directions
Although backdoor learning is an active research topic in the NLP domain, the
literature lacks studies that systematically categorize and summarize backdoor
attacks and defenses. To bridge the gap, we present a comprehensive and
unifying study of backdoor learning for NLP by summarizing the literature in a
systematic manner. We first present and motivate the importance of backdoor
learning for building robust NLP systems. Next, we provide a thorough account
of backdoor attack techniques, their applications, defenses against backdoor
attacks, and various mitigation techniques to remove backdoor attacks. We then
provide a detailed review and analysis of evaluation metrics, benchmark
datasets, threat models, and challenges related to backdoor learning in NLP.
Ultimately, our work aims to crystallize and contextualize the landscape of
existing literature in backdoor learning for the text domain and motivate
further research in the field. To this end, we identify troubling gaps in the
literature and offer insights and ideas into open challenges and future
research directions. Finally, we provide a GitHub repository with a list of
backdoor learning papers that will be continuously updated at
https://github.com/marwanomar1/Backdoor-Learning-for-NLP
Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
The success of machine learning is fueled by the increasing availability of computing power and large training datasets. The training data is used to learn new models or update existing ones, assuming that it is sufficiently representative of the data that will be encountered at test time. This assumption is challenged by the threat of poisoning, an attack that manipulates the training data to compromise the model's performance at test time. Although poisoning has been acknowledged as a relevant threat in industry applications, and a variety of different attacks and defenses have been proposed so far, a complete systematization and critical review of the field is still missing. In this survey, we provide a comprehensive systematization of poisoning attacks and defenses in machine learning, reviewing more than 100 papers published in the field in the last 15 years. We start by categorizing the current threat models and attacks, and then organize existing defenses accordingly. While we focus mostly on computer-vision applications, we argue that our systematization also encompasses state-of-the-art attacks and defenses for other data modalities. Finally, we discuss existing resources for research in poisoning, and shed light on the current limitations and open research questions in this research field
You Can Backdoor Personalized Federated Learning
Existing research primarily focuses on backdoor attacks and defenses within
the generic federated learning scenario, where all clients collaborate to train
a single global model. A recent study conducted by Qin et al. (2023) marks the
initial exploration of backdoor attacks within the personalized federated
learning (pFL) scenario, where each client constructs a personalized model
based on its local data. Notably, the study demonstrates that pFL methods with
\textit{parameter decoupling} can significantly enhance robustness against
backdoor attacks. However, in this paper, we whistleblow that pFL methods with
parameter decoupling are still vulnerable to backdoor attacks. The resistance
of pFL methods with parameter decoupling is attributed to the heterogeneous
classifiers between malicious clients and benign counterparts. We analyze two
direct causes of the heterogeneous classifiers: (1) data heterogeneity
inherently exists among clients and (2) poisoning by malicious clients further
exacerbates the data heterogeneity. To address these issues, we propose a
two-pronged attack method, BapFL, which comprises two simple yet effective
strategies: (1) poisoning only the feature encoder while keeping the classifier
fixed and (2) diversifying the classifier through noise introduction to
simulate that of the benign clients. Extensive experiments on three benchmark
datasets under varying conditions demonstrate the effectiveness of our proposed
attack. Additionally, we evaluate the effectiveness of six widely used defense
methods and find that BapFL still poses a significant threat even in the
presence of the best defense, Multi-Krum. We hope to inspire further research
on attack and defense strategies in pFL scenarios. The code is available at:
https://github.com/BapFL/code.Comment: Submitted to TKD
Edge Learning for 6G-enabled Internet of Things: A Comprehensive Survey of Vulnerabilities, Datasets, and Defenses
The ongoing deployment of the fifth generation (5G) wireless networks
constantly reveals limitations concerning its original concept as a key driver
of Internet of Everything (IoE) applications. These 5G challenges are behind
worldwide efforts to enable future networks, such as sixth generation (6G)
networks, to efficiently support sophisticated applications ranging from
autonomous driving capabilities to the Metaverse. Edge learning is a new and
powerful approach to training models across distributed clients while
protecting the privacy of their data. This approach is expected to be embedded
within future network infrastructures, including 6G, to solve challenging
problems such as resource management and behavior prediction. This survey
article provides a holistic review of the most recent research focused on edge
learning vulnerabilities and defenses for 6G-enabled IoT. We summarize the
existing surveys on machine learning for 6G IoT security and machine
learning-associated threats in three different learning modes: centralized,
federated, and distributed. Then, we provide an overview of enabling emerging
technologies for 6G IoT intelligence. Moreover, we provide a holistic survey of
existing research on attacks against machine learning and classify threat
models into eight categories, including backdoor attacks, adversarial examples,
combined attacks, poisoning attacks, Sybil attacks, byzantine attacks,
inference attacks, and dropping attacks. In addition, we provide a
comprehensive and detailed taxonomy and a side-by-side comparison of the
state-of-the-art defense methods against edge learning vulnerabilities.
Finally, as new attacks and defense technologies are realized, new research and
future overall prospects for 6G-enabled IoT are discussed
Fortifying robustness: unveiling the intricacies of training and inference vulnerabilities in centralized and federated neural networks
Neural network (NN) classifiers have gained significant traction in diverse domains such as natural language processing, computer vision, and cybersecurity, owing to their remarkable ability to approximate complex latent distributions from data. Nevertheless, the conventional assumption of an attack-free operating environment has been challenged by the emergence of adversarial examples. These perturbed samples, which are typically imperceptible to human observers, can lead to misclassifications by the NN classifiers. Moreover, recent studies have uncovered the ability of poisoned training data to generate Trojan backdoored classifiers that exhibit misclassification behavior triggered by predefined patterns.
In recent years, significant research efforts have been dedicated to uncovering the vulnerabilities of NN classifiers and developing defenses or mitigations against them. However, the existing approaches still fall short of providing mature solutions to address this ever-evolving problem. The widely adopted defense mechanisms against adversarial examples are computationally expensive and impractical for certain real-world applications. Likewise, the practical black-box defense against Trojan backdoors has failed to achieve state-of-the-art performance. More concerning is the limited exploration of these vulnerabilities within the context of cooperative attack or Federated learning, leaving NN classifiers exposed to unknown risks. This dissertation aims to address these critical gaps and refine our understanding of these vulnerabilities. The research conducted within this dissertation encompasses both the attack and defense perspectives, aiming to shed light on future research directions for vulnerabilities in NN classifiers
- …