Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Learning-based pattern classifiers, including deep networks, have shown impressive performance in several application domains, ranging from computer vision to cybersecurity. However, it has also been shown that adversarial input perturbations carefully crafted either at training or at test time can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (also referred to as adversarial examples), along with the design of suitable countermeasures, have been investigated in the research field of adversarial machine learning. In this work, we provide a thorough overview of the evolution of this research area over the last ten years and beyond, starting from pioneering, earlier work on the security of non-deep learning algorithms up to more recent work aimed to understand the security properties of deep learning algorithms, in the context of computer vision and cybersecurity tasks. We report interesting connections between these apparently-different lines of work, highlighting common misconceptions related to the security evaluation of machine-learning algorithms. We review the main threat models and attacks defined to this end, and discuss the main limitations of current work, along with the corresponding future challenges towards the design of more secure learning algorithms.Comment: Accepted for publication on Pattern Recognition, 201

Cyber threat ontology and adversarial machine learning attacks: analysis and prediction perturbance

Machine learning has been used in the cybersecurity domain to predict cyberattack trends. However, adversaries can inject malicious data into the dataset during training and testing to cause perturbance and predict false narratives. It has become challenging to analyse and predicate cyberattack correlations due to their fuzzy nature and lack of understanding of the threat landscape. Thus, it is imperative to use cyber threat ontology (CTO) concepts to extract relevant attack instances in CSC security for knowledge representation. This paper explores the challenges of CTO and adversarial machine learning (AML) attacks for threat prediction to improve cybersecurity. The novelty contributions are threefold. First, CTO concepts are considered for semantic mapping and definition of relationships for explicit knowledge of threat indicators. Secondly, AML techniques are deployed maliciously to manipulate algorithms during training and testing to predict false classifications models. Finally, we discuss the performance analysis of the classification models and how CTO provides automated means. The result shows that analysis of AML attacks and CTO concepts could be used for validating a mediated schema for specific vulnerabilities

Dynamic adversarial mining - effectively applying machine learning in adversarial non-stationary environments.

While understanding of machine learning and data mining is still in its budding stages, the engineering applications of the same has found immense acceptance and success. Cybersecurity applications such as intrusion detection systems, spam filtering, and CAPTCHA authentication, have all begun adopting machine learning as a viable technique to deal with large scale adversarial activity. However, the naive usage of machine learning in an adversarial setting is prone to reverse engineering and evasion attacks, as most of these techniques were designed primarily for a static setting. The security domain is a dynamic landscape, with an ongoing never ending arms race between the system designer and the attackers. Any solution designed for such a domain needs to take into account an active adversary and needs to evolve over time, in the face of emerging threats. We term this as the ‘Dynamic Adversarial Mining’ problem, and the presented work provides the foundation for this new interdisciplinary area of research, at the crossroads of Machine Learning, Cybersecurity, and Streaming Data Mining. We start with a white hat analysis of the vulnerabilities of classification systems to exploratory attack. The proposed ‘Seed-Explore-Exploit’ framework provides characterization and modeling of attacks, ranging from simple random evasion attacks to sophisticated reverse engineering. It is observed that, even systems having prediction accuracy close to 100%, can be easily evaded with more than 90% precision. This evasion can be performed without any information about the underlying classifier, training dataset, or the domain of application. Attacks on machine learning systems cause the data to exhibit non stationarity (i.e., the training and the testing data have different distributions). It is necessary to detect these changes in distribution, called concept drift, as they could cause the prediction performance of the model to degrade over time. However, the detection cannot overly rely on labeled data to compute performance explicitly and monitor a drop, as labeling is expensive and time consuming, and at times may not be a possibility altogether. As such, we propose the ‘Margin Density Drift Detection (MD3)’ algorithm, which can reliably detect concept drift from unlabeled data only. MD3 provides high detection accuracy with a low false alarm rate, making it suitable for cybersecurity applications; where excessive false alarms are expensive and can lead to loss of trust in the warning system. Additionally, MD3 is designed as a classifier independent and streaming algorithm for usage in a variety of continuous never-ending learning systems. We then propose a ‘Dynamic Adversarial Mining’ based learning framework, for learning in non-stationary and adversarial environments, which provides ‘security by design’. The proposed ‘Predict-Detect’ classifier framework, aims to provide: robustness against attacks, ease of attack detection using unlabeled data, and swift recovery from attacks. Ideas of feature hiding and obfuscation of feature importance are proposed as strategies to enhance the learning framework\u27s security. Metrics for evaluating the dynamic security of a system and recover-ability after an attack are introduced to provide a practical way of measuring efficacy of dynamic security strategies. The framework is developed as a streaming data methodology, capable of continually functioning with limited supervision and effectively responding to adversarial dynamics. The developed ideas, methodology, algorithms, and experimental analysis, aim to provide a foundation for future work in the area of ‘Dynamic Adversarial Mining’, wherein a holistic approach to machine learning based security is motivated

Three Decades of Deception Techniques in Active Cyber Defense -- Retrospect and Outlook

Deception techniques have been widely seen as a game changer in cyber defense. In this paper, we review representative techniques in honeypots, honeytokens, and moving target defense, spanning from the late 1980s to the year 2021. Techniques from these three domains complement with each other and may be leveraged to build a holistic deception based defense. However, to the best of our knowledge, there has not been a work that provides a systematic retrospect of these three domains all together and investigates their integrated usage for orchestrated deceptions. Our paper aims to fill this gap. By utilizing a tailored cyber kill chain model which can reflect the current threat landscape and a four-layer deception stack, a two-dimensional taxonomy is developed, based on which the deception techniques are classified. The taxonomy literally answers which phases of a cyber attack campaign the techniques can disrupt and which layers of the deception stack they belong to. Cyber defenders may use the taxonomy as a reference to design an organized and comprehensive deception plan, or to prioritize deception efforts for a budget conscious solution. We also discuss two important points for achieving active and resilient cyber defense, namely deception in depth and deception lifecycle, where several notable proposals are illustrated. Finally, some outlooks on future research directions are presented, including dynamic integration of different deception techniques, quantified deception effects and deception operation cost, hardware-supported deception techniques, as well as techniques developed based on better understanding of the human element.Comment: 19 page

Towards Robust and Interpretable Deep Learning

Recent progress in deep learning has enabled applications in many areas, such as business, security, and science, that could impact our lives. Despite recent advances in these domains, deep neural network models have been shown to be vulnerable to adversarial attacks and lack of interpretability of their predictions. Therefore, it is crucial to investigate robust and interpretable deep learning models and algorithms to solve the above issues. In this dissertation, we proposed a series of algorithms for delivering robust and interpretable deep learning methods. To begin with, we study the problem of how to defend against adversarial attacks with a purification-based algorithm called defense-VAE. Secondly, we proposed GDPA, a patch attack algorithm that can be readily used in adversarial training. With this algorithm, we can train deep learning models that are robust to patch attacks. Thirdly, we proposed an interpretation algorithm NICE, which learns sparse masks on input images. We also showed how to use this interpretation algorithm for semantic compression on images. Fourthly, we applied NICE on brain MRI data for the schizophrenia discrimination task, in which we detected the important regions of the brain for schizophrenia discrimination. Lastly, we proposed the PSP algorithm, which applied parameter-wise smooth policy in the PPO algorithm to improve the performance and robustness of reinforcement learning (RL) agents