Red Teaming Deep Neural Networks with Feature Synthesis Tools

Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using \emph{feature synthesis} methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. A website for this paper and its code is at https://benchmarking-interpretability.csail.mit.edu/Comment: In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023

Investigation into Detection of Hardware Trojans on Printed Circuit Boards

The modern semiconductor device manufacturing flow is becoming increasingly vulnerable to malicious implants called Hardware Trojans (HT). With HTs becoming stealthier, a need for more accurate and efficient detection methods is becoming increasingly crucial at both Integrated Circuit (IC) and Printed Circuit Board (PCB) levels. While HT detection at an IC level has been widely studied, there is still very limited research on detecting and preventing HTs implanted on PCBs. In recent years the rise of outsourcing design and fabrication of electronics, including PCBs, to third parties has dramatically increased the possibility of malicious alteration and consequently the security risk for systems incorporating PCBs. Providing mechanical support for the electrical interconnections between different components, PCBs are an important part of electronic systems. Modern, complex and highly integrated designs may contain up to thirty layers, with concealed micro-vias and embedded passive components. An adversary can aim to modify the PCB design by tampering the copper interconnections or inserting extra components in an internal layer of a multi-layer board. Similar to its IC counterpart, a PCB HT can, among other things, cause system failure or leakage of private information. The disruptive actions of a carefully designed HT attack can have catastrophic implications and should therefore be taken seriously by industry, academia and the government. This thesis gives an account of work carried out in three projects concerned with HT detection on a PCB. In the first contribution a power analysis method is proposed for detecting HT components, implanted on the surface or otherwise, consuming power from the power distribution network. The assumption is that any HT device actively tampering or eavesdropping on the signals in the PCB circuit will consume electrical power. Harvesting this side-channel effect and observing the fluctuations of power consumption on the PCB power distribution network enables evincing the HT. Using a purpose-built PCB prototype, an experimental setup is developed for verification of the methodology. The results confirm the ability to detect alien components on a PCB without interference with its main functionality. In the second contribution the monitoring methodology is further developed by applying machine learning (ML) techniques to detect stealthier HTs, consuming power from I/O ports of legitimate ICs on the PCB. Two algorithms, One-Class Support Vector Machine (SVM) and Local Outlier Factor (LOF), are implemented on the legitimate power consumption data harvested experimentally from the PCB prototype. Simulation results are validated through real-life measurements and experiments are carried out on the prototype PCB. For validation of the ML classification models, one hundred categories of HTs are modelled and inserted into the datasets. Simulation results show that using the proposed methodology an HT can be detected with high prediction accuracy (F1-score at 99% for a 15 mW HT). Further, the developed ML model is uploaded to the prototype PCB for experimental validation. The results show consistency between simulations and experiments, with an average discrepancy of ±5.9% observed between One-Class SVM simulations and real-life experiments. The machine learning models developed for HT detection are low-cost in terms of memory (around 27 KB). In the third contribution an automated visual inspection methodology is proposed for detecting HTs on the surface of a PCB. It is based on a combination of conventional computer vision techniques and a dual tower Siamese Neural Network (SNN), modelled in a three stage pipeline. In the interest of making the proposed methodology broadly applicable a particular emphasis is made on the imaging modality of choice, whereby a regular digital optical camera is chosen. The dataset of PCB images is developed in a controlled environment of a photographic tent. The novelty in this work is that, instead of a generic production fault detection, the algorithm is optimised and trained specifically for implanted HT component detection on a PCB, be it active or passive. The proposed HT detection methodology is trained and tested with three groups of HTs, categorised based on their surface area, ranging from 4 mm² to 280 mm² and above. The results show that it is possible to reach effective detection accuracy of 95.1% for HTs as small as 4 mm². In case of HTs with surface area larger than 280 mm² the detection accuracy is around 96.1%, while the average performance across all HT groups is 95.6%

Enhancing the governance of information security in developing countries: the case of Zanzibar

A thesis submitted to the University of Bedfordshire, in partial fulfilment of the requirements of the degree of Doctor of PhilosophyOrganisations in the developing countries need to protect their information assets (IA) in an optimal way. This thesis is based upon the argument that in order to achieve fully effective information security management (ISM) strategy, it is essential to look at information security in a socio-technical context, i.e. the cultural, ethical, moral, legal dimensions, tools, devices and techniques. The motivation for this study originated from the concern of social chaos, which results from ineffective information security practices in organisations in the developing nations. The present strategies were developed for organisations in countries where culture is different to culture of the developing world. Culture has been pointed out as an important factor of human behaviour. This research is trying to enhance information security culture in the context of Zanzibar by integrating both social and technical issues. The theoretical foundation for this research is based on cultural theories and the theory of semiotics. In particular, the study utilised the GLOBE Project (House et al, 2004), Competing Values Framework (Quinn and Cameron; 1983) and Semiotic Framework (Liu, 2000). These studies guide the cultural study and the semiotics study. The research seeks to better understand how culture impact the governance of information security and develop a framework that enhances the governance of information security in non-profit organisations. ISO/IEC 27002 best practices in information security management provided technical guidance in this work. The major findings include lack of benchmarking in the governance of information security. Cultural issues impact the governance of information security. Drawing the evidence from the case study a framework for information security culture was proposed. In addition, a novel process model for information security analysis based on semiotics was developed. The process model and the framework integrated both social and technical issues and could be implemented in any non-profit organisation operating within a societal context with similar cultural feature as Zanzibar. The framework was evaluated using this process model developed in this research. The evaluated framework provides opportunities for future research in this area

Automating Cyber Analytics

Model based security metrics are a growing area of cyber security research concerned with measuring the risk exposure of an information system. These metrics are typically studied in isolation, with the formulation of the test itself being the primary finding in publications. As a result, there is a flood of metric specifications available in the literature but a corresponding dearth of analyses verifying results for a given metric calculation under different conditions or comparing the efficacy of one measurement technique over another. The motivation of this thesis is to create a systematic methodology for model based security metric development, analysis, integration, and validation. In doing so we hope to fill a critical gap in the way we view and improve a system’s security. In order to understand the security posture of a system before it is rolled out and as it evolves, we present in this dissertation an end to end solution for the automated measurement of security metrics needed to identify risk early and accurately. To our knowledge this is a novel capability in design time security analysis which provides the foundation for ongoing research into predictive cyber security analytics. Modern development environments contain a wealth of information in infrastructure-as-code repositories, continuous build systems, and container descriptions that could inform security models, but risk evaluation based on these sources is ad-hoc at best, and often simply left until deployment. Our goal in this work is to lay the groundwork for security measurement to be a practical part of the system design, development, and integration lifecycle. In this thesis we provide a framework for the systematic validation of the existing security metrics body of knowledge. In doing so we endeavour not only to survey the current state of the art, but to create a common platform for future research in the area to be conducted. We then demonstrate the utility of our framework through the evaluation of leading security metrics against a reference set of system models we have created. We investigate how to calibrate security metrics for different use cases and establish a new methodology for security metric benchmarking. We further explore the research avenues unlocked by automation through our concept of an API driven S-MaaS (Security Metrics-as-a-Service) offering. We review our design considerations in packaging security metrics for programmatic access, and discuss how various client access-patterns are anticipated in our implementation strategy. Using existing metric processing pipelines as reference, we show how the simple, modular interfaces in S-MaaS support dynamic composition and orchestration. Next we review aspects of our framework which can benefit from optimization and further automation through machine learning. First we create a dataset of network models labeled with the corresponding security metrics. By training classifiers to predict security values based only on network inputs, we can avoid the computationally expensive attack graph generation steps. We use our findings from this simple experiment to motivate our current lines of research into supervised and unsupervised techniques such as network embeddings, interaction rule synthesis, and reinforcement learning environments. Finally, we examine the results of our case studies. We summarize our security analysis of a large scale network migration, and list the friction points along the way which are remediated by this work. We relate how our research for a large-scale performance benchmarking project has influenced our vision for the future of security metrics collection and analysis through dev-ops automation. We then describe how we applied our framework to measure the incremental security impact of running a distributed stream processing system inside a hardware trusted execution environment

ECHO Information sharing models

As part of the ECHO project, the Early Warning System (EWS) is one of four technologies under development. The E-EWS will provide the capability to share information to provide up to date information to all constituents involved in the E-EWS. The development of the E-EWS will be rooted in a comprehensive review of information sharing and trust models from within the cyber domain as well as models from other domains