Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Due to the increasing usage of machine learning (ML) techniques in security- and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VeriVis, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VeriVis leverage different input space reduction techniques for efficient verification of different safety properties. VeriVis is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VeriVis can successfully verify local safety properties, on average, for around 31.7% of the test images. VeriVis finds up to 64.8x more violations than existing gradient-based methods that, unlike VeriVis, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60.2%.Comment: 16 pages, 11 tables, 11 figure

Evaluating Explanation Methods for Deep Learning in Security

Deep learning is increasingly used as a building block of security systems. Unfortunately, neural networks are hard to interpret and typically opaque to the practitioner. The machine learning community has started to address this problem by developing methods for explaining the predictions of neural networks. While several of these approaches have been successfully applied in the area of computer vision, their application in security has received little attention so far. It is an open question which explanation methods are appropriate for computer security and what requirements they need to satisfy. In this paper, we introduce criteria for comparing and evaluating explanation methods in the context of computer security. These cover general properties, such as the accuracy of explanations, as well as security-focused aspects, such as the completeness, efficiency, and robustness. Based on our criteria, we investigate six popular explanation methods and assess their utility in security systems for malware detection and vulnerability discovery. We observe significant differences between the methods and build on these to derive general recommendations for selecting and applying explanation methods in computer security.Comment: IEEE European Symposium on Security and Privacy, 202

Verification of temporal-epistemic properties of access control systems

Verification of access control systems against vulnerabilities has always been a challenging problem in the world of computer security. The complication of security policies in large- scale multi-agent systems increases the possible existence of vulnerabilities as a result of mistakes in policy definition. This thesis explores automated methods in order to verify temporal and epistemic properties of access control systems. While temporal property verification can reveal a considerable number of security holes, verification of epistemic properties in multi-agent systems enable us to infer about agents' knowledge in the system and hence, to detect unauthorized information flow. This thesis first presents a framework for knowledge-based verification of dynamic access control policies. This framework models a coalition-based system, which evaluates if a property or a goal can be achieved by a coalition of agents restricted by a set of permissions defined in the policy. Knowledge is restricted to the information that agents can acquire by reading system information in order to increase time and memory efficiency. The framework has its own model-checking method and is implemented in Java and released as an open source tool named \char{cmmi10}{0x50}\char{cmmi10}{0x6f}\char{cmmi10}{0x6c}\char{cmmi10}{0x69}\char{cmmi10}{0x56}\char{cmmi10}{0x65}\char{cmmi10}{0x72}. In order to detect information leakage as a result of reasoning, the second part of this thesis presents a complimentary technique that evaluates access control policies over temporal-epistemic properties where the knowledge is gained by reasoning. We will demonstrate several case studies for a subset of properties that deal with reasoning about knowledge. To increase the efficiency, we develop an automated abstraction refinement technique for evaluating temporal-epistemic properties. For the last part of the thesis, we develop a sound and complete algorithm in order to identify information leakage in Datalog-based trust management systems

Snake-Oil Security Claims the Systematic Misrepresentation of Product Security in the E-Commerce Arena

The modern commercial systems and software industry in the United States have grown up in a snake-oil salesman\u27s paradise. The largest sector of this industry by far is composed of standard commercial systems that are marketed to provide specified functionality (e.g. Internet web server, firewall, router, etc.) Such products are generally provided with a blanket disclaimer stating that the purchaser must evaluate the suitability of the product for use, and that the user assumes all liability for product behavior. In general, users cannot evaluate and cannot be expected to evaluate the security claims of a product. The ability to analyze security claims is important because a consumer may place unwarranted trust in the security abilities of a web server (or other computer device) to perform its stated purpose, thereby putting his own organization at risk, as well as third parties (consumers, business partners, etc.) All but the largest and most capable organizations lack the resources or expertise to evaluate the security claims of a product. More importantly, no reasonable and knowledgeable person would expect them to be able to do so. The normal legal presumptions of approximate equality of bargaining power and comparable sophistication in evaluating benefits and risks are grievously unjust in the context of software security. In these transactions, it is far wiser to view the general purchaser, even if that purchaser is a sizable corporation, as an ignorant consumer. Hence, often purchasers accept what appear to be either implied merchantability claims of the vendor or claims of salespersons\u27 made outside of the context of a written document. These claims frequently have little, if any, basis in fact. These standard commercial systems form the bulk of the critical infrastructure of existing Internet functionality and e-commerce systems. Often, these systems are not trustworthy, yet the use of these systems by misinformed purchasers created massive vulnerability for both purchasers and third parties (including a substantial fraction of both U.S. and international citizens). The frequent disclosure of individual credit card information from supposedly secure commercial systems illustrates an aspect of this vulnerability and raises serious questions concerning the merchantability of these systems. While it is impossible to avoid all risks, they can be reduced to a very small fraction of their current level. Vendors have willfully taken approaches and used processes that do not allow assurance of appropriate security properties, while simultaneously and recklessly misrepresenting the security properties of their products to their customers

On a Catalogue of Metrics for Evaluating Commercial Cloud Services

Given the continually increasing amount of commercial Cloud services in the market, evaluation of different services plays a significant role in cost-benefit analysis or decision making for choosing Cloud Computing. In particular, employing suitable metrics is essential in evaluation implementations. However, to the best of our knowledge, there is not any systematic discussion about metrics for evaluating Cloud services. By using the method of Systematic Literature Review (SLR), we have collected the de facto metrics adopted in the existing Cloud services evaluation work. The collected metrics were arranged following different Cloud service features to be evaluated, which essentially constructed an evaluation metrics catalogue, as shown in this paper. This metrics catalogue can be used to facilitate the future practice and research in the area of Cloud services evaluation. Moreover, considering metrics selection is a prerequisite of benchmark selection in evaluation implementations, this work also supplements the existing research in benchmarking the commercial Cloud services.Comment: 10 pages, Proceedings of the 13th ACM/IEEE International Conference on Grid Computing (Grid 2012), pp. 164-173, Beijing, China, September 20-23, 201

On Evaluating Commercial Cloud Services: A Systematic Review

Background: Cloud Computing is increasingly booming in industry with many competing providers and services. Accordingly, evaluation of commercial Cloud services is necessary. However, the existing evaluation studies are relatively chaotic. There exists tremendous confusion and gap between practices and theory about Cloud services evaluation. Aim: To facilitate relieving the aforementioned chaos, this work aims to synthesize the existing evaluation implementations to outline the state-of-the-practice and also identify research opportunities in Cloud services evaluation. Method: Based on a conceptual evaluation model comprising six steps, the Systematic Literature Review (SLR) method was employed to collect relevant evidence to investigate the Cloud services evaluation step by step. Results: This SLR identified 82 relevant evaluation studies. The overall data collected from these studies essentially represent the current practical landscape of implementing Cloud services evaluation, and in turn can be reused to facilitate future evaluation work. Conclusions: Evaluation of commercial Cloud services has become a world-wide research topic. Some of the findings of this SLR identify several research gaps in the area of Cloud services evaluation (e.g., the Elasticity and Security evaluation of commercial Cloud services could be a long-term challenge), while some other findings suggest the trend of applying commercial Cloud services (e.g., compared with PaaS, IaaS seems more suitable for customers and is particularly important in industry). This SLR study itself also confirms some previous experiences and reveals new Evidence-Based Software Engineering (EBSE) lessons

On Properties of Policy-Based Specifications

The advent of large-scale, complex computing systems has dramatically increased the difficulties of securing accesses to systems' resources. To ensure confidentiality and integrity, the exploitation of access control mechanisms has thus become a crucial issue in the design of modern computing systems. Among the different access control approaches proposed in the last decades, the policy-based one permits to capture, by resorting to the concept of attribute, all systems' security-relevant information and to be, at the same time, sufficiently flexible and expressive to represent the other approaches. In this paper, we move a step further to understand the effectiveness of policy-based specifications by studying how they permit to enforce traditional security properties. To support system designers in developing and maintaining policy-based specifications, we formalise also some relevant properties regarding the structure of policies. By means of a case study from the banking domain, we present real instances of such properties and outline an approach towards their automatised verification.Comment: In Proceedings WWV 2015, arXiv:1508.0338

Understanding and Specifying Information Security Needs to Support the Delivery of High Quality Security Services

In this paper we present an approach for specifying and prioritizing information security requirements in organizations. It is important to prioritize security requirements since hundred per cent security is\ud not achievable and the limited resources available should be directed to satisfy the most important ones. We propose to explicitly link security requirements with the organization’s business vision, i.e. to provide business\ud rationale for security requirements. The rationale is then used as a basis for comparing the importance of different security requirements.\ud Furthermore we discuss how to integrate the aforementioned solution concepts into a service level management process for security services, which is an important step in IT Governance. We validate our approach by way of a focus group session