Coyote C++: An Industrial-Strength Fully Automated Unit Testing Tool

Coyote C++ is an automated testing tool that uses a sophisticated concolic-execution-based approach to realize fully automated unit testing for C and C++. While concolic testing has proven effective for languages such as C and Java, tools have struggled to achieve a practical level of automation for C++ due to its many syntactical intricacies and overall complexity. Coyote C++ is the first automated testing tool to breach the barrier and bring automated unit testing for C++ to a practical level suitable for industrial adoption, consistently reaching around 90% code coverage. Notably, this testing process requires no user involvement and performs test harness generation, test case generation and test execution with "one-click" automation. In this paper, we introduce Coyote C++ by outlining its high-level structure and discussing the core design decisions that shaped the implementation of its concolic execution engine. Finally, we demonstrate that Coyote C++ is capable of achieving high coverage results within a reasonable timespan by presenting the results from experiments on both open-source and industrial software

Proceedings of the First NASA Formal Methods Symposium

Topics covered include: Model Checking - My 27-Year Quest to Overcome the State Explosion Problem; Applying Formal Methods to NASA Projects: Transition from Research to Practice; TLA+: Whence, Wherefore, and Whither; Formal Methods Applications in Air Transportation; Theorem Proving in Intel Hardware Design; Building a Formal Model of a Human-Interactive System: Insights into the Integration of Formal Methods and Human Factors Engineering; Model Checking for Autonomic Systems Specified with ASSL; A Game-Theoretic Approach to Branching Time Abstract-Check-Refine Process; Software Model Checking Without Source Code; Generalized Abstract Symbolic Summaries; A Comparative Study of Randomized Constraint Solvers for Random-Symbolic Testing; Component-Oriented Behavior Extraction for Autonomic System Design; Automated Verification of Design Patterns with LePUS3; A Module Language for Typing by Contracts; From Goal-Oriented Requirements to Event-B Specifications; Introduction of Virtualization Technology to Multi-Process Model Checking; Comparing Techniques for Certified Static Analysis; Towards a Framework for Generating Tests to Satisfy Complex Code Coverage in Java Pathfinder; jFuzz: A Concolic Whitebox Fuzzer for Java; Machine-Checkable Timed CSP; Stochastic Formal Correctness of Numerical Algorithms; Deductive Verification of Cryptographic Software; Coloured Petri Net Refinement Specification and Correctness Proof with Coq; Modeling Guidelines for Code Generation in the Railway Signaling Context; Tactical Synthesis Of Efficient Global Search Algorithms; Towards Co-Engineering Communicating Autonomous Cyber-Physical Systems; and Formal Methods for Automated Diagnosis of Autosub 6000

Coding Guidelines and Undecidability

The C and C++ programming languages are widely used for the implementation of software in critical systems. They are complex languages with subtle features and peculiarities that might baffle even the more expert programmers. Hence, the general prescription of language subsetting, which occurs in most functional safety standards and amounts to only using a "safer" subset of the language, is particularly applicable to them. Coding guidelines are the preferred way of expressing language subsets. Some guidelines are formulated in terms of the programming language and its implementation only: in this case they are amenable to automatic checking. However, due to fundamental limitations of computing, some guidelines are undecidable, that is, they are based on program properties that no current and future algorithm can capture in all cases. The most mature and widespread coding standards, the MISRA ones, explicitly tag guidelines with undecidable or decidable. It turns out that this information is not of secondary nature and must be taken into account for a full understanding of what the guideline is asking for. As a matter of fact, undecidability is a common source of confusion affecting many users of coding standards and of the associated checking tools. In this paper, we recall the notions of decidability and undecidability in terms that are understandable to any C/C++ programmer. The paper includes a systematic study of all the undecidable MISRA C:2012 guidelines, discussing the reasons for the undecidability and its consequences. We pay particular attention to undecidable guidelines that have decidable approximations whose enforcement would not overly constrain the source code. We also discuss some coding guidelines for which compliance is hard, if not impossible, to prove, even beyond the issue of decidability.Comment: 12 pages, 5 figures, 1 tabl

New Opportunities for Integrated Formal Methods

Formal methods have provided approaches for investigating software engineering fundamentals and also have high potential to improve current practices in dependability assurance. In this article, we summarise known strengths and weaknesses of formal methods. From the perspective of the assurance of robots and autonomous systems~(RAS), we highlight new opportunities for integrated formal methods and identify threats to their adoption to be mitigated. Based on these opportunities and threats, we develop an agenda for fundamental and empirical research on integrated formal methods and for successful transfer of validated research to RAS assurance. Furthermore, we outline our expectations on useful outcomes of such an agenda

Securing the in-vehicle network

Recent research into automotive security has shown that once a single electronic vehicle component is compromised, it is possible to take control of the vehicle. These components, called Electronic Control Units, are embedded systems which manage a significant part of the functionality of a modern car. They communicate with each other via the in-vehicle network, known as the Controller Area Network, which is the most widely used automotive bus. In this thesis, we introduce a series of novel proposals to improve the security of both the Controller Area Network bus and the Electronic Control Units. The Controller Area Network suffers from a number of shortfalls, one of which is the lack of source authentication. We propose a protocol that mitigates this fundamental shortcoming in the Controller Area Network bus design, and protects against a number of high profile media attacks that have been published. We derive a set of desirable security and compatibility properties which an authentication protocol for the Controller Area Network bus should possess. We evaluate our protocol, along with other proposed protocols in the literature, with respect to the defined properties. Our systematic analysis of the protocols allows the automotive industry to make an informed choice regarding the adoption suitability of these solutions. However, it is not only the communication of Electronic Control Units that needs to be secure, but the firmware running on them as well. The growing number of Electronic Control Units in a vehicle, together with their increasing complexity, prompts the need for automated tools to test their security. Part of the challenge in designing such a tool is the diversity of Electronic Control Unit architectures. To this end, this thesis presents a methodology for extracting the Control Flow Graph from the Electronic Control Unit firmware. The Control Flow Graph is a platform independent representation of the firmware control flow, allowing us to abstract from the underlying architecture. We present a fuzzer for Electronic Control Unit firmware fuzz-testing via Controller Area Network. The extracted Control Flow Graph is tagged with static data used in instructions which influence the control flow of the firmware. It is then used to create a set of input seeds for the fuzzer, and in altering the inputs during the fuzzing process. This approach represents a step towards an efficient fuzzing methodology for Electronic Control Units. To our knowledge, this is the first proposal that uses static analysis to guide the fuzzing of Electronic Control Units

Improving Scalability of Symbolic Execution for Software with Complex Environment Interfaces

Manual software testing is laborious and prone to human error. Yet, among practitioners, it is the most popular method for quality assurance. Automating the test case generation promises better effectiveness, especially for exposing corner-case bugs. Symbolic execution stands out as an automated testing technique that has no false positives, it eventually enumerates all feasible program executions, and can prioritize executions of interest. However, path explosionâthe fact that the number of program executions is typically at least exponential in the size of the programâhinders the applicability of symbolic execution in the real world, where software commonly reaches millions of lines of code. In practice, large systems can be efficiently executed symbolically by exploiting their modularity and thus symbolically execute the different parts of the system separately. However, a component typically depends on its environment to perform its task. Thus, a symbolic execution engine needs to provide an environment interface that is efficient, while maintaining accuracy and completeness. This conundrum is known as the environment problem. Systematically addressing the environment problem is challenging, as its instantiation depends on the nature of the environment and its interface. This thesis addresses two instances of the environment problem in symbolic execution, which are at opposite ends of the spectrum of interface stability: (1) system software interacting with an operating system with stable and well-documented semantics (e.g., POSIX), and (2) high-level programs written in dynamic languages, such as Python, Ruby, or JavaScript, whose semantics and interfaces are continuously evolving. To address the environment problem for stable operating system interfaces, this thesis introduces the idea of splitting an operating system model into a core set of primitives built into the engine at host level and, on top of it, the full operating system interface emulated inside the guest. As few as two primitives are sufficient to support a complex interface such as POSIX: threads with synchronization and address spaces with shared memory. We prototyped this idea in the Cloud9 symbolic execution platform. Cloud9's accurate and efficient POSIX model exposes hard-to-reproduce bugs in systems such as UNIX utilities, web servers, and distributed systems. Cloud9 is available at http://cloud9.epfl.ch. For programs written in high-level interpreted languages, this thesis introduces the idea of using the language interpreter as an "executable language specification". The interpreter runs inside a low-level (e.g., x86) symbolic execution engine while it executes the target program. The aggregate system acts as a high-level symbolic execution engine for the program. To manage the complexity of symbolically executing the entire interpreter, this thesis introduces Class-Uniform Path Analysis (CUPA), an algorithm for prioritizing paths that groups paths into equivalence classes according to a coverage goal. We built a prototype of these ideas in the form of Chef, a symbolic execution platform for interpreted languages that generates up to 1000 times more tests in popular Python and Lua packages compared to a plain execution of the interpreters. Chef is available at http://dslab.epfl.ch/proj/chef/