Engineering with logic: Rigorous test-oracle specification and validation for TCP/IP and the Sockets API

Conventional computer engineering relies on test-and-debug development processes, with the behavior of common interfaces described (at best) with prose specification documents. But prose specifications cannot be used in test-and-debug development in any automated way, and prose is a poor medium for expressing complex (and loose) specifications. The TCP/IP protocols and Sockets API are a good example of this: they play a vital role in modern communication and computation, and interoperability between implementations is essential. But what exactly they are is surprisingly obscure: their original development focused on “rough consensus and running code,” augmented by prose RFC specifications that do not precisely define what it means for an implementation to be correct. Ultimately, the actual standard is the de facto one of the common implementations, including, for example, the 15 000 to 20 000 lines of the BSD implementation—optimized and multithreaded C code, time dependent, with asynchronous event handlers, intertwined with the operating system, and security critical. This article reports on work done in the Netsem project to develop lightweight mathematically rigorous techniques that can be applied to such systems: to specify their behavior precisely (but loosely enough to permit the required implementation variation) and to test whether these specifications and the implementations correspond with specifications that are executable as test oracles . We developed post hoc specifications of TCP, UDP, and the Sockets API, both of the service that they provide to applications (in terms of TCP bidirectional stream connections) and of the internal operation of the protocol (in terms of TCP segments and UDP datagrams), together with a testable abstraction function relating the two. These specifications are rigorous, detailed, readable, with broad coverage, and rather accurate. Working within a general-purpose proof assistant (HOL4), we developed language idioms (within higher-order logic) in which to write the specifications: operational semantics with nondeterminism, time, system calls, monadic relational programming, and so forth. We followed an experimental semantics approach, validating the specifications against several thousand traces captured from three implementations (FreeBSD, Linux, and WinXP). Many differences between these were identified, as were a number of bugs. Validation was done using a special-purpose symbolic model checker programmed above HOL4. Having demonstrated that our logic-based engineering techniques suffice for handling real-world protocols, we argue that similar techniques could be applied to future critical software infrastructure at design time, leading to cleaner designs and (via specification-based testing) more robust and predictable implementations. In cases where specification looseness can be controlled, this should be possible with lightweight techniques, without the need for a general-purpose proof assistant, at relatively little cost.EPSRC Programme Grant EP/K008528/1 REMS: Rigorous Engineering for Mainstream Systems EPSRC Leadership Fellowship EP/H005633 (Sewell) Royal Society University Research Fellowship (Sewell) St Catharine's College Heller Research Fellowship (Wansbrough), EPSRC grant GR/N24872 Wide-area programming: Language, Semantics and Infrastructure Design EPSRC grant EP/C510712 NETSEM: Rigorous Semantics for Real Systems EC FET-GC project IST-2001-33234 PEPITO Peer-to-Peer Computing: Implementation and Theory CMI UROP internship support (Smith) EC Thematic Network IST-2001-38957 APPSEM 2 NICTA was funded by the Australian Government's Backing Australia's Ability initiative, in part through the Australian Research Council

Observing the Evolution of QUIC Implementations

The QUIC protocol combines features that were initially found inside the TCP, TLS and HTTP/2 protocols. The IETF is currently finalising a complete specification of this protocol. More than a dozen of independent implementations have been developed in parallel with these standardisation activities. We propose and implement a QUIC test suite that interacts with public QUIC servers to verify their conformance with key features of the IETF specification. Our measurements, gathered over a semester, provide a unique viewpoint on the evolution of a protocol and of its implementations. They highlight the arrival of new features and some regressions among the different implementations.Comment: 6 pages, 8 figure

SymbexNet: Checking Network Protocol Implementations using Symbolic Execution

The implementations of network protocols, such as DNS, DHCP and Zeroconf, are prone to flaws, security vulnerabilities and interoperability issues caused by ambiguous requirements in protocol specifications. Detecting such problems is not easy because (i) many bugs manifest themselves only after prolonged operation; (ii) the state space of complex protocol implementations is large; and (iii) problems often require additional information about correct behaviour from specifications. This thesis presents a novel approach to detect various types of flaws in network protocol implementations by combining symbolic execution and rule-based packet matching. The core idea behind our approach is to generate automatically high-coverage test input packets for a network protocol implementation. For this, the protocol implementation is run using a symbolic execution engine to obtain test input packets. These packets are then used to detect potential violations of rules that constrain permitted input and output packets and were derived from the protocol specification. We propose a technique that repeatedly performs symbolic execution on selected test input packets to achieve broad and deep exploration of the implementation state space. In addition, we use the generated test packets to check interoperability between different implementations of the same network protocol. We present a system based on these techniques, SYMBEXNET, and show that it can automatically generate test input packets that achieve high source code coverage and discover various bugs. We evaluate SYMBEXNET on multiple implementations of two network protocols: Zeroconf, a service discovery protocol, and DHCP, a network configuration protocol. SYMBEXNET is able to discover non-trivial bugs as well as interoperability problems, most of which have been confirmed by the developers

Applying Formal Methods to Networking: Theory, Techniques and Applications

Despite its great importance, modern network infrastructure is remarkable for the lack of rigor in its engineering. The Internet which began as a research experiment was never designed to handle the users and applications it hosts today. The lack of formalization of the Internet architecture meant limited abstractions and modularity, especially for the control and management planes, thus requiring for every new need a new protocol built from scratch. This led to an unwieldy ossified Internet architecture resistant to any attempts at formal verification, and an Internet culture where expediency and pragmatism are favored over formal correctness. Fortunately, recent work in the space of clean slate Internet design---especially, the software defined networking (SDN) paradigm---offers the Internet community another chance to develop the right kind of architecture and abstractions. This has also led to a great resurgence in interest of applying formal methods to specification, verification, and synthesis of networking protocols and applications. In this paper, we present a self-contained tutorial of the formidable amount of work that has been done in formal methods, and present a survey of its applications to networking.Comment: 30 pages, submitted to IEEE Communications Surveys and Tutorial

A Test Generation Framework for Distributed Fault-Tolerant Algorithms

Heavyweight formal methods such as theorem proving have been successfully applied to the analysis of safety critical fault-tolerant systems. Typically, the models and proofs performed during such analysis do not inform the testing process of actual implementations. We propose a framework for generating test vectors from specifications written in the Prototype Verification System (PVS). The methodology uses a translator to produce a Java prototype from a PVS specification. Symbolic (Java) PathFinder is then employed to generate a collection of test cases. A small example is employed to illustrate how the framework can be used in practice