The implementations of network protocols, such as DNS, DHCP and Zeroconf, are prone to flaws,
security vulnerabilities and interoperability issues caused by ambiguous requirements in protocol
specifications. Detecting such problems is not easy because (i) many bugs manifest themselves
only after prolonged operation; (ii) the state space of complex protocol implementations is large;
and (iii) problems often require additional information about correct behaviour from specifications.
This thesis presents a novel approach to detect various types of flaws in network protocol implementations
by combining symbolic execution and rule-based packet matching. The core idea
behind our approach is to generate automatically high-coverage test input packets for a network
protocol implementation. For this, the protocol implementation is run using a symbolic execution
engine to obtain test input packets. These packets are then used to detect potential violations of
rules that constrain permitted input and output packets and were derived from the protocol specification.
We propose a technique that repeatedly performs symbolic execution on selected test input
packets to achieve broad and deep exploration of the implementation state space. In addition, we
use the generated test packets to check interoperability between different implementations of the
same network protocol.
We present a system based on these techniques, SYMBEXNET, and show that it can automatically
generate test input packets that achieve high source code coverage and discover various bugs. We
evaluate SYMBEXNET on multiple implementations of two network protocols: Zeroconf, a service
discovery protocol, and DHCP, a network configuration protocol. SYMBEXNET is able to discover
non-trivial bugs as well as interoperability problems, most of which have been confirmed by the
developers