With the increased popularity of embedded devices, low-level programming languages like C and C++ are currently experiencing a strong renewed interest. However, these languages are, meaning that programming errors may lead to undefined behaviour, which, in turn, may be exploited to compromise a system's integrity. Many programs written in these languages contain such programming errors, most infamous of which are buffer overflows. In order to fight this, there exists a large range of mitigation techniques designed to hinder exploitation, some of which are integral parts of most major operating systems' security concept. Even the most sophisticated mitigations, however, can often be bypassed by modern exploits, which are based on the principle of code reuse: they assemble, or chain, together existing code fragments (known as gadgets) in a way to achieve malicious behaviour. This technique is currently the cornerstone of modern exploits.
In this dissertation, we present ROPocop, an approach to mitigate code-reuse attacks. ROPocop is a configurable, heuristic-based detector that monitors program execution and raises an alarm if it detects suspicious behaviour. It monitors the frequency of indirect branches and the length of basic blocks, two characteristics in which code-reuse attacks differ greatly from normal program behaviour. However, like all mitigations, ROPocop has its weaknesses and we show that it and other similar approaches can be bypassed in an automatic way by an aware attacker. To this end, we present PSHAPE, a practical, cross-platform framework to support the construction of code-reuse exploits. It offers two distinguishing features, namely it creates concise semantic summaries for gadgets, which allow exploit developers to assess the utility of a gadget much quicker than by going through the individual assembly instructions. And secondly, PSHAPE automatically composes gadgets to construct a chain of gadgets that can invoke any arbitrary function with user-supplied parameters. Invoking a function is indeed the most common goal of concurrent exploits, as calling a function such as mprotect greatly simplifies later steps of exploitation.
For a mitigation to be viable, it must detect actual attacks reliably while at the same time avoiding false positives and ensuring that protected applications remain usable, i.e., do not crash or become very slow. In the tested sample set of applications, ROPocop detects and stops all twelve real attacks with no false positives. When executed with ROPocop, real-world programs exhibit only some slight input lag at startup but otherwise remain responsive. Yet, we further show how PSHAPE can be used to fully automatically create exploits that bypass various mitigations, for example, ROPocop itself. We also show gadgets PSHAPE found easily, that have great relevance in real exploits, and which previously required intense manual searches to find. Lastly, using PSHAPE, we also discovered a new and very useful gadget type that greatly simplifies gadget chaining
