ECONOMICALLY PROTECTING COMPLEX, LEGACY OPERATING SYSTEMS USING SECURE DESIGN PRINCIPLES

In modern computer systems, complex legacy operating systems, such as Linux, are deployed ubiquitously. Many design choices in these legacy operating systems predate a modern understanding of security risks. As a result, new attack opportunities are routinely discovered to subvert such systems, which reveal design flaws that spur new research about secure design principles and other security mechanisms to thwart these attacks. Most research falls into two categories: encapsulating the threat and redesigning the system from scratch. Each approach has its challenge. Encapsulation can only limit the exposure to the risk, but not entirely prevent it. Rewriting the huge codebase of these operating systems is impractical in terms of developer effort, but appealing inasmuch as it can comprehensively eliminate security risks. This thesis pursues a third, understudied option: retrofitting security design principles in the existing kernel design. Conventional wisdom discourages retrofitting security because retrofitting is a hard problem, may require the use of new abstractions or break backward compatibility, may have unforeseen consequences, and may be equivalent to redesigning the system from scratch in terms of effort. This thesis offers new evidence to challenge this conventional wisdom, indicating that one can economically retrofit a comprehensive security policy onto complex, legacy systems. To demonstrate this assertion, this thesis firstly surveys the alternative of encapsulating the threat to the complex, legacy system by adding a monitoring layer using a technique called Virtual Machine Introspection, and discusses the shortcomings of this technique. Secondly, this thesis shows how to enforce the principle of least privilege by removing the need to run setuid-to-root binaries with administrator privilege. Finally, this thesis takes the first steps to show how to economically retrofit secure design principles to the OS virtualization feature of the Linux kernel called containers without rewriting the whole system. This approach can be applied more generally to other legacy systems.Doctor of Philosoph