Injecting software faults in Python applications

As técnicas de injeção de falhas de software têm sido amplamente utilizadas como meio para avaliar a confiabilidade de sistemas na presença de certos tipos de falhas. Apesar da grande diversidade de ferramentas que oferecem a possibilidade de emular a presença de falhas de software, há pouco suporte prático para emular a presença de falhas de soft ware em aplicações Python, que cada vez mais são usados para suportar serviços cloud críticos para negócios. Nesta tese, apresentamos uma ferramenta (de nome Fit4Python) para injetar falhas de software em código Python e, de seguida, usamo-la para analisar a eficácia da bateria de testes do OpenStack contra estas novas, prováveis, falhas de software. Começamos por analisar os tipos de falhas que afetam o Nova Compute, um componente central do OpenStack. Usamos a nossa ferramenta para emular a presença de novas falhas na API Nova Compute de forma a entender como a bateria de testes unitários, funcionais e de integração do OpenStack cobre essas novas, mas prováveis, situações. Os resultados mostram limitações claras na eficácia da bateria de testes dos programadores do Open Stack, com muitos casos de falhas injetadas a passarem sem serem detectadas por todos os três tipos de testes. Para além disto, observamos que que a maioria dos problemas analisados poderia ser detectada com mudanças ou acréscimos triviais aos testes unitários

Certifications of Critical Systems – The CECRIS Experience

In recent years, a considerable amount of effort has been devoted, both in industry and academia, to the development, validation and verification of critical systems, i.e. those systems whose malfunctions or failures reach a critical level both in terms of risks to human life as well as having a large economic impact. Certifications of Critical Systems – The CECRIS Experience documents the main insights on Cost Effective Verification and Validation processes that were gained during work in the European Research Project CECRIS (Certification of Critical Systems). The objective of the research was to tackle the challenges of certification by focusing on those aspects that turn out to be more difficult/important for current and future critical systems industry: the effective use of methodologies, processes and tools. Starting from both the scientific and industrial state of the art methodologies for system development and the impact of their usage on the verification and validation and certification of critical systems, the project aimed at developing strategies and techniques supported by automatic or semi-automatic tools and methods for these activities, setting guidelines to support engineers during the planning of the verification and validation phases. Topics covered include: Safety Assessment, Reliability Analysis, Critical Systems and Applications, Functional Safety, Dependability Validation, Dependable Software Systems, Embedded Systems, System Certification

Vulnerability detection in device drivers

Tese de doutoramento, Informática (Ciência da Computação), Universidade de Lisboa, Faculdade de Ciências, 2017The constant evolution in electronics lets new equipment/devices to be regularly made available on the market, which has led to the situation where common operating systems (OS) include many device drivers(DD) produced by very diverse manufactures. Experience has shown that the development of DD is error prone, as a majority of the OS crashes can be attributed to flaws in their implementation. This thesis addresses the challenge of designing methodologies and tools to facilitate the detection of flaws in DD, contributing to decrease the errors in this kind of software, their impact in the OS stability, and the security threats caused by them. This is especially relevant because it can help developers to improve the quality of drivers during their implementation or when they are integrated into a system. The thesis work started by assessing how DD flaws can impact the correct execution of the Windows OS. The employed approach used a statistical analysis to obtain the list of kernel functions most used by the DD, and then automatically generated synthetic drivers that introduce parameter errors when calling a kernel function, thus mimicking a faulty interaction. The experimental results showed that most targeted functions were ineffective in the defence of the incorrect parameters. A reasonable number of crashes and a small number of hangs were observed suggesting a poor error containment capability of these OS functions. Then, we produced an architecture and a tool that supported the automatic injection of network attacks in mobile equipment (e.g., phone), with the objective of finding security flaws (or vulnerabilities) in Wi-Fi drivers. These DD were selected because they are of easy access to an external adversary, which simply needs to create malicious traffic to exploit them, and therefore the flaws in their implementation could have an important impact. Experiments with the tool uncovered a previously unknown vulnerability that causes OS hangs, when a specific value was assigned to the TIM element in the Beacon frame. The experiments also revealed a potential implementation problem of the TCP-IP stack by the use of disassociation frames when the target device was associated and authenticated with a Wi-Fi access point. Next, we developed a tool capable of registering and instrumenting the interactions between a DD and the OS. The solution used a wrapper DD around the binary of the driver under test, enabling full control over the function calls and parameters involved in the OS-DD interface. This tool can support very diverse testing operations, including the log of system activity and to reverse engineer the driver behaviour. Some experiments were performed with the tool, allowing to record the insights of the behaviour of the interactions between the DD and the OS, the parameter values and return values. Results also showed the ability to identify bugs in drivers, by executing tests based on the knowledge obtained from the driver’s dynamics. Our final contribution is a methodology and framework for the discovery of errors and vulnerabilities in Windows DD by resorting to the execution of the drivers in a fully emulated environment. This approach is capable of testing the drivers without requiring access to the associated hardware or the DD source code, and has a granular control over each machine instruction. Experiments performed with Off the Shelf DD confirmed a high dependency of the correctness of the parameters passed by the OS, identified the precise location and the motive of memory leaks, the existence of dormant and vulnerable code.A constante evolução da eletrónica tem como consequência a disponibilização regular no mercado de novos equipamentos/dispositivos, levando a uma situação em que os sistemas operativos (SO) mais comuns incluem uma grande quantidade de gestores de dispositivos (GD) produzidos por diversos fabricantes. A experiência tem mostrado que o desenvolvimento dos GD é sujeito a erros uma vez que a causa da maioria das paragens do SO pode ser atribuída a falhas na sua implementação. Esta tese centra-se no desafio da criação de metodologias e ferramentas que facilitam a deteção de falhas nos GD, contribuindo para uma diminuição nos erros neste tipo de software, o seu impacto na estabilidade do SO, e as ameaças de segurança por eles causadas. Isto é especialmente relevante porque pode ajudar a melhorar a qualidade dos GD tanto na sua implementação como quando estes são integrados em sistemas. Este trabalho inicia-se com uma avaliação de como as falhas nos GD podem levar a um funcionamento incorreto do SO Windows. A metodologia empregue usa uma análise estatística para obter a lista das funções do SO que são mais utilizadas pelos GD, e posteriormente constrói GD sintéticos que introduzem erros nos parâmetros passados durante a chamada às funções do SO, e desta forma, imita a integração duma falta. Os resultados das experiências mostraram que a maioria das funções testadas não se protege eficazmente dos parâmetros incorretos. Observou-se a ocorrência de um número razoável de paragens e um pequeno número de bloqueios, o que sugere uma pobre capacidade das funções do SO na contenção de erros. Posteriormente, produzimos uma arquitetura e uma ferramenta que suporta a injeção automática de ataques em equipamentos móveis (e.g., telemóveis), com o objetivo de encontrar falhas de segurança (ou vulnerabilidades) em GD de placas de rede Wi-Fi. Estes GD foram selecionados porque são de fácil acesso a um atacante remoto, o qual apenas necessita de criar tráfego malicioso para explorar falhas na sua implementação podendo ter um impacto importante. As experiências realizadas com a ferramenta revelaram uma vulnerabilidade anteriormente desconhecida que provoca um bloqueio no SO quando é atribuído um valor específico ao campo TIM da mensagem de Beacon. As experiências também revelaram um potencial problema na implementação do protocolo TCP-IP no uso das mensagens de desassociação quando o dispositivo alvo estava associado e autenticado com o ponto de acesso Wi-Fi. A seguir, desenvolvemos uma ferramenta com a capacidade de registar e instrumentar as interações entre os GD e o SO. A solução usa um GD que envolve o código binário do GD em teste, permitindo um controlo total sobre as chamadas a funções e aos parâmetros envolvidos na interface SO-GD. Esta ferramenta suporta diversas operações de teste, incluindo o registo da atividade do sistema e compreensão do comportamento do GD. Foram realizadas algumas experiências com esta ferramenta, permitindo o registo das interações entre o GD e o SO, os valores dos parâmetros e os valores de retorno das funções. Os resultados mostraram a capacidade de identificação de erros nos GD, através da execução de testes baseados no conhecimento da dinâmica do GD. A nossa contribuição final é uma metodologia e uma ferramenta para a descoberta de erros e vulnerabilidades em GD Windows recorrendo à execução do GD num ambiente totalmente emulado. Esta abordagem permite testar GD sem a necessidade do respetivo hardware ou o código fonte, e possuí controlo granular sobre a execução de cada instrução máquina. As experiências realizadas com GD disponíveis comercialmente confirmaram a grande dependência que os GD têm nos parâmetros das funções do SO, e identificaram o motivo e a localização precisa de fugas de memória, a existência de código não usado e vulnerável

Testing communication reliability with fault injection : Implementation using Robot Framework and SoC-FPGA

Taajuusmuuttajia käytetään teollisuudessa laajasti, sillä merkittävän osan teollisuuden sähkönkulutuksesta muodostavat oikosulkumoottorit, joita ajetaan taajuusmuuttajien avulla. Taajuusmuuttajiin on mahdollista kytkeä optiokortteja, jotka lisäävät taajuusmuuttajaan valvonta-, ohjaus- ym. toiminnallisuuksia. Nämä kortit kommunikoivat sarjaliikenneväylän kautta taajuusmuuttajan pääyksikön kanssa. Sarjaliikennelinkissä, kuten taajuusmuuttajan väylällä, voi syntyä virheitä, jotka häiritsevät tietoliikennettä. Sen takia sarjaliikenneprotokolliin on luotu virheentunnistus- ja -korjausmekanismeja, joilla pyritään varmistamaan virheetön tiedon kuljettaminen. Luotettavuutta testaamaan voidaan väylälle generoida virheitä siihen tarkoitetulla laitteella. Tässä diplomityössä luotiin taajuusmuuttajia valmistavan yrityksen, Danfoss Drivesin (aik. Vacon), pyynnöstä häiriögeneraattorijärjestelmä. Järjestelmä koostuu SoC-FPGA-piirillä luodusta virheitä syöttävästä laitteesta, PC-työkalulle luodusta testirajapinnasta sekä Ethernet-kommunikaatiosta niiden välillä. Laite kytketään väylään, ja testirajapinta tekee testaajalle mahdolliseksi luoda mukautettavia testejä ja ajaa testejä käyttäen Robot Framework -testiympäristöä. Diplomityössä tutkittiin ensin sarjakommunikointiväylien yleisimpiä virheentunnistus- ja korjauskeinoja sekä SoC-FPGA-piirien sekä työssä käytetyn Robot Frameworkin ominaisuuksia. Järjestelmä suunniteltiin ylhäältä-alas-periaatteella ensin tunnistamalla kolmen edellä mainitun komponentin päärakenne päätyen lopulta yksittäisten ohjelmafunktioiden logiikan suunnitteluun. Tämän jälkeen laite ja testirajapinta toteutettiin C- ja Python-ohjelmointikielillä käyttäen suunnitellun kaltaista kommunikaatiota näiden kahden komponentin välillä. Lopulta järjestelmä testattiin kaikki komponentit yhteen kytkettynä. Varsinainen injektorilogiikka, joka luo virheitä väylään, ei ollut työn loppuun mennessä vielä toimittavan tahon puolelta valmis, joten järjestelmää ei voitu testata todellisessa ympäristössä. Työssä luodut osuudet voidaan kuitenkin myöhemmin kytkeä kokonaiseen järjestelmään. Työn tärkeimpänä johtopäätöksenä on, että tavoitteiden mukainen järjestelmä saatiin luotua ja testattua toimivaksi mahdollisin osin. Jatkokehityskohteeksi jäi mm. kokonaisen järjestelmän luonti ja testaus oikeaan kommunikaatioväylään kytkettynä.Frequency converters are widely used in industry because a notable part of the industrial electricity consumption is by electrical induction motors driven by frequency converters. It is possible to connect option boards into a frequency converter to add monitoring and control features. These option boards communicate with the main control unit of the frequency converter over a serial communication link. In a serial communication link, e.g. in a frequency converter, it can occur faults that interfere with the transfer. Hence, error handling mechanisms are used to secure transmission of the data without errors. A fault injector device, which generates errors into the data travelling in the link, can be used to test the communication reliability. In this master’s thesis, an error generator system was created for a company, Danfoss Drives (previously Vacon), manufacturing frequency converters. The system consists of a fault injector device created with a SoC-FPGA, a testing interface for a PC tool, and an Ethernet-based communication between these two. The device is connected to a serial communication link, and the testing interface makes it easy for a tester to create and run modifiable fault injection tests using a Robot Framework test environment. At the beginning of the thesis, the most common error detection and correction mechanisms in serial communication and properties of SoC-FPGAs, and Robot Framework were studied. Following this, the system was designed with top-down approach, first identifying the main structure of the components, and finally ending up in designing the logic of individual functions. After this, the device and the testing interface were implemented in C and Python using the designed Ethernet communication between them. After the implementation, the system was tested with all the components combined. The actual fault injection logic was not ready by the end of the thesis, so the tests were not run in a real environment. However, the work is done so that the implemented parts can be later used in a complete system. The most important conclusion is that the system was created and tested to meet the requirements with applicable parts. Further development includes creating a complete system and testing it with a real communication link

Certifications of Critical Systems – The CECRIS Experience

In recent years, a considerable amount of effort has been devoted, both in industry and academia, to the development, validation and verification of critical systems, i.e. those systems whose malfunctions or failures reach a critical level both in terms of risks to human life as well as having a large economic impact.Certifications of Critical Systems – The CECRIS Experience documents the main insights on Cost Effective Verification and Validation processes that were gained during work in the European Research Project CECRIS (acronym for Certification of Critical Systems). The objective of the research was to tackle the challenges of certification by focusing on those aspects that turn out to be more difficult/important for current and future critical systems industry: the effective use of methodologies, processes and tools.The CECRIS project took a step forward in the growing field of development, verification and validation and certification of critical systems. It focused on the more difficult/important aspects of critical system development, verification and validation and certification process. Starting from both the scientific and industrial state of the art methodologies for system development and the impact of their usage on the verification and validation and certification of critical systems, the project aimed at developing strategies and techniques supported by automatic or semi-automatic tools and methods for these activities, setting guidelines to support engineers during the planning of the verification and validation phases