Fault Injection for Embedded Microprocessor-based Systems

Microprocessor-based embedded systems are increasingly used to control safety-critical systems (e.g., air and railway traffic control, nuclear plant control, aircraft and car control). In this case, fault tolerance mechanisms are introduced at the hardware and software level. Debugging and verifying the correct design and implementation of these mechanisms ask for effective environments, and Fault Injection represents a viable solution for their implementation. In this paper we present a Fault Injection environment, named FlexFI, suitable to assess the correctness of the design and implementation of the hardware and software mechanisms existing in embedded microprocessor-based systems, and to compute the fault coverage they provide. The paper describes and analyzes different solutions for implementing the most critical modules, which differ in terms of cost, speed, and intrusiveness in the original system behavio

Study and development of a software implemented fault injection plug-in for the Xception tool/powerPC 750

Estágio realizado na Critical Software e orientado pelo Eng.º Ricardo BarbosaTese de mestrado integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto. 200

Testing communication reliability with fault injection : Implementation using Robot Framework and SoC-FPGA

Taajuusmuuttajia käytetään teollisuudessa laajasti, sillä merkittävän osan teollisuuden sähkönkulutuksesta muodostavat oikosulkumoottorit, joita ajetaan taajuusmuuttajien avulla. Taajuusmuuttajiin on mahdollista kytkeä optiokortteja, jotka lisäävät taajuusmuuttajaan valvonta-, ohjaus- ym. toiminnallisuuksia. Nämä kortit kommunikoivat sarjaliikenneväylän kautta taajuusmuuttajan pääyksikön kanssa. Sarjaliikennelinkissä, kuten taajuusmuuttajan väylällä, voi syntyä virheitä, jotka häiritsevät tietoliikennettä. Sen takia sarjaliikenneprotokolliin on luotu virheentunnistus- ja -korjausmekanismeja, joilla pyritään varmistamaan virheetön tiedon kuljettaminen. Luotettavuutta testaamaan voidaan väylälle generoida virheitä siihen tarkoitetulla laitteella. Tässä diplomityössä luotiin taajuusmuuttajia valmistavan yrityksen, Danfoss Drivesin (aik. Vacon), pyynnöstä häiriögeneraattorijärjestelmä. Järjestelmä koostuu SoC-FPGA-piirillä luodusta virheitä syöttävästä laitteesta, PC-työkalulle luodusta testirajapinnasta sekä Ethernet-kommunikaatiosta niiden välillä. Laite kytketään väylään, ja testirajapinta tekee testaajalle mahdolliseksi luoda mukautettavia testejä ja ajaa testejä käyttäen Robot Framework -testiympäristöä. Diplomityössä tutkittiin ensin sarjakommunikointiväylien yleisimpiä virheentunnistus- ja korjauskeinoja sekä SoC-FPGA-piirien sekä työssä käytetyn Robot Frameworkin ominaisuuksia. Järjestelmä suunniteltiin ylhäältä-alas-periaatteella ensin tunnistamalla kolmen edellä mainitun komponentin päärakenne päätyen lopulta yksittäisten ohjelmafunktioiden logiikan suunnitteluun. Tämän jälkeen laite ja testirajapinta toteutettiin C- ja Python-ohjelmointikielillä käyttäen suunnitellun kaltaista kommunikaatiota näiden kahden komponentin välillä. Lopulta järjestelmä testattiin kaikki komponentit yhteen kytkettynä. Varsinainen injektorilogiikka, joka luo virheitä väylään, ei ollut työn loppuun mennessä vielä toimittavan tahon puolelta valmis, joten järjestelmää ei voitu testata todellisessa ympäristössä. Työssä luodut osuudet voidaan kuitenkin myöhemmin kytkeä kokonaiseen järjestelmään. Työn tärkeimpänä johtopäätöksenä on, että tavoitteiden mukainen järjestelmä saatiin luotua ja testattua toimivaksi mahdollisin osin. Jatkokehityskohteeksi jäi mm. kokonaisen järjestelmän luonti ja testaus oikeaan kommunikaatioväylään kytkettynä.Frequency converters are widely used in industry because a notable part of the industrial electricity consumption is by electrical induction motors driven by frequency converters. It is possible to connect option boards into a frequency converter to add monitoring and control features. These option boards communicate with the main control unit of the frequency converter over a serial communication link. In a serial communication link, e.g. in a frequency converter, it can occur faults that interfere with the transfer. Hence, error handling mechanisms are used to secure transmission of the data without errors. A fault injector device, which generates errors into the data travelling in the link, can be used to test the communication reliability. In this master’s thesis, an error generator system was created for a company, Danfoss Drives (previously Vacon), manufacturing frequency converters. The system consists of a fault injector device created with a SoC-FPGA, a testing interface for a PC tool, and an Ethernet-based communication between these two. The device is connected to a serial communication link, and the testing interface makes it easy for a tester to create and run modifiable fault injection tests using a Robot Framework test environment. At the beginning of the thesis, the most common error detection and correction mechanisms in serial communication and properties of SoC-FPGAs, and Robot Framework were studied. Following this, the system was designed with top-down approach, first identifying the main structure of the components, and finally ending up in designing the logic of individual functions. After this, the device and the testing interface were implemented in C and Python using the designed Ethernet communication between them. After the implementation, the system was tested with all the components combined. The actual fault injection logic was not ready by the end of the thesis, so the tests were not run in a real environment. However, the work is done so that the implemented parts can be later used in a complete system. The most important conclusion is that the system was created and tested to meet the requirements with applicable parts. Further development includes creating a complete system and testing it with a real communication link

Mixed-Criticality Systems on Commercial-Off-the-Shelf Multi-Processor Systems-on-Chip

Avionics and space industries are struggling with the adoption of technologies like multi-processor system-on-chips (MPSoCs) due to strict safety requirements. This thesis propose a new reference architecture for MPSoC-based mixed-criticality systems (MCS) - i.e., systems integrating applications with different level of criticality - which are a common use case for aforementioned industries. This thesis proposes a system architecture capable of granting partitioning - which is, for short, the property of fault containment. It is based on the detection of spatial and temporal interference, and has been named the online detection of interference (ODIn) architecture. Spatial partitioning requires that an application is not able to corrupt resources used by a different application. In the architecture proposed in this thesis, spatial partitioning is implemented using type-1 hypervisors, which allow definition of resource partitions. An application running in a partition can only access resources granted to that partition, therefore it cannot corrupt resources used by applications running in other partitions. Temporal partitioning requires that an application is not able to unexpectedly change the execution time of other applications. In the proposed architecture, temporal partitioning has been solved using a bounded interference approach, composed of an offline analysis phase and an online safety net. The offline phase is based on a statistical profiling of a metric sensitive to temporal interference’s, performed in nominal conditions, which allows definition of a set of three thresholds: 1. the detection threshold TD; 2. the warning threshold TW ; 3. the α threshold. Two rules of detection are defined using such thresholds: Alarm rule When the value of the metric is above TD. Warning rule When the value of the metric is in the warning region [TW ;TD] for more than α consecutive times. ODIn’s online safety-net exploits performance counters, available in many MPSoC architectures; such counters are configured at bootstrap to monitor the selected metric(s), and to raise an interrupt request (IRQ) in case the metric value goes above TD, implementing the alarm rule. The warning rule is implemented in a software detection module, which reads the value of performance counters when the monitored task yields control to the scheduler and reset them if there is no detection. ODIn also uses two additional detection mechanisms: 1. a control flow check technique, based on compile-time defined block signatures, is implemented through a set of watchdog processors, each monitoring one partition. 2. a timeout is implemented through a system watchdog timer (SWDT), which is able to send an external signal when the timeout is violated. The recovery actions implemented in ODIn are: • graceful degradation, to react to IRQs of WDPs monitoring non-critical applications or to warning rule violations; it temporarily stops non-critical applications to grant resources to the critical application; • hard recovery, to react to the SWDT, to the WDP of the critical application, or to alarm rule violations; it causes a switch to a hot stand-by spare computer. Experimental validation of ODIn was performed on two hardware platforms: the ZedBoard - dual-core - and the Inventami board - quad-core. A space benchmark and an avionic benchmark were implemented on both platforms, composed by different modules as showed in Table 1 Each version of the final application was evaluated through fault injection (FI) campaigns, performed using a specifically designed FI system. There were three types of FI campaigns: 1. HW FI, to emulate single event effects; 2. SW FI, to emulate bugs in non-critical applications; 3. artificial bug FI, to emulate a bug in non-critical applications introducing unexpected interference on the critical application. Experimental results show that ODIn is resilient to all considered types of faul

Vulnerability detection in device drivers

Tese de doutoramento, Informática (Ciência da Computação), Universidade de Lisboa, Faculdade de Ciências, 2017The constant evolution in electronics lets new equipment/devices to be regularly made available on the market, which has led to the situation where common operating systems (OS) include many device drivers(DD) produced by very diverse manufactures. Experience has shown that the development of DD is error prone, as a majority of the OS crashes can be attributed to flaws in their implementation. This thesis addresses the challenge of designing methodologies and tools to facilitate the detection of flaws in DD, contributing to decrease the errors in this kind of software, their impact in the OS stability, and the security threats caused by them. This is especially relevant because it can help developers to improve the quality of drivers during their implementation or when they are integrated into a system. The thesis work started by assessing how DD flaws can impact the correct execution of the Windows OS. The employed approach used a statistical analysis to obtain the list of kernel functions most used by the DD, and then automatically generated synthetic drivers that introduce parameter errors when calling a kernel function, thus mimicking a faulty interaction. The experimental results showed that most targeted functions were ineffective in the defence of the incorrect parameters. A reasonable number of crashes and a small number of hangs were observed suggesting a poor error containment capability of these OS functions. Then, we produced an architecture and a tool that supported the automatic injection of network attacks in mobile equipment (e.g., phone), with the objective of finding security flaws (or vulnerabilities) in Wi-Fi drivers. These DD were selected because they are of easy access to an external adversary, which simply needs to create malicious traffic to exploit them, and therefore the flaws in their implementation could have an important impact. Experiments with the tool uncovered a previously unknown vulnerability that causes OS hangs, when a specific value was assigned to the TIM element in the Beacon frame. The experiments also revealed a potential implementation problem of the TCP-IP stack by the use of disassociation frames when the target device was associated and authenticated with a Wi-Fi access point. Next, we developed a tool capable of registering and instrumenting the interactions between a DD and the OS. The solution used a wrapper DD around the binary of the driver under test, enabling full control over the function calls and parameters involved in the OS-DD interface. This tool can support very diverse testing operations, including the log of system activity and to reverse engineer the driver behaviour. Some experiments were performed with the tool, allowing to record the insights of the behaviour of the interactions between the DD and the OS, the parameter values and return values. Results also showed the ability to identify bugs in drivers, by executing tests based on the knowledge obtained from the driver’s dynamics. Our final contribution is a methodology and framework for the discovery of errors and vulnerabilities in Windows DD by resorting to the execution of the drivers in a fully emulated environment. This approach is capable of testing the drivers without requiring access to the associated hardware or the DD source code, and has a granular control over each machine instruction. Experiments performed with Off the Shelf DD confirmed a high dependency of the correctness of the parameters passed by the OS, identified the precise location and the motive of memory leaks, the existence of dormant and vulnerable code.A constante evolução da eletrónica tem como consequência a disponibilização regular no mercado de novos equipamentos/dispositivos, levando a uma situação em que os sistemas operativos (SO) mais comuns incluem uma grande quantidade de gestores de dispositivos (GD) produzidos por diversos fabricantes. A experiência tem mostrado que o desenvolvimento dos GD é sujeito a erros uma vez que a causa da maioria das paragens do SO pode ser atribuída a falhas na sua implementação. Esta tese centra-se no desafio da criação de metodologias e ferramentas que facilitam a deteção de falhas nos GD, contribuindo para uma diminuição nos erros neste tipo de software, o seu impacto na estabilidade do SO, e as ameaças de segurança por eles causadas. Isto é especialmente relevante porque pode ajudar a melhorar a qualidade dos GD tanto na sua implementação como quando estes são integrados em sistemas. Este trabalho inicia-se com uma avaliação de como as falhas nos GD podem levar a um funcionamento incorreto do SO Windows. A metodologia empregue usa uma análise estatística para obter a lista das funções do SO que são mais utilizadas pelos GD, e posteriormente constrói GD sintéticos que introduzem erros nos parâmetros passados durante a chamada às funções do SO, e desta forma, imita a integração duma falta. Os resultados das experiências mostraram que a maioria das funções testadas não se protege eficazmente dos parâmetros incorretos. Observou-se a ocorrência de um número razoável de paragens e um pequeno número de bloqueios, o que sugere uma pobre capacidade das funções do SO na contenção de erros. Posteriormente, produzimos uma arquitetura e uma ferramenta que suporta a injeção automática de ataques em equipamentos móveis (e.g., telemóveis), com o objetivo de encontrar falhas de segurança (ou vulnerabilidades) em GD de placas de rede Wi-Fi. Estes GD foram selecionados porque são de fácil acesso a um atacante remoto, o qual apenas necessita de criar tráfego malicioso para explorar falhas na sua implementação podendo ter um impacto importante. As experiências realizadas com a ferramenta revelaram uma vulnerabilidade anteriormente desconhecida que provoca um bloqueio no SO quando é atribuído um valor específico ao campo TIM da mensagem de Beacon. As experiências também revelaram um potencial problema na implementação do protocolo TCP-IP no uso das mensagens de desassociação quando o dispositivo alvo estava associado e autenticado com o ponto de acesso Wi-Fi. A seguir, desenvolvemos uma ferramenta com a capacidade de registar e instrumentar as interações entre os GD e o SO. A solução usa um GD que envolve o código binário do GD em teste, permitindo um controlo total sobre as chamadas a funções e aos parâmetros envolvidos na interface SO-GD. Esta ferramenta suporta diversas operações de teste, incluindo o registo da atividade do sistema e compreensão do comportamento do GD. Foram realizadas algumas experiências com esta ferramenta, permitindo o registo das interações entre o GD e o SO, os valores dos parâmetros e os valores de retorno das funções. Os resultados mostraram a capacidade de identificação de erros nos GD, através da execução de testes baseados no conhecimento da dinâmica do GD. A nossa contribuição final é uma metodologia e uma ferramenta para a descoberta de erros e vulnerabilidades em GD Windows recorrendo à execução do GD num ambiente totalmente emulado. Esta abordagem permite testar GD sem a necessidade do respetivo hardware ou o código fonte, e possuí controlo granular sobre a execução de cada instrução máquina. As experiências realizadas com GD disponíveis comercialmente confirmaram a grande dependência que os GD têm nos parâmetros das funções do SO, e identificaram o motivo e a localização precisa de fugas de memória, a existência de código não usado e vulnerável

Automated performance attack discovery in distributed system implementations

Security and performance are critical goals for distributed systems. The increased complexity in design, incomplete expertise of developers, and limited functionality of existing testing tools often result in implementations with vulnerabilities and make the debugging process difficult and costly. The deployed vulnerabilities are often exploited by adversaries preventing the system from achieving its design goals. We refer to attacks that slow down the performance of a system as performance attacks. In the past, finding performance attacks has been a painstaking manual process that involved an expert of the target implementation. Given the cost associated with each vulnerability that occurs in the production, there is a need for tools to automatically check that the implementation of a protocol achieves its performance goals with respect to malicious components in the system. In this dissertation, we find performance attacks automatically from implementations of distributed systems. We do not try to show that an implementation is free from all attacks. Our goal is to find attacks and report them to the user in a timely manner. We first investigate how to find attacks automatically from implementations under a simulated environment. A simulated approach, however, has a fundamental limitation in terms of applicable target systems, as certain assumptions are made about languages, operating systems or libraries used. Therefore, we next investigate challenges and requirements to automatically find attacks in implementations of distributed systems under an emulated environment where no limiting assumptions are made

Dependability analysis of web services

Web Services form the basis of the web based eCommerce eScience applications so it is vital that robust services are developed. Traditional validation and verification techniques are centred around the concept of removing all faults to guarantee correct operation whereas Dependability gives an assessment of how dependably a system can deliver the required functionality by assessing attributes, and by eliminating threats via means attempts to improve dependability. Fault injection is a well-proven dependability assessment method. Although much work has been done in the area of fault injection and distributed systems in general, there appears to have been little research carried out on applying this to middleware systems and Web Services in particular. There are additional problems associated with applying existing fault injection technologies to Web Services running in a virtual machine environment since most are either invasive or work at a machine level. The Fault Injection Technology (FIT) method has been devised to address these problems for middleware systems. The Web Service-Fault Injection Technology (WS-FIT) implementation applies the FIT method, based on network level fault injection, to Web Services to create a non-invasive dependability assessment method. It allows targeted perturbation of Web Service RFC parameters as well as more traditional network level fault injection operations. The WS-FIT tool includes taxonomies that define a system under test, fault models to apply and failure modes to be detected, and uses these taxonomies to generate fault injection campaigns. WS-FIT has been applied to a number of case studies and has successfully demonstrated its effectiveness. It has also been successfully applied to a third-party system to evaluate dependability means. It performed this dependability assessment as well as allowing debugging of the means to be undertaken uncovering unknown faults