Fault-Injection-Based Assessment of Fail-Silence Provided by Process Duplication versus Internal Error Detection in Scientific-Based Applications

Coordinated Science Laboratory was formerly known as Control Systems Laborator

Constructing fail-controlled nodes for distributed systems: a software approach

PhD ThesisDesigning and implementing distributed systems which continue to provide specified services in the presence of processing site and communication failures is a difficult task. To facilitate their development, distributed systems have been built assuming that their underlying hardware components are Jail-controlled, i.e. present a well defined failure mode. However, if conventional hardware cannot provide the assumed failure mode, there is a need to build processing sites or nodes, and communication infra-structure that present the fail-controlled behaviour assumed. Coupling a number of redundant processors within a replicated node is a well known way of constructing fail-controlled nodes. Computation is replicated and executed simultaneously at each processor, and by employing suitable validation techniques to the outputs generated by processors (e.g. majority voting, comparison), outputs from faulty processors can be prevented from appearing at the application level. One way of constructing replicated nodes is by introducing hardwired mechanisms to couple replicated processors with specialised validation hardware circuits. Processors are tightly synchronised at the clock cycle level, and have their outputs validated by a reliable validation hardware. Another approach is to use software mechanisms to perform synchronisation of processors and validation of the outputs. The main advantage of hardware based nodes is the minimum performance overhead incurred. However, the introduction of special circuits may increase the complexity of the design tremendously. Further, every new microprocessor architecture requires considerable redesign overhead. Software based nodes do not present these problems, on the other hand, they introduce much bigger performance overheads to the system. In this thesis we investigate alternative ways of constructing efficient fail-controlled, software based replicated nodes. In particular, we present much more efficient order protocols, which are necessary for the implementation of these nodes. Our protocols, unlike others published to date, do not require processors' physical clocks to be explicitly synchronised. The main contribution of this thesis is the precise definition of the semantics of a software based Jail-silent node, along with its efficient design, implementation and performance evaluation.The Brazilian National Research Council (CNPq/Brasil)

Fault injection testing of software implemented fault tolerance mechanisms of distributed systems

PhD ThesisOne way of gaining confidence in the adequacy of fault tolerance mechanisms of a system is to test the system by injecting faults and see how the system performs under faulty conditions. This thesis investigates the issues of testing software-implemented fault tolerance mechanisms of distributed systems through fault injection. A fault injection method has been developed. The method requires that the target software system be structured as a collection of objects interacting via messages. This enables easy insertion of fault injection objects into the target system to emulate incorrect behaviour of faulty processors by manipulating messages. This approach allows one to inject specific classes of faults while not requiring any significant changes to the target system. The method differs from the previous work in that it exploits an object oriented approach of software implementation to support the injection of specific classes of faults at the system level. The proposed fault injection method has been applied to test software-implemented reliable node systems: a TMR (triple modular redundant) node and a fail-silent node. The nodes have integrated fault tolerance mechanisms and are expected to exhibit certain behaviour in the presence of a failure. The thesis describes how various such mechanisms (for example, clock synchronisation protocol, and atomic broadcast protocol) were tested. The testing revealed flaws in implementation that had not been discovered before, thereby demonstrating the usefulness of the method. Application of the approach to other distributed systems is also described in the thesis.CEC ESPRIT programme, UK Engineering and Physical Sciences Research Council (EPSRC)

Automatic Generation of Distributed System Simulations from UML

Nowadays, an object-oriented approach is commonly used for building computer systems. The benefits of the object-oriented method, such as scalability, stability and reusability, make this method suitable for building complex systems, including those in the distributed system area. A distributed system application usually needs to satisfy quite stringent requirements such as reliability, availability, security, etc. and the cost of building such an application will be quite high. It is therefore desirable to be able to predict the performance of the proposed system before the construction begins. In order to do this, it is important to evaluate the requirements of the new system and translate them into a specification (design). The design process helps the system developers to understand the requirements better as well as to avoid misconceptions about the system. From the specification, a simulation program can be built to mimic the execution of the proposed system. The simulation run provides some data about the states of the system and from these data, the performance of the system can be predicted and analysed. UML (Unified Modeling Language) is one example of the object-oriented design methods that has been widely used for specifying system requirements. There are also some object-oriented simulation languages/packages available, for example, SIMULA or C++SIM package, but it is often difficult to transform the system’ s requirements into a simulation program without sound knowledge of some simulation techniques. On top of that, a new simulation program needs to be built each time for different systems, which can be quite tedious. The currently available UML tools do not provide a feature to generate simulation programs automatically from UML specifications. In this paper, we describe a tool for constructing simulation programs in a generic way, based on a simple specification (preferably in a UML notation) by identifying the simulation components and their structure

Overcoming Byzantine Failures Using Checkpointing

Coordinated Science Laboratory was formerly known as Control Systems LaboratoryDARPA / F30602-00-C-0172 and F30602-02-C-013

Replication of non-deterministic objects

This thesis discusses replication of non-deterministic objects in distributed systems to achieve fault tolerance against crash failures. The objects replicated are the virtual nodes of a distributed application. Replication is viewed as an issue that is to be dealt with only during the configuration of a distributed application and that should not affect the development of the application. Hence, replication of virtual nodes should be transparent to the application. Like all measures to achieve fault tolerance, replication introduces redundancy in the system. Not surprisingly, the main difficulty is guaranteeing the consistency of all replicas such that they behave in the same way as if the object was not replicated (replication transparency). This is further complicated if active objects (like virtual nodes) are replicated, and these objects themselves can be clients of still further objects in the distributed application. The problems of replication of active non-deterministic objects are analyzed in the context of distributed Ada 95 applications. The ISO standard for Ada 95 defines a model for distributed execution based on remote procedure calls (RPC). Virtual nodes in Ada 95 use this as their sole communication paradigm, but they may contain tasks to execute activities concurrently, thus making the execution potentially non-deterministic due to implicit timing dependencies. Such non-determinism cannot be avoided by choosing deterministic tasking policies. I present two different approaches to maintain replica consistency despite this non-determinism. In a first approach, I consider the run-time support of Ada 95 as a black box (except for the part handling remote communications). This corresponds to a non-deterministic computation model. I show that replication of non-deterministic virtual nodes requires that remote procedure calls are implemented as nested transactions. Unfortunately, effects of failures are not local to the replicas of a virtual node: when a failure occurs, nested remote calls made to other virtual nodes must be undone. Also, using transactional semantics for RPCs necessitates a compromise regarding transparency: the application must identify global state for it cannot be determined reliably in an automatic way. Further study reveals that this approach cannot be implemented in a transparent way at all because the consistency criterion of Ada 95 (linearizability) is much weaker than that of transactions (serializability). An execution of remote procedure calls as transactions may thus lead to incompatibilities with the semantics of the programming language. If remotely called subprograms on a replicated virtual node perform partial operations, i.e., entry calls on global protected objects, deadlocks that cannot be broken can occur in certain cases. Such deadlocks do not occur when the virtual node is not replicated. The transactional semantics of RPCs must therefore be exposed to the application. A second approach is based on a piecewise deterministic computation model, i.e., the execution of a virtual node is seen as a sequence of deterministic state intervals. Whenever a non-deterministic event occurs, a new state interval is started. I study replica organization under this computation model (semi-active replication). In this model, all non-deterministic decisions are made on one distinguished replica (the leader), while all other replicas (the followers) are forced to follow the same sequence of non-deterministic events. I show that it suffices to synchronize the followers with the leader upon each observable event, i.e., when the leader sends a message to some other virtual node. It is not necessary to synchronize upon each and every non-deterministic event — which would incur a prohibitively high overhead. Non-deterministic events occurring on the leader between observable events are logged and sent to the followers just before the leader executes an observable event. Consequently, it is guaranteed that the followers will reach the same state as the leader, and thus the effects of failures remain mostly local to the replicas. A prototype implementation called RAPIDS (Replicated Ada Partitions In Distributed Systems) serves as a proof of concept for this second approach, demonstrating its feasibility. RAPIDS is an Ada 95 implementation of a replication manager for semi-active replication for the GNAT development system for Ada 95. It is entirely contained within the run-time support and hence largely transparent for the application

Tolerância a falhas em sistemas de comunicação de tempo-real flexíveis

Nas últimas décadas, os sistemas embutidos distribuídos, têm sido usados em variados domínios de aplicação, desde o controlo de processos industriais até ao controlo de aviões e automóveis, sendo expectável que esta tendência se mantenha e até se intensifique durante os próximos anos. Os requisitos de confiabilidade de algumas destas aplicações são extremamente importantes, visto que o não cumprimento de serviços de uma forma previsível e pontual pode causar graves danos económicos ou até pôr em risco vidas humanas. A adopção das melhores práticas de projecto no desenvolvimento destes sistemas não elimina, por si só, a ocorrência de falhas causadas pelo comportamento não determinístico do ambiente onde o sistema embutido distribuído operará. Desta forma, é necessário incluir mecanismos de tolerância a falhas que impeçam que eventuais falhas possam comprometer todo o sistema. Contudo, para serem eficazes, os mecanismos de tolerância a falhas necessitam ter conhecimento a priori do comportamento correcto do sistema de modo a poderem ser capazes de distinguir os modos correctos de funcionamento dos incorrectos. Tradicionalmente, quando se projectam mecanismos de tolerância a falhas, o conhecimento a priori significa que todos os possíveis modos de funcionamento são conhecidos na fase de projecto, não os podendo adaptar nem fazer evoluir durante a operação do sistema. Como consequência, os sistemas projectados de acordo com este princípio ou são completamente estáticos ou permitem apenas um pequeno número de modos de operação. Contudo, é desejável que os sistemas disponham de alguma flexibilidade de modo a suportarem a evolução dos requisitos durante a fase de operação, simplificar a manutenção e reparação, bem como melhorar a eficiência usando apenas os recursos do sistema que são efectivamente necessários em cada instante. Além disto, esta eficiência pode ter um impacto positivo no custo do sistema, em virtude deste poder disponibilizar mais funcionalidades com o mesmo custo ou a mesma funcionalidade a um menor custo. Porém, flexibilidade e confiabilidade têm sido encarados como conceitos conflituais. Isto deve-se ao facto de flexibilidade implicar a capacidade de permitir a evolução dos requisitos que, por sua vez, podem levar a cenários de operação imprevisíveis e possivelmente inseguros. Desta fora, é comummente aceite que apenas um sistema completamente estático pode ser tornado confiável, o que significa que todos os aspectos operacionais têm de ser completamente definidos durante a fase de projecto. Num sentido lato, esta constatação é verdadeira. Contudo, se os modos como o sistema se adapta a requisitos evolutivos puderem ser restringidos e controlados, então talvez seja possível garantir a confiabilidade permanente apesar das alterações aos requisitos durante a fase de operação. A tese suportada por esta dissertação defende que é possível flexibilizar um sistema, dentro de limites bem definidos, sem comprometer a sua confiabilidade e propõe alguns mecanismos que permitem a construção de sistemas de segurança crítica baseados no protocolo Controller Area Network (CAN). Mais concretamente, o foco principal deste trabalho incide sobre o protocolo Flexible Time-Triggered CAN (FTT-CAN), que foi especialmente desenvolvido para disponibilizar um grande nível de flexibilidade operacional combinando, não só as vantagens dos paradigmas de transmissão de mensagens baseados em eventos e em tempo, mas também a flexibilidade associada ao escalonamento dinâmico do tráfego cuja transmissão é despoletada apenas pela evolução do tempo. Este facto condiciona e torna mais complexo o desenvolvimento de mecanismos de tolerância a falhas para FTT-CAN do que para outros protocolos como por exemplo, TTCAN ou FlexRay, nos quais existe um conhecimento estático, antecipado e comum a todos os nodos, do escalonamento de mensagens cuja transmissão é despoletada pela evolução do tempo. Contudo, e apesar desta complexidade adicional, este trabalho demonstra que é possível construir mecanismos de tolerância a falhas para FTT-CAN preservando a sua flexibilidade operacional. É também defendido nesta dissertação que um sistema baseado no protocolo FTT-CAN e equipado com os mecanismos de tolerância a falhas propostos é passível de ser usado em aplicações de segurança crítica. Esta afirmação é suportada, no âmbito do protocolo FTT-CAN, através da definição de uma arquitectura tolerante a falhas integrando nodos com modos de falha tipo falha-silêncio e nodos mestre replicados. Os vários problemas resultantes da replicação dos nodos mestre são, também eles, analisados e várias soluções são propostas para os obviar. Concretamente, é proposto um protocolo que garante a consistência das estruturas de dados replicadas a quando da sua actualização e um outro protocolo que permite a transferência dessas estruturas de dados para um nodo mestre que se encontre não sincronizado com os restantes depois de inicializado ou reinicializado de modo assíncrono. Além disto, esta dissertação também discute o projecto de nodos FTT-CAN que exibam um modo de falha do tipo falha-silêncio e propõe duas soluções baseadas em componentes de hardware localizados no interface de rede de cada nodo, para resolver este problema. Uma das soluções propostas baseiase em bus guardians que permitem a imposição de comportamento falhasilêncio nos nodos escravos e suportam o escalonamento dinâmico de tráfego na rede. A outra solução baseia-se num interface de rede que arbitra o acesso de dois microprocessadores ao barramento. Este interface permite que a replicação interna de um nodo seja efectuada de forma transparente e assegura um comportamento falha-silêncio quer no domínio temporal quer no domínio do valor ao permitir transmissões do nodo apenas quando ambas as réplicas coincidam no conteúdo das mensagens e nos instantes de transmissão. Esta última solução está mais adaptada para ser usada nos nodos mestre, contudo também poderá ser usada nos nodos escravo, sempre que tal se revele fundamental.Distributed embedded systems (DES) have been widely used in the last few decades in several application fields, ranging from industrial process control to avionics and automotive systems. In fact, it is expectable that this trend will continue over the years to come. In some of these application domains the dependability requirements are of utmost importance since failing to provide services in a timely and predictable manner may cause important economic losses or even put human life in risk. The adoption of the best practices in the design of distributed embedded systems does not fully avoid the occurrence of faults, arising from the nondeterministic behavior of the environment where each particular DES operates. Thus, fault-tolerance mechanisms need to be included in the DES to prevent possible faults leading to system failure. To be effective, fault-tolerance mechanisms require an a priori knowledge of the correct system behavior to be capable of distinguishing them from the erroneous ones. Traditionally, when designing fault-tolerance mechanisms, the a priori knowledge means that all possible operational modes are known at system design time and cannot adapt nor evolve during runtime. As a consequence, systems designed according to this principle are either fully static or allow a small number of operational modes only. Flexibility, however, is a desired property in a system in order to support evolving requirements, simplify maintenance and repair, and improve the efficiency in using system resources by using only the resources that are effectively required at each instant. This efficiency might impact positively on the system cost because with the same resources one can add more functionality or one can offer the same functionality with fewer resources. However, flexibility and dependability are often regarded as conflicting concepts. This is so because flexibility implies the ability to deal with evolving requirements that, in turn, can lead to unpredictable and possibly unsafe operating scenarios. Therefore, it is commonly accepted that only a fully static system can be made dependable, meaning that all operating conditions are completely defined at pre-runtime. In the broad sense and assuming unbounded flexibility this assessment is true, but if one restricts and controls the ways the system could adapt to evolving requirements, then it might be possible to enforce continuous dependability. This thesis claims that it is possible to provide a bounded degree of flexibility without compromising dependability and proposes some mechanisms to build safety-critical systems based on the Controller Area Network (CAN). In particular, the main focus of this work is the Flexible Time-Triggered CAN protocol (FTT-CAN), which was specifically developed to provide such high level of operational flexibility, not only combining the advantages of time- and event-triggered paradigms but also providing flexibility to the time-triggered traffic. This fact makes the development of fault-tolerant mechanisms more complex in FTT-CAN than in other protocols, such as TTCAN or FlexRay, in which there is a priori static common knowledge of the time-triggered message schedule shared by all nodes. Nevertheless, as it is demonstrated in this work, it is possible to build fault-tolerant mechanisms for FTT-CAN that preserve its high level of operational flexibility, particularly concerning the time-triggered traffic. With such mechanisms it is argued that FTT-CAN is suitable for safetycritical applications, too. This claim was validated in the scope of the FTT-CAN protocol by presenting a fault-tolerant system architecture with replicated masters and fail-silent nodes. The specific problems and mechanisms related with master replication, particularly a protocol to enforce consistency during updates of replicated data structures and another protocol to transfer these data structures to an unsynchronized node upon asynchronous startup or restart, are also addressed. Moreover, this thesis also discusses the implementations of fail-silence in FTTCAN nodes and proposes two solutions, both based on hardware components that are attached to the node network interface. One solution relies on bus guardians that allow enforcing fail-silence in the time domain. These bus guardians are adapted to support dynamic traffic scheduling and are fit for use in FTT-CAN slave nodes, only. The other solution relies on a special network interface, with duplicated microprocessor interface, that supports internal replication of the node, transparently. In this case, fail-silence can be assured both in the time and value domain since transmissions are carried out only if both internal nodes agree on the transmission instant and message contents. This solution is well adapted for use in the masters but it can also be used, if desired, in slave nodes

Comunicações confiáveis sem-fios para redes veiculares

Vehicular communications are a promising field of research, with numerous potential services that can enhance traffic experience. Road safety is the most important objective behind the development of wireless vehicular networks, since many of the current accidents and fatalities could be avoided if vehicles had the ability to share information among them, with the road-side infrastructure and other road users. A future with safe, efficient and comfortable road transportation systems is envisaged by the different traffic stakeholders - users, manufacturers, road operators and public authorities. Cooperative Intelligent Transportation Systems (ITS) applications will contribute to achieve this goal, as well as other technological progress, such as automated driving or improved road infrastructure based on advanced sensoring and the Internet of Things (IoT) paradigm. Despite these significant benefits, the design of vehicular communications systems poses difficult challenges, mainly due to the very dynamic environments in which they operate. In order to attain the safety-critical requirements involved in this type of scenarios, careful planning is necessary, so that a trustworthy behaviour of the system can be achieved. Dependability and real-time systems concepts provide essential tools to handle this challenging task of enabling determinism and fault-tolerance in vehicular networks. This thesis aims to address some of these issues by proposing architectures and implementing mechanisms that improve the dependability levels of realtime vehicular communications. The developed strategies always try to preserve the required system’s flexibity, a fundamental property in such unpredictable scenarios, where unexpected events may occur and force the system to quickly adapt to the new circumnstances.The core contribution of this thesis focuses on the design of a fault-tolerant architecture for infrastructure-based vehicular networks. It encompasses a set of mechanisms that allow error detection and fault-tolerant behaviour both in the mobile and static nodes of the network. Road-side infrastructure plays a key role in this context, since it provides the support for coordinating all communications taking place in the wireless medium. Furthermore, it is also responsible for admission control policies and exchanging information with the backbone network. The proposed methods rely on a deterministic medium access control (MAC) protocol that provides real-time guarantees in wireless channel access, ensuring that communications take place before a given deadline. However, the presented solutions are generic and can be easily adapted to other protocols and wireless technologies. Interference mitigation techniques, mechanisms to enforce fail-silent behaviour and redundancy schemes are introduced in this work, so that vehicular communications systems may present higher dependability levels. In addition to this, all of these methods are included in the design of vehicular network components, guaranteeing that the real-time constraints are still fulfilled. In conclusion, wireless vehicular networks hold the potential to drastically improve road safety. However, these systems should present dependable behaviour in order to reliably prevent the occurrence of catastrophic events under all possible traffic scenarios.As comunicações veiculares são uma área de investigação bastante promissora, com inúmeros potenciais serviços que podem melhorar a experiência vivida no tráfego. A segurança rodoviária é o objectivo mais importante por detrás do desenvolvimento das redes veiculares sem-fios, visto que muitos dos atuais acidentes e vítimas mortais poderiam ser evitados caso os veículos tivessem a capacidade de trocar informação entre eles, com a infraestrutura rodoviária e outros utilizadores da estrada. Um futuro com sistemas de transporte rodoviário seguros, eficientes e confortáveis é algo ambicionado pelas diferentes partes envolvidas - utilizadores, fabricantes, operadores da infraestrutura e autoridades públicas. As aplicações de Sistemas Inteligentes de Transporte (ITS) cooperativas vão contribuir para alcançar este propósito, em conjunto com outros avanços tecnológicos, nomeadamente a condução autónoma ou uma melhor infraestrutura rodoviária baseada em sensorização avançada e no paradigma da Internet das Coisas (IoT). Apesar destes benefícios significativos, o desenho de sistemas de comunicações veiculares coloca desafios difíceis, em grande parte devido aos ambientes extremamente dinâmicos em que estes operam. De modo a atingir os requisitos de segurança crítica envolvidos neste tipo de cenários, é necessário um cuidadoso planeamento por forma a que o sistema apresente um comportamento confiável. Conceitos de dependabilidade e de sistemas de tempo-real constituem ferramentas essenciais para lidar com esta desafiante tarefa de dotar as redes veiculares de determinismo e tolerância a faltas. Esta tese pretende endereçar alguns destes problemas através da proposta de arquitecturas e da implementação de mecanismos que melhorem os níveis da dependabilidade das comunicações veiculares de tempo-real. As estratégias desenvolvidas tentam sempre preservar a necessária flexibilidade do sistema, uma propriedade fundamental em cenários tão imprevisíveis, onde eventos inesperados podem ocorrer e forçar o sistema a adaptar-se rapidamente às novas circunstâncias.A contribuição principal desta tese foca-se no desenho de uma arquitectura tolerante a faltas para redes veiculares com suporte da infraestrutura de beira de estrada. Esta arquitectura engloba um conjunto de mecanismos que permite detecção de erros e comportamento tolerante a faltas, tanto nos nós móveis como nos nós estáticos da rede. A infraestrutura de beira de estrada desempenha um papel fundamental neste contexto, pois fornece o suporte que permite coordenar todas as comunicações que ocorrem no meio sem-fios. Para além disso, é também responsável pelos mecanismos de controlo de admissão e pela troca de informação com a rede de transporte. Os métodos propostos baseiam-se num protocolo determinístico de controlo de acesso ao meio (MAC) que fornece garantias de tempo-real no accesso ao canal semfios, assegurando que as comunicações ocorrem antes de um determinado limite temporal. No entanto, as soluções apresentadas são genéricas e podem ser facilmente adaptadas a outros protocolos e tecnologias sem-fios. Neste trabalho são introduzidas técnicas de mitigação de interferência, mecanismos para assegurar comportamento falha-silêncio e esquemas de redundância, de modo a que os sistemas de comunicações veiculares apresentem elevados níveis de dependabilidade. Além disso, todos estes métodos são incorporados no desenho dos componentes da rede veicular, guarantindo que as restrições de tempo-real continuam a ser cumpridas. Em suma, as redes veiculares sem-fios têm o potential para melhorar drasticamente a segurança rodoviária. Contudo, estes sistemas precisam de apresentar um comportamento confiável, de forma a prevenir a ocorrência de eventos catastróficos em todos os cenários de tráfego possíveis.Programa Doutoral em Telecomunicaçõe

Combatting Advanced Persistent Threat via Causality Inference and Program Analysis

Cyber attackers are becoming more and more sophisticated. In particular, Advanced Persistent Threat (APT) is a new class of attack that targets a specifc organization and compromises systems over a long time without being detected. Over the years, we have seen notorious examples of APTs including Stuxnet which disrupted Iranian nuclear centrifuges and data breaches affecting millions of users. Investigating APT is challenging as it occurs over an extended period of time and the attack process is highly sophisticated and stealthy. Also, preventing APTs is diffcult due to ever-expanding attack vectors. In this dissertation, we present proposals for dealing with challenges in attack investigation. Specifcally, we present LDX which conducts precise counter-factual causality inference to determine dependencies between system calls (e.g., between input and output system calls) and allows investigators to determine the origin of an attack (e.g., receiving a spam email) and the propagation path of the attack, and assess the consequences of the attack. LDX is four times more accurate and two orders of magnitude faster than state-of-the-art taint analysis techniques. Moreover, we then present a practical model-based causality inference system, MCI, which achieves precise and accurate causality inference without requiring any modifcation or instrumentation in end-user systems. Second, we show a general protection system against a wide spectrum of attack vectors and methods. Specifcally, we present A2C that prevents a wide range of attacks by randomizing inputs such that any malicious payloads contained in the inputs are corrupted. The protection provided by A2C is both general (e.g., against various attack vectors) and practical (7% runtime overhead)