Scheduling and locking in multiprocessor real-time operating systems

With the widespread adoption of multicore architectures, multiprocessors are now a standard deployment platform for (soft) real-time applications. This dissertation addresses two questions fundamental to the design of multicore-ready real-time operating systems: (1) Which scheduling policies offer the greatest flexibility in satisfying temporal constraints; and (2) which locking algorithms should be used to avoid unpredictable delays? With regard to Question 1, LITMUSRT, a real-time extension of the Linux kernel, is presented and its design is discussed in detail. Notably, LITMUSRT implements link-based scheduling, a novel approach to controlling blocking due to non-preemptive sections. Each implemented scheduler (22 configurations in total) is evaluated under consideration of overheads on a 24-core Intel Xeon platform. The experiments show that partitioned earliest-deadline first (EDF) scheduling is generally preferable in a hard real-time setting, whereas global and clustered EDF scheduling are effective in a soft real-time setting. With regard to Question 2, real-time locking protocols are required to ensure that the maximum delay due to priority inversion can be bounded a priori. Several spinlock- and semaphore-based multiprocessor real-time locking protocols for mutual exclusion (mutex), reader-writer (RW) exclusion, and k-exclusion are proposed and analyzed. A new category of RW locks suited to worst-case analysis, termed phase-fair locks, is proposed and three efficient phase-fair spinlock implementations are provided (one with few atomic operations, one with low space requirements, and one with constant RMR complexity). Maximum priority-inversion blocking is proposed as a natural complexity measure for semaphore protocols. It is shown that there are two classes of schedulability analysis, namely suspension-oblivious and suspension-aware analysis, that yield two different lower bounds on blocking. Five asymptotically optimal locking protocols are designed and analyzed: a family of mutex, RW, and k-exclusion protocols for global, partitioned, and clustered scheduling that are asymptotically optimal in the suspension-oblivious case, and a mutex protocol for partitioned scheduling that is asymptotically optimal in the suspension-aware case. A LITMUSRT-based empirical evaluation is presented that shows these protocols to be practical

Performance modelling of replication protocols

PhD ThesisThis thesis is concerned with the performance modelling of data replication protocols. Data replication is used to provide fault tolerance and to improve the performance of a distributed system. Replication not only needs extra storage but also has an extra cost associated with it when performing an update. It is not always clear which algorithm will give best performance in a given scenario, how many copies should be maintained or where these copies should be located to yield the best performance. The consistency requirements also change with application. One has to choose these parameters to maximize reliability and speed and minimize cost. A study showing the effect of change in different parameters on the performance of these protocols would be helpful in making these decisions. With the use of data replication techniques in wide-area systems where hundreds or even thousands of sites may be involved, it has become important to evaluate the performance of the schemes maintaining copies of data. This thesis evaluates the performance of replication protocols that provide differ- ent levels of data consistency ranging from strong to weak consistency. The protocols that try to integrate strong and weak consistency are also examined. Queueing theory techniques are used to evaluate the performance of these protocols. The performance measures of interest are the response times of read and write jobs. These times are evaluated both when replicas are reliable and when they are subject to random breakdowns and repairs.Commonwealth Scholarshi

Dependence-driven techniques in system design

Burstiness in workloads is often found in multi-tier architectures, storage systems, and communication networks. This feature is extremely important in system design because it can significantly degrade system performance and availability. This dissertation focuses on how to use knowledge of burstiness to develop new techniques and tools for performance prediction, scheduling, and resource allocation under bursty workload conditions.;For multi-tier enterprise systems, burstiness in the service times is catastrophic for performance. Via detailed experimentation, we identify the cause of performance degradation on the persistent bottleneck switch among various servers. This results in an unstable behavior that cannot be captured by existing capacity planning models. In this dissertation, beyond identifying the cause and effects of bottleneck switch in multi-tier systems, we also propose modifications to the classic TPC-W benchmark to emulate bursty arrivals in multi-tier systems.;This dissertation also demonstrates how burstiness can be used to improve system performance. Two dependence-driven scheduling policies, SWAP and ALoC, are developed. These general scheduling policies counteract burstiness in workloads and maintain high availability by delaying selected requests that contribute to burstiness. Extensive experiments show that both SWAP and ALoC achieve good estimates of service times based on the knowledge of burstiness in the service process. as a result, SWAP successfully approximates the shortest job first (SJF) scheduling without requiring a priori information of job service times. ALoC adaptively controls system load by infinitely delaying only a small fraction of the incoming requests.;The knowledge of burstiness can also be used to forecast the length of idle intervals in storage systems. In practice, background activities are scheduled during system idle times. The scheduling of background jobs is crucial in terms of the performance degradation of foreground jobs and the utilization of idle times. In this dissertation, new background scheduling schemes are designed to determine when and for how long idle times can be used for serving background jobs, without violating predefined performance targets of foreground jobs. Extensive trace-driven simulation results illustrate that the proposed schemes are effective and robust in a wide range of system conditions. Furthermore, if there is burstiness within idle times, then maintenance features like disk scrubbing and intra-disk data redundancy can be successfully scheduled as background activities during idle times

Sharing Non-Processor Resources in Multiprocessor Real-Time Systems

Computing devices are increasingly being leveraged in cyber-physical systems, in which computing devices sense, control, and interact with the physical world. Associated with many such real-world interactions are strict timing constraints, which if unsatisfied, can lead to catastrophic consequences. Modern examples of such timing constraints are prevalent in automotive systems, such as airbag controllers, anti-lock brakes, and new autonomous features. In all of these examples, a failure to correctly respond to an event in a timely fashion could lead to a crash, damage, injury and even loss of life. Systems with imperative timing constraints are called real-time systems, and are broadly the subject of this dissertation. Much previous work on real-time systems and scheduling theory assumes that computing tasks are independent, i.e., the only resource they share is the platform upon which they are executed. In practice, however, tasks share many resources, ranging from more overt resources such as shared memory objects, to less overt ones, including data buses and other hardware and I/O devices. Accesses to some such resources must be synchronized to ensure safety, i.e., logical correctness, while other resources may exhibit better run-time performance if accesses are explicitly synchronized. The goal of this dissertation was to develop new synchronization algorithms and associated analysis techniques that can be used to synchronize access to many classes of resources, while improving the overall resource utilization, specifically as measured by real-time schedulability. Towards that goal, the Real-Time Nested Locking Protocol (RNLP), the first multiprocessor real-time locking protocol that supports lock nesting or fine-grained locking is proposed and analyzed. Furthermore, the RNLP is extended to support reader/writer locking, as well as k-exclusion locking. All presented RNLP variants are proven optimal. Furthermore, experimental results demonstrate the schedulability-related benefits of the RNLP. Additionally, three new synchronization algorithms are presented, which are specifically motivated by the need to manage shared hardware resources to improve real-time predictability. Furthermore, two new classes of shared resources are defined, and the first synchronization algorithms for them are proposed. To analyze these new algorithms, a novel analysis technique called idleness analysis is presented, which can be used to incorporate the effects of blocking into schedulability analysis.Doctor of Philosoph

Cooperative testing of multitihreaded Java applications

Tese de mestrado, Engenharia Informática (Engenharia de Software), Universidade de Lisboa, Faculdade de Ciências, 2014Para desenvolver programas concorrentes, o modelo de programação prevalente é o uso de threads. Uma thread define uma linha de execução sequencial num programa, em que partilha o espaço de memória com todas as outras threads que estão a executar o programa ao mesmo tempo. Além de memória partilhada, as threads de um programa também interagem mediante primitivas de concorrência como locks ou barreiras. O uso de threads é suportado por alguma das linguagens de programação mais conhecidas e utilizadas, como por exemplo, o Java ou C. Em programas multithreaded, erros de programação são fáceis de cometer, com o uso de memória partilhada e primitivas de interacção. Por exemplo, a partilha de memória entre as threads pode levar a condições de corrida sobre os dados (data races), e inerente inconsistência de valores para os mesmos. Outro problema típico é a ocorrência de impasses (deadlocks) no uso de primitivas de concorrência, como locks, que não permitem que uma ou mais threads progridem na sua execução. Quando um bug se manifesta, um programador normalmente tenta determinar manualmente a sua origem, através de t´técnicas de debugging, e/ou executa o conjunto de testes associados ao programa, observando em quais testes são reportados erros. Frequentemente, estas duas abordagens falham para programas multithreaded. A razão para estas dificuldades é que numa execução¸ com múltiplas threads as mudanças de contexto acontecem de forma não-determinista, o que torna impossível observar e controlar de forma precisa o fluxo de execução¸ e escalonamento de threads associado. Ao mesmo tempo, os bugs tipicamente apenas se manifestam num sub-conjunto de todos os escalonamentos possíveis, e com frequência apenas em escalonamentos bastante específicos e difíceis de reproduzir. Um teste de software pode ser repetido imensas vezes, sem expor um bug. Se e quando este é detectado, pode ser extremamente difícil identificar e replicar o fluxo de execução¸ ocorrido. Até padrões de bugs relativamente simples são difíceis de detectar e replicar com precisão (por exemplo, ver [9]). Esta tese trata do problema geral de testar programas concorrentes escritos em Java [16], de modo a que bugs possam ser descobertos e replicados de forma determinista. A nossa abordagem passa pela execução¸ de testes de software com semântica cooperativa [23, 24]. A observação fundamental subjacente ao uso de semântica cooperativa é que as mudanças de contexto relevantes entre threads acontecem apenas nos pontos de potencial interferência entre estas, tais como o acesso a dados partilhados ou o uso das primitivas para concorrência, chamados yield points cooperativos. Uma execução¸ é considerada cooperativa se o código entre dois yield points consecutivos da mesma thread executa de forma sequencial como uma transação, sem qualquer interferência de outras threads. A semântica cooperativa pode ser explorada para testes de software deterministas, em que a reproducibilidade de execução é garantida, e a cobertura sistemática do espaço de estados de escalonamento possa ser efectuada. A ideia base é executar as threads com semântica cooperativa, por forma a que a execução repetida de um teste possa explorar o espaço de estados de forma controlada e customizada, e até eventualmente exaustiva. Esta técnica é usada em algumas ferramentas [3, 7, 17] para programas concorrentes escritos em C/C++. Nesta tese nós apresentamos o desenho, implementação e avaliação de uma ferramenta de testes cooperativa para programas concorrentes escritos em Java. A ferramenta desenvolvida chama-se Cooperari e está disponível em https://bitbucket.org/edrdo/Cooperari . Tanto quanto pudemos determinar pela avaliação do estado da arte, trata-se da primeira ferramenta de testes cooperativos para programas em Java. As contribuições desta tese são as seguintes:• A ferramenta instrumenta o bytecode Java da aplicação, por forma a interceptar yield points na execução, e a tornar a execução cooperativa. Os yield points suportados concernem operações sobre monitores Java (operações de locking e primitivas de notificação), operações do ciclo de vida de uma thread definidos na API java.lang.Thread, e ainda acessos (de leitura e escrita) a campos de objectos ou posições de vectores. A instrumentação dos yield points é definida recorrendo à linguagem AspectJ [14]. • Para a execução cooperativa definimos um ambiente de execução, que compreende um escalonador (scheduler) cooperativo e uma implementação cooperativa de monitores Java e operações do ciclo de vida de uma thread. O escalonador cooperativo condiciona o escalonador nativo da máquina virtual Java (que se mantém activo), por forma a se obter uma semântica cooperativa na execução. Aquando da intercepcão de um yield point, o escalonador cooperativo toma controlo da execução, e deterministicamente seleciona a próxima thread que irá executar.• Desenvolvimento de duas políticas de cobertura. A escolha feita num passo de escalonamento é determinado pela política de cobertura do espaço de estados de escalonamentos. A primeira delas simplesmente escolhe a próxima thread a executar de forma pseudo-aleatória, sem manter estado entre várias execuções de um programa/teste; para execução determinista, o gerador de números aleatórios é inicializado com uma semente fixa. A outra política mantém um histórico de decisões de escalonamento anteriores, por forma a evitar decisões repetidas de escalonamento para o mesmo estado do programa.• Implementação de mecanismos de monitorização em tempo de execução para a deteção de data races e alguns tipos de deadlock. É assinalado um erro quando na monitorização do acesso aos dados ´e verificado que para a mesma variável ou posição do vector está a ser acedido por duas threads, em que uma delas está a escrever e a outra está a escrever ou ler. Nós detectamos deadlocks verificando se existe ciclos nas dependências dos locks ou se alguma thread está bloqueada eternamente pois está à espera de uma notificação. • A ferramenta está integrada com o JUnit [20], o popular ambiente de testes para programas Java. Os programadores de testes podem usar a ferramenta JUnit da forma usual, recorrendo apenas a anotações adicionais às classes de teste. Na nossa ferramenta adicionamos novas anotações no JUnit, como por exemplo a definição da pasta que se deve instrumentar, o critério de avaliação para os testes e ainda uma que define que a execução irá ser cooperativa (execução por nós desenvolvida). Após a execução de um teste, a nossa ferramenta detecta se ocorreu um erro, se sim termina a execução, se não irá executar mais testes até cobrir a política ou então ter atingido o número máximo de execuções. • A ferramenta foi avaliada com exemplos de programas multithreaded retirados dos repositórios ConTest e SIR, referenciados na literatura. No caso geral, a ferramenta consegue detectar e replicar deterministicamente os bugs dos programas em causa, em contraste a testes com execução preemptiva que na vasta maioria dos casos não o consegue fazer.Software bugs are easy to introduce in multithreaded programs, resulting in wellknown errors, such as data races in the use of shared memory among threads, or deadlocks when using multithread primitives like locks. These bugs are hard to detect, replicate, and trace precisely, and are typically manifested only for a subset of all possible thread interleavings, many times even for a very particular thread interleaving. At the same time, given the non-determinism and irreproducibility of scheduling decisions at runtime, it is generally hard to control and observe thread interleaving and explore the associated state-space appropriately, when programming/executing software tests for a multithreaded program. This thesis presents Cooperari, a tool for deterministic testing of multithreaded Java programs, based on the use of cooperative semantics. In a cooperative execution, threads voluntarily suspend at interference points (e.g., lock acquisition, shared data access), called yield points, and the code between two consecutive yield points of the same thread always executes serially as a transaction. A cooperative scheduler takes over control at yield points and deterministically selects the next thread to run. A program test runs multiple times, until it either fails or the state-space of thread interleavings is deemed as covered by a configurable policy that is responsible for the scheduling decisions. Beyond failed assertions in software tests, some types of deadlock and data races are also detected by Cooperari at runtime. The tool integrates with the popular JUnit framework for Java software tests, and was evaluated using standard benchmark programs. Cooperari can find, characterize, and deterministically reproduce bugs that are not detected under unconstrained preemptive semantics