Mesure et analyse de latences dans les systèmes parallèles en temps réel

RÉSUMÉ Avec les infrastructures de type infonuagiques qui augmentent de plus en plus et les services qui exploitent les avantages de la parallélisation, on arrive à un point où les analyses de problèmes de performance sont de plus en plus complexes. En particulier, avec la parallélisation, un problème de latence sur un des composants peut ralentir une requête complète. Avec la multiplication du nombre de serveurs responsables d'une seule requête, la quantité de tests et de combinaisons à valider pour trouver une source de latence peut augmenter de manière exponentielle. Les problèmes à analyser ne sont pas nouveaux, ils sont similaires à ceux étudiés dans les systèmes temps-réel. La problématique cependant se situe au niveau de la détection automatisée en temps réel des problèmes dans des conditions réelles d'exploitation, et la mise à l'échelle de la collecte de données de contexte permettant la résolution. Dans cette thèse, nous proposons le \texttt{latency-tracker} comme solution efficace pour la mesure et l'analyse en temps réel de latences, et de le combiner avec le traceur \texttt{LTTng} pour la collecte et l'extraction de traces localement et sur le réseau. L'objectif principal est de rendre ces analyses complexes assez efficaces et non-intrusives pour fonctionner sur des machines de production, que ce soit sur des serveurs ou des appareils embarqués dédiés aux applications temps réel. Cette approche de la détection et de la compréhension des problèmes de latence dans l'ordre des dizaines de micro-secondes au niveau du noyau Linux est nouvelle et il n'existe pas d'équivalent à l'heure actuelle. En mesurant l'impact de tous les composants ajoutés dans le chemin critique des applications de manière individuelle, nous démontrons qu'il est possible d'utiliser cette approche dans des environnements très exigeants. Les mesures se concentrent au niveau de la consommation des ressources, jusqu'à l'effet sur les lignes de cache, mais également sur la mise à l'échelle sur des applications concurrentes et distribuées. La contribution principale de cette recherche se situe au niveau de l'ensemble des algorithmes développés permettant de mesurer précisément les latences avec un impact minimal, et de collecter assez d'informations de contexte pour en expliquer les causes. Ce faible impact permet l'application de ces méthodes dans des situations réelles où il était jusqu'à présent impossible de faire ce type de mesures sans modifier les conditions d'exécution. La spécialisation et l'optimisation des techniques actuelles d'agrégation, et la combinaison avec le domaine du traçage, donne ainsi naissance au domaine du traçage à état.----------ABSTRACT Today's server infrastructures are more and more organized around the cloud and virtualization technologies, and the services that run on these infrastructures tend to heavily use parallelisation to scale up to the demand. With this type of distributed systems, the performance analyses are becoming increasingly complex. Indeed, the work required to answer a single request can be divided among multiple servers, and a problem with any of the nodes can slow down the whole request. Finding the exact source of an abnormal latency in this kind of configuration can be really difficult and requires a lot of time. The problems we encounter are not new, they are similar to the ones faced by real-time systems. The biggest issue is to automatically detect in real-time these problems in production, and to have a scalable way to collect the context information required to understand and solve the problems. In this thesis, we propose the \texttt{latency-tracker} as a solution to efficiently measure and analyse latency problems in real-time, and to combine it with the \texttt{LTTng} tracer to gather and extract traces locally and on the network. The main objective is to make these complex analyses efficient enough to run on production machines, either servers in data-centers, or embedded platforms dedicated to real-time tasks. This approach to detect and explain latency issues in the order of tens of microseconds in the Linux kernel is new and there is no equivalent solution today. By individually measuring the impact of all the components added in the critical path of the applications, we demonstrate that it is possible to use this approach in very demanding environments. We measure the impact on the usage of resources, down to the impact on cache lines, but we also study the scalability of our approach on highly concurrent and distributed applications. The main contribution of this research is the set of algorithms developed to accurately measure latencies with a minimal impact, and to collect and extract enough context informations to understand the latency causes. This low impact enables the use of these methodologies in production, under real loads, which would be impossible with the existing tools today without risking to modify the execution conditions. We specialize and optimize the current techniques related to event agregation, and combine it with tracing to create the new domain of stateful tracing

Virtuaalinen kenttäväylä -Sovellettavuus, teknologiat ja arviointi

In the present-day software and automation development, different methods of virtualization have become popular, as the final hardware is then not required for software development. This allows earlier and faster software process, reduced time to market and more fluent workflow. As distributed automation systems generally rely on fieldbuses of various types, implementing a virtual and fully operational fieldbus is a necessity for efficient utilization of the virtualized system. In this thesis, we take a comprehensive approach to virtual fieldbuses, from concept definition to experimental performance characteristics. We discuss the common behavior of different fieldbuses, list applications for virtual buses and compare possible implementation technologies such as TCP/IP and shared memory. Virtualization tools VirtualBox and QEMU are closely studied, as they bring additional challenges to data transfer. From the practical point of view, the study presents our experiences on implementing a virtual CAN bus for embedded development. With an extensive set of features and platform support in our design, it demonstrates the utilization of multiple technologies. Using the virtual CAN implementation, we then show the measured performance characteristics and evaluate the solution against actual hardware. Potential of virtual bus technology was proven by the performance measurements. Shared memory implementation provided extremely good performance, sufficient for implementing any virtual fieldbus system. It was also found to be efficient in respect to CPU load. Unfortunately, shared memory usually cannot cross virtualization boundaries. TCP was found as the best option for the rest of the use cases. In restricted local Ethernet or between VirtualBox and host OS, it is able to provide latencies under 700µs, similar to hardware performance. Observed bottlenecks were the use of the QEMU emulation tool without optimizations, and slow USB fieldbus adapters. We recommend using virtual fieldbuses in virtualized development and debugging of distributed systems and for automatic system level testing, if timing requirements are not extremely strict. Remote virtual connection to a hardware fieldbus is also seen as a valid application. The technologies and adapters must still be carefully selected for best results

Hard Real-Time Linux for Off-The-Shelf Multicore Architectures

This document describes the research results that were obtained from the development of a real-time extension for the Linux operating system. The paper describes a full extension of the kernel, which enables hard real-time performance on a 64-bit x86 architecture. In the first part of this study, real-time systems are categorized and concepts of real-time operating systems are introduced to the reader. In addition, numerous well-known real-time operating systems are considered. QNX Neutrino, RT_PREEMPT Linux Patch and HLRT Linux Patch are analyzed in detail. The core concepts of these systems are shown and discussed. Furthermore, a test suite is developed, which is used to obtain expressive benchmarks from the systems that were analyzed before. The systems are evaluated on the basis of these benchmarks and compared to the real-time extension which is developed in this work. A requirements catalogue is defined based on the analysis of the stated operating systems. The design of a real-time extension is developed based on the specification catalogue and the identified core concepts. Furthermore, the concrete implementation of the developed real-time extension is presented in detail. Finally, the benchmarks of all analyzed systems, including the developed real-time extension, are compared to each other and evaluated

Static Analysis of Lockless Microcontroller C Programs

Concurrently accessing shared data without locking is usually a subject to race conditions resulting in inconsistent or corrupted data. However, there are programs operating correctly without locking by exploiting the atomicity of certain operations on a specific hardware. In this paper, we describe how to precisely analyze lockless microcontroller C programs with interrupts by taking the hardware architecture into account. We evaluate this technique in an octagon-based value range analysis using access-based localization to increase efficiency