Evaluating Latency in Multiprocessing Embedded Systems for the Smart Grid

Smart grid endpoints need to use two environments within a processing system (PS), one with a Linux-type operating system (OS) using the Arm Cortex-A53 cores for management tasks, and the other with a standalone execution or a real-time OS using the Arm Cortex-R5 cores. The Xen hypervisor and the OpenAMP framework allow this, but they may introduce a delay in the system, and some messages in the smart grid need a latency lower than 3 ms. In this paper, the Linux thread latencies are characterized by the Cyclictest tool. It is shown that when Xen hypervisor is used, this scenario is not suitable for the smart grid as it does not meet the 3 ms timing constraint. Then, standalone execution as the real-time part is evaluated, measuring the delay to handle an interrupt created in programmable logic (PL). The standalone application was run in A53 and R5 cores, with Xen hypervisor and OpenAMP framework. These scenarios all met the 3 ms constraint. The main contribution of the present work is the detailed characterization of each real-time execution, in order to facilitate selecting the most suitable one for each application.This work has been supported by the Ministerio de Economía y Competitividad of Spain within the project TEC2017-84011-R and FEDER funds as well as by the Department of Education of the Basque Government within the fund for research groups of the Basque university system IT978-16. It has also been supported by the Basque Government within the project HAZITEK ZE-2020/00022 as well as the Ministerio de Ciencia e Innovación of Spain through the Centro para el Desarrollo Tecnológico Industrial (CDTI) within the project IDI-20201264; in both cases, they have been financed through the Fondo Europeo de Desarrollo Regional 2014-2020 (FEDER funds). It has also been supported by the University of the Basque Country within the scholarship for training of research staff with code PIF20/135

Efficient Formal Verification for the Linux Kernel

Formal verification of the Linux kernel has been receiving increasing attention in recent years, with the development of many models, from memory subsystems to the synchronization primitives of the real-time kernel. The effort in developing formal verification methods is justified considering the large code-base, the complexity in synchronization required in a monolithic kernel and the support for multiple architectures, along with the usage of Linux on critical systems, from high-frequency trading to self-driven cars. Despite recent developments in the area, none of the proposed approaches are suitable and flexible enough to be applied in an efficient way to a running kernel. Aiming to fill such a gap, this paper proposes a formal verification approach for the Linux kernel, based on automata models. It presents a method to auto-generate verification code from an automaton, which can be integrated into a module and dynamically added into the kernel for efficient on-the-fly verification of the system, using in-kernel tracing features. Finally, a set of experiments demonstrate verification of three models, along with performance analysis of the impact of the verification, in terms of latency and throughput of the system, showing the efficiency of the approach

Demystifying the Real-Time Linux Scheduling Latency

Linux has become a viable operating system for many real-time workloads. However, the black-box approach adopted by cyclictest, the tool used to evaluate the main real-time metric of the kernel, the scheduling latency, along with the absence of a theoretically-sound description of the in-kernel behavior, sheds some doubts about Linux meriting the real-time adjective. Aiming at clarifying the PREEMPT_RT Linux scheduling latency, this paper leverages the Thread Synchronization Model of Linux to derive a set of properties and rules defining the Linux kernel behavior from a scheduling perspective. These rules are then leveraged to derive a sound bound to the scheduling latency, considering all the sources of delays occurring in all possible sequences of synchronization events in the kernel. This paper also presents a tracing method, efficient in time and memory overheads, to observe the kernel events needed to define the variables used in the analysis. This results in an easy-to-use tool for deriving reliable scheduling latency bounds that can be used in practice. Finally, an experimental analysis compares the cyclictest and the proposed tool, showing that the proposed method can find sound bounds faster with acceptable overheads

Time Sensitive Networking Protocol Implementation for Linux End Equipment

By bringing industrial-grade robustness and reliability to Ethernet, Time Sensitive Networking (TSN) offers an IEEE standard communication technology that enables interoperability between standard-conformant industrial devices from any vendor. It also eliminates the need for physical separation of critical and non-critical communication networks, which allows a direct exchange of data between operation centers and companies, a concept at the heart of the Industrial Internet of Things (IIoT). This article describes creating an end-to-end TSN network using specialized PCI Express (PCIe) cards and two final Linux endpoints. For this purpose, the two primary standards of TSN, IEEE 802.1AS (regarding clock synchronization), and IEEE 802.1Qbv (regarding time scheduled traffic) have been implemented in Linux equipment as well as a configuration and monitoring system.This work has been supported by the Ministerio de Economía y Competitividad of Spain within the project TEC2017-84011-R and FEDER funds as well as by the Department of Education of the Basque Government within the fund for research groups of the Basque university system IT978-16

RMem: An OS Service for Transparent Remote Memory Access in Lightweight Manycores

International audienceLightweight manycores deliver high performance and scal-ability at low power consumption. However, architectural intricacies of these processors impose programmability challenges that keep them away from mass adoption. While several efforts aim at introducing parallel programming environments to lightweight manycores, few initiatives are concerned about how to design rich Operating Systems (OSs) to them. In this work, we focus on the open challenges that arise from constrained memory subsystems of lightweight manycores, such as the presence of multiple address spaces and limited on-chip memory. To cope with transparent data access in this scenario, we introduce an OS service, named RMem. This service provides a shared memory abstraction over multiple address spaces and exposes system calls that enable one-sided communication on top of this abstraction. We implemented a prototype of our service in the Nanvix research OS, and we deployed the system the Kalray MPPA-256 lightweight manycore. Our experimental results with a microbenchmark unveiled that, while exposing an easier-to-program interface, the RMem Service may deliver about 91% of the write performance and up to 2.4× better read performance than the primitives in the libraries of the experimental platform

Analyzing FreeRTOS Scheduling Behaviors with the Spin Model Checker

FreeRTOS is a real-time operating system with configurable scheduling policies. Its portability and configurability make FreeRTOS one of the most popular real-time operating systems for embedded devices. We formally analyze the FreeRTOS scheduler on ARM Cortex-M4 processor in this work. Specifically, we build a formal model for the FreeRTOS ARM Cortex-M4 port and apply model checking to find errors in our models for FreeRTOS example applications. Intriguingly, several errors are found in our application models under different scheduling policies. In order to confirm our findings, we modify application programs distributed by FreeRTOS and reproduce assertion failures on the STM32F429I-DISC1 board

MAFC: Multi-Agent Fog Computing Model for Healthcare Critical Tasks Management

Producción CientíficaIn healthcare applications, numerous sensors and devices produce massive amounts of data which are the focus of critical tasks. Their management at the edge of the network can be done by Fog computing implementation. However, Fog Nodes suffer from lake of resources That could limit the time needed for final outcome/analytics. Fog Nodes could perform just a small number of tasks. A difficult decision concerns which tasks will perform locally by Fog Nodes. Each node should select such tasks carefully based on the current contextual information, for example, tasks’ priority, resource load, and resource availability. We suggest in this paper a Multi-Agent Fog Computing model for healthcare critical tasks management. The main role of the multi-agent system is mapping between three decision tables to optimize scheduling the critical tasks by assigning tasks with their priority, load in the network, and network resource availability. The first step is to decide whether a critical task can be processed locally; otherwise, the second step involves the sophisticated selection of the most suitable neighbor Fog Node to allocate it. If no Fog Node is capable of processing the task throughout the network, it is then sent to the Cloud facing the highest latency. We test the proposed scheme thoroughly, demonstrating its applicability and optimality at the edge of the network using iFogSim simulator and UTeM clinic data

Compensating Adaptive Mixed Criticality Scheduling

The majority of prior academic research into mixed criticality systems assumes that if high-criticality tasks continue to execute beyond the execution time limits at which they would normally finish, then further workload due to low-criticality tasks may be dropped in order to ensure that the high-criticality tasks can still meet their deadlines. Industry, however, takes a different view of the importance of low-criticality tasks, with many practical systems unable to tolerate the abandonment of such tasks. In this paper, we address the challenge of supporting genuinely graceful degradation in mixed criticality systems, thus avoiding the abandonment problem. We explore the Compensating Adaptive Mixed Criticality (C-AMC) scheduling scheme. C-AMC ensures that both high- and low-criticality tasks meet their deadlines in both normal and degraded modes. Under C-AMC, jobs of low-criticality tasks, released in degraded mode, execute imprecise versions that provide essential functionality and outputs of sufficient quality, while also reducing the overall workload. This compensates, at least in part, for the overload due to the abnormal behavior of high-criticality tasks. C-AMC is based on fixed-priority preemptive scheduling and hence provides a viable migration path along which industry can make an evolutionary transition from current practice

An inter-cluster communication facility for lightweight manycore processors in the Nanvix OS

TCC(graduação) - Universidade Federal de Santa Catarina. Centro Tecnológico. Ciências da Computação.Em conjunto com a maior escalabilidade e eficiência energética, os processadores lightweight manycores trouxeram um novo conjunto de desafios no desenvolvimento de software provenientes de suas particularidades arquiteturais. Neste contexto, sistemas operacionais tornam o desenvolvimento de aplicações menos onerosos, menos suscetíveis a erros e mais eficientes. A camada de abstração provida pelos sistemas operacionais suprime as características do hardware sob uma perspectiva simplificada e eficaz. No entanto, parte dos desafios de desenvolvimento encontrados em lightweight manycores deriva diretamente de runtimes e sistemas operacionais existentes, que não lidam completamente com a complexidade arquitetural desses processadores. Acreditamos que sistemas operacionais para a próxima geração de lightweight manycores necessitam ser repensados a partir de seus conceitos básicos considerando as severas restrições arquiteturais. Em particular, as abstrações de comunicação desempenham um papel crucial na escalabilidade e desempenho das aplicações devido à natureza distribuída dos manycores. O objetivo deste trabalho é propor mecanismos de comunicação entre clusters para o processador manycore emergente MPPA-256. Estes mecanismos fazem parte de uma Camada de Abstração de Hardware (HAL) genérica e flexível para lightweight manycores que lida diretamente com os principais problemas encontrados no projeto de um sistema operacional para esses processadores. Sob estes mecanismos, serviços de comunicação também serão propostos para um sistema operacional baseado no modelo microkernel, que busca fornecer um esqueleto básico para as abstrações de comunicação. As contribuições deste trabalho estão inseridas em um contexto de pesquisa mais amplo, que procura investigar a criação de um sistema operacional distribuído baseado em uma abordagem multikernel, denominado Nanvix OS. O Nanvix OS se concentrará em questões de programabilidade e portabilidade através de um sistema operacional compatível com o padrão POSIX para lightweight manycore. Os resultados mostram como algoritmos distribuídos conhecidos podem ser eficientemente suportados pelo Nanvix OS e incentivam melhorias providas pelo uso adequado dos aceleradores de Acesso Direto à Memória (DMA).Jointly with further scalability and energy efficiency, lightweight manycores brought a new set of challenges in software development coming from their architectural particularities. In this context, Operating Systems (OSs) make application development less costly, less error-prone, and more efficient. The abstraction layer provided by OSs suppresses hardware characteristics from a simplified and productive perspective. However, part of the development challenges encountered in lightweight manycores stems from the existing runtimes and OSs, which do not entirely address the complexity of these processors. We believe that OSs for the next generation of lightweight manycores must be redesigned from scratch to cope with their tight architectural constraints. In particular, communication abstractions play a crucial role in application scalability and performance due to the distributed nature of manycores. The purpose of this undergraduate dissertation is to propose an inter-cluster communication facility for the emerging manycore MPPA-256 processor. This facility is part of a generic and flexible Hardware Abstraction Layer (HAL) that deals directly with the key issues encountered in designing an OS for these processors. Above this facility, communication services will also be proposed for an OS based on the microkernel model, which seeks to provide a basic framework for communication abstractions. The contributions of this undergraduate dissertation are embedded in a broader research context that aims to investigate the creation of a distributed OS based on a multikernel approach, called Nanvix OS. Nanvix OS focuses on programmability and portability issues for manycores through a POSIX-compliant OS. The results present how well known distributed algorithms can be efficiently supported by Nanvix OS and encourage improvements provided by the proper use of Direct memory access (DMA) accelerators

Applying Hypervisor-Based Fault Tolerance Techniques to Safety-Critical Embedded Systems

This document details the work conducted through the development of this thesis, and it is structured as follows: • Chapter 1, Introduction, has briefly presented the motivation, objectives, and contributions of this thesis. • Chapter 2, Fundamentals, exposes a series of concepts that are necessary to correctly understand the information presented in the rest of the thesis, such as the concepts of virtualization, hypervisors, or software-based fault tolerance. In addition, this chapter includes an exhaustive review and comparison between the different hypervisors used in scientific studies dealing with safety-critical systems, and a brief review of some works that try to improve fault tolerance in the hypervisor itself, an area of research that is outside the scope of this work, but that complements the mechanism presented and could be established as a line of future work. • Chapter 3, Problem Statement and Related Work, explains the main reasons why the concept of Hypervisor-Based Fault Tolerance was born and reviews the main articles and research papers on the subject. This review includes both papers related to safety-critical embedded systems (such as the research carried out in this thesis) and papers related to cloud servers and cluster computing that, although not directly applicable to embedded systems, may raise useful concepts that make our solution more complete or allow us to establish future lines of work. • Chapter 4, Proposed Solution, begins with a brief comparison of the work presented in Chapter 3 to establish the requirements that our solution must meet in order to be as complete and innovative as possible. It then sets out the architecture of the proposed solution and explains in detail the two main elements of the solution: the Voter and the Health Monitoring partition. • Chapter 5, Prototype, explains in detail the prototyping of the proposed solution, including the choice of the hypervisor, the processing board, and the critical functionality to be redundant. With respect to the voter, it includes prototypes for both the software version (the voter is implemented in a virtual machine) and the hardware version (the voter is implemented as IP cores on the FPGA). • Chapter 6, Evaluation, includes the evaluation of the prototype developed in Chapter 5. As a preliminary step and given that there is no evidence in this regard, an exercise is carried out to measure the overhead involved in using the XtratuM hypervisor versus not using it. Subsequently, qualitative tests are carried out to check that Health Monitoring is working as expected and a fault injection campaign is carried out to check the error detection and correction rate of our solution. Finally, a comparison is made between the performance of the hardware and software versions of Voter. • Chapter 7, Conclusions and Future Work, is dedicated to collect the conclusions obtained and the contributions made during the research (in the form of articles in journals, conferences and contributions to projects and proposals in the industry). In addition, it establishes some lines of future work that could complete and extend the research carried out during this doctoral thesis.Programa de Doctorado en Ciencia y Tecnología Informática por la Universidad Carlos III de MadridPresidente: Katzalin Olcoz Herrero.- Secretario: Félix García Carballeira.- Vocal: Santiago Rodríguez de la Fuent