Macromodeling and characterization of filesystem energy consumption for diskless embedded systems

The use and application of embedded systems in everyday life has proliferated in the past few years. These systems are constrained in terms of power consumption, available memory and processing requirements. Typical embedded systems like handheld devices, cell phones, single board computer based systems are diskless and use flash for secondary storage. The choice of filesystem for these diskless systems can greatly impact the performance and the energy consumption of the system as well as lifetime of flash. In this thesis work, the energy consumption of flash based filesystems has been characterized. Both the processor and flash energy consumption are characterized as a function of filesystem specific operations. The work is aimed at helping a system designer compare and contrast different filesystems based on energy consumption as a metric. The macromodel can be used to characterize and estimate the energy consumption of applications due to filesystem running on flash. The study is done on a StrongARM based processor running Linux. Two of the popular filesystems JFFS2 and ext3 are profiled

Macro-modeling and energy efficiency studies of file management in embedded systems with flash memory

Technological advancements in computer hardware and software have made embedded systems highly affordable and widely used. Consumers have ever increasing demands for powerful embedded devices such as cell phones, PDAs and media players. Such complex and feature-rich embedded devices are strictly limited by their battery life- time. Embedded systems typically are diskless and use flash for secondary storage due to their low power, persistent storage and small form factor needs. The energy efficiency of a processor and flash in an embedded system heavily depends on the choice of file system in use. To address this problem, it is necessary to provide sys- tem developers with energy profiles of file system activities and energy efficient file systems. In the first part of the thesis, a macro-model for the CRAMFS file system is established which characterizes the processor and flash energy consumption due to file system calls. This macro-model allows a system developer to estimate the energy consumed by CRAMFS without using an actual power setup. The second part of the thesis examines the effects of using non-volatile memory as a write-behind buffer to improve the energy efficiency of JFFS2. Experimental results show that a 4KB write-behind buffer significantly reduces energy consumption by up to 2-3 times for consecutive small writes. In addition, the write-behind buffer conserves flash space since transient data may never be written to flash

Failure avoidance techniques for HPC systems based on failure prediction

A increasingly larger percentage of computing capacity in today's large high-performance computing systems is wasted due to failures and recoveries. Moreover, it is expected that high performance computing will reach exascale within a decade, decreasing the mean time between failures to one day or even a few hours, making fault tolerance a major challenge for the HPC community. As a consequence, current research is focusing on providing fault tolerance strategies that aim to minimize fault's effects on applications. By far, the most popular and used techniques from this field are rollback-recovery protocols. However, existing rollback-recovery techniques have severe scalability limitations and without further optimizations the use of current protocols is put under serious questions for future exascale systems. A way of reducing the overhead induced by these strategies is by combining them with failure avoidance methods. Failure avoidance is based on a prediction model that detects fault occurrences ahead of time and allows preventive measures to be taken, such as task migration or checkpointing the application before failure. The same methodology can be generalized and applied to anomaly avoidance, where anomaly can mean anything from system failures to performance degradation at the application level. For this, monitoring systems require a reliable prediction system to give information on when failures will occur and at what location. Thus far, research in this field used ideal predictors that do not have any implementation in real HPC systems. This thesis focuses on analyzing and characterizing anomaly patterns at both the application and system levels and on offering solutions to prevent anomalies from affecting applications running in the system. Currently, there is no good characterization of normal behavior for system state data or how different components react to failures within HPC systems. For example, in case a node experiences a network failure and is incapable of generating log messages, the failure is announced in the log files by a lack of generated messages. Conversely, some component failures may cause logging a large numbers of notifications. For example, memory failures can result in a single faulty component generating hundreds or thousands of messages in less than a day. It is important to be able to capture the behavior of each event type and understand what is the normal behavior and how each failure type affects it. This idea represents the building block of a novel way of characterizing the state of the system in time by analyzing the properties of each event described in different system metrics, considering its own trend and behavior. The method introduces the integration between signal processing concepts and data mining techniques in the context of analysis for large-scale systems. By shaping the normal and faulty behavior of each event and of the whole system, appropriate models and methods for descriptive and forecasting purposes are proposed. After having an accurate overview of the whole system, the thesis analyzes how the prediction model impacts current fault tolerance techniques and in the end integrates it into a fault avoidance solution. This hybrid protocol optimizes the overhead that current fault tolerance strategies impose on applications and presents a viable solution for future large-scale systems

Efficient I/O for Computational Grid Applications

High-performance computing increasingly occurs on computational grids composed of heterogeneous and geographically distributed systems of computers, networks, and storage devices that collectively act as a single virtual computer. A key challenge in this environment is to provide efficient access to data distributed across remote data servers. This dissertation explores some of the issues associated with I/O for wide-area distributed computing and describes an I/O system, called Armada, with the following features: a framework to allow application and dataset providers to flexibly compose graphs of processing modules that describe the distribution, application interfaces, and processing required of the dataset before or after computation; an algorithm to restructure application graphs to increase parallelism and to improve network performance in a wide-area network; and a hierarchical graph-partitioning scheme that deploys components of the application graph in a way that is both beneficial to the application and sensitive to the administrative policies of the different administrative domains. Experiments show that applications using Armada perform well in both low- and high-bandwidth environments, and that our approach does an exceptional job of hiding the network latency inherent in grid computing

Selected Papers from the First International Symposium on Future ICT (Future-ICT 2019) in Conjunction with 4th International Symposium on Mobile Internet Security (MobiSec 2019)

The International Symposium on Future ICT (Future-ICT 2019) in conjunction with the 4th International Symposium on Mobile Internet Security (MobiSec 2019) was held on 17–19 October 2019 in Taichung, Taiwan. The symposium provided academic and industry professionals an opportunity to discuss the latest issues and progress in advancing smart applications based on future ICT and its relative security. The symposium aimed to publish high-quality papers strictly related to the various theories and practical applications concerning advanced smart applications, future ICT, and related communications and networks. It was expected that the symposium and its publications would be a trigger for further related research and technology improvements in this field

Contribution à la convergence d'infrastructure entre le calcul haute performance et le traitement de données à large échelle

The amount of produced data, either in the scientific community or the commercialworld, is constantly growing. The field of Big Data has emerged to handle largeamounts of data on distributed computing infrastructures. High-Performance Computing (HPC) infrastructures are traditionally used for the execution of computeintensive workloads. However, the HPC community is also facing an increasingneed to process large amounts of data derived from high definition sensors andlarge physics apparati. The convergence of the two fields -HPC and Big Data- iscurrently taking place. In fact, the HPC community already uses Big Data tools,which are not always integrated correctly, especially at the level of the file systemand the Resource and Job Management System (RJMS).In order to understand how we can leverage HPC clusters for Big Data usage, andwhat are the challenges for the HPC infrastructures, we have studied multipleaspects of the convergence: We initially provide a survey on the software provisioning methods, with a focus on data-intensive applications. We contribute a newRJMS collaboration technique called BeBiDa which is based on 50 lines of codewhereas similar solutions use at least 1000 times more. We evaluate this mechanism on real conditions and in simulated environment with our simulator Batsim.Furthermore, we provide extensions to Batsim to support I/O, and showcase thedevelopments of a generic file system model along with a Big Data applicationmodel. This allows us to complement BeBiDa real conditions experiments withsimulations while enabling us to study file system dimensioning and trade-offs.All the experiments and analysis of this work have been done with reproducibilityin mind. Based on this experience, we propose to integrate the developmentworkflow and data analysis in the reproducibility mindset, and give feedback onour experiences with a list of best practices.RésuméLa quantité de données produites, que ce soit dans la communauté scientifiqueou commerciale, est en croissance constante. Le domaine du Big Data a émergéface au traitement de grandes quantités de données sur les infrastructures informatiques distribuées. Les infrastructures de calcul haute performance (HPC) sont traditionnellement utilisées pour l’exécution de charges de travail intensives en calcul. Cependant, la communauté HPC fait également face à un nombre croissant debesoin de traitement de grandes quantités de données dérivées de capteurs hautedéfinition et de grands appareils physique. La convergence des deux domaines-HPC et Big Data- est en cours. En fait, la communauté HPC utilise déjà des outilsBig Data, qui ne sont pas toujours correctement intégrés, en particulier au niveaudu système de fichiers ainsi que du système de gestion des ressources (RJMS).Afin de comprendre comment nous pouvons tirer parti des clusters HPC pourl’utilisation du Big Data, et quels sont les défis pour les infrastructures HPC, nousavons étudié plusieurs aspects de la convergence: nous avons d’abord proposé uneétude sur les méthodes de provisionnement logiciel, en mettant l’accent sur lesapplications utilisant beaucoup de données. Nous contribuons a l’état de l’art avecune nouvelle technique de collaboration entre RJMS appelée BeBiDa basée sur 50lignes de code alors que des solutions similaires en utilisent au moins 1000 fois plus.Nous évaluons ce mécanisme en conditions réelles et en environnement simuléavec notre simulateur Batsim. En outre, nous fournissons des extensions à Batsimpour prendre en charge les entrées/sorties et présentons le développements d’unmodèle de système de fichiers générique accompagné d’un modèle d’applicationBig Data. Cela nous permet de compléter les expériences en conditions réellesde BeBiDa en simulation tout en étudiant le dimensionnement et les différentscompromis autours des systèmes de fichiers.Toutes les expériences et analyses de ce travail ont été effectuées avec la reproductibilité à l’esprit. Sur la base de cette expérience, nous proposons d’intégrerle flux de travail du développement et de l’analyse des données dans l’esprit dela reproductibilité, et de donner un retour sur nos expériences avec une liste debonnes pratiques

Estimation de performances et de consommation énergétique de systèmes de stockage à base de mémoire flash dans les systèmes embarqués

Controlling and optimizing embedded system performance and power consumption is critical. In this context, estimation techniques are used when performing measurement campaigns is difficult due to time or financial constraints. This work targets the performance and power consumption evaluation of the secondary storage service in an embedded operating system using NAND flash memory. One way to manage flash memory is to used dedicated Flash File Systems (FFS). One can observe a lack of work in the literature concerning FFS performance and power consumption estimation techniques.The contributions presented in this thesis rely on a three steps performance and power consumption modeling methodology. During the exploration phase, we identify through micro-benchmarking the main elements of a FFS based system impacting performance and power consumption of the embedded system. In the modeling phase, this impact is represented by building models of various types. The main models types are the functional, performance and power consumption models. Models parameters are extracted through measurements on a real platform. During the simulation phase the models are implemented in a simulator. This tool allows obtaining performance and power consumption estimations concerning a flash-based storage system processing a given I/O workload.Maitriser et optimiser les performances et la consommation énergétique dans les systèmes embarqués est aujourd'hui crucial. Pour ce faire, des techniques d'estimation de ces métriques sont utilisées dans des environnements où la réalisation de mesures est difficile. Ce travail cible l'évaluation des performances et de la consommation énergétique du service du stockage secondaire dans un système d'exploitation embarqué utilisant une mémoire flash NAND. L'un des moyens de gérer ce type de média est l'utilisation de systèmes de fichiers dédiés (Flash File Systems, FFS), pour lequel on peut constater un manque de travaux dans la littérature concernant les techniques d'estimation des performances et de la consommation. Les contributions apportées dans cette thèse s'articulent autour d'une méthodologie de modélisation pour l'estimation des performances et de la consommation des systèmes de stockage embarqués de type FFS. Cette méthodologie est divisée en trois phases. En phase d'exploration on identifie, via des micro-benchmarks, les éléments du système de stockage impactant les performances et la consommation du système embarqué. En phase de modélisation, cet impact est représenté sous la forme de modèles de différents types, dont les principaux sont les modèles fonctionnels, de performances et de consommation. Les paramètres de ces modèles sont extraits via des mesures. En phase de simulation, les modèles sont implémenté dans un simulateur, développé dans le cadre de cette thèse, permettant d'obtenir des estimations concernant les performances et la consommation d'un système de stockage à base de mémoire flash soumis à une charge d'entrées / sorties donnée