Improving OpenStack Swift interaction with the I/O stack to enable software defined storage

This paper analyses how OpenStack Swift, a distributed object storage service for a globally used middleware, interacts with the I/O subsystem through the Operating System. This interaction, which seems organised and clean on the middleware side, becomes disordered on the device side when using mechanical disk drives, due to the way threads are used internally to request data. We will show that only modifying the Swift threading model we achieve an 18% mean improvement in performance with objects larger than 512 KiB and obtain a similar performance with smaller objects. Compared to the original scenario, the performance obtained on both scenarios is obtained in a fair way: the bandwidth is shared equally between concurrently accessed objects. Moreover, this threading model allows us to apply techniques for Software Defined Storage (SDS). We show an implementation of a Bandwidth Differentiation technique that can control each data stream and that guarantees a high utilization of the device.The research leading to these results has received funding from the European Community under the IOStack (H2020-ICT-2014-7-1) project, by the Spanish Ministry of Economy and Competitiveness under the TIN2015-65316-P grant and by the Catalan Government under the 2014-SGR-1051 grant. To learn more about the IOStack H2020 project, please visit http:nnwww.iostack.eu.Peer ReviewedPostprint (author's final draft

ECHOFS: a scheduler-guided temporary filesystem to leverage node-local NVMS

© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.The growth in data-intensive scientific applications poses strong demands on the HPC storage subsystem, as data needs to be copied from compute nodes to I/O nodes and vice versa for jobs to run. The emerging trend of adding denser, NVM-based burst buffers to compute nodes, however, offers the possibility of using these resources to build temporary file systems with specific I/O optimizations for a batch job. In this work, we present echofs, a temporary filesystem that coordinates with the job scheduler to preload a job's input files into node-local burst buffers. We present the results measured with NVM emulation, and different FS backends with DAX/FUSE on a local node, to show the benefits of our proposal and such coordination.This work was partially supported by the Spanish Ministry of Science and Innovation under the TIN2015–65316 grant, the Generalitat de Catalunya under contract 2014– SGR–1051, as well as the European Union’s Horizon 2020 Research and Innovation Programme, under Grant Agreement no. 671951 (NEXTGenIO). Source code available at https://github.com/bsc-ssrg/echofs.Peer ReviewedPostprint (author's final draft

Freezing Time: a new approach for emulating fast storage devices using VM

© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Recently we are seeing a considerable effort from both academy and industry in proposing new technologies for storage devices. Often these devices are not readily available for evaluation and methods to allow performing their tests just from their performance parameters are an important tool for system administrators. Simulators are a traditional approach for carrying out such evaluations, however, they are more suitable for evaluating the storage device as an isolate component, mostly due to time constraints. In this paper, we propose an approach based on virtual machine technology that is capable of emulate storage devices transparently for the operating system allowing evaluation of simulating devices within a real system using any synthetic or real workload. To emulate devices in real environments it is necessary to use the currently available devices as a storage medium which creates a difficulty when the device to be emulated is faster than this storage medium. To circumvent this limitation we introduce a new technique called Freezing Time, which takes advantage of virtual machine pausing mechanism to manipulate the virtual machine clock and hide the real I/O completion time. Our approach can be implemented just requiring the hypervisor to be modified, providing a high degree of compatibility and flexibility since it is not necessary to modify neither the operating system nor the application. We evaluate our tool under a real system using old magnetic disks to emulate faster storage devices. Experiments using our technique presented an average latency error of 6.08% for read operations and 6.78% for write operations when comparing a real to device.This work was partially supported by the Spanish Ministry of Science and Innovation under the TIN2015–65316 grant, the Generalitat de Catalunya under contract 2014–SGR–1051.Peer ReviewedPostprint (author's final draft

GekkoFS: A temporary distributed file system for HPC applications

We present GekkoFS, a temporary, highly-scalable burst buffer file system which has been specifically optimized for new access patterns of data-intensive High-Performance Computing (HPC) applications. The file system provides relaxed POSIX semantics, only offering features which are actually required by most (not all) applications. It is able to provide scalable I/O performance and reaches millions of metadata operations already for a small number of nodes, significantly outperforming the capabilities of general-purpose parallel file systems.The work has been funded by the German Research Foundation (DFG) through the ADA-FS project as part of the Priority Programme 1648. It is also supported by the Spanish Ministry of Science and Innovation (TIN2015–65316), the Generalitat de Catalunya (2014–SGR–1051), as well as the European Union’s Horizon 2020 Research and Innovation Programme (NEXTGenIO, 671951) and the European Comission’s BigStorage project (H2020-MSCA-ITN-2014-642963). This research was conducted using the supercomputer MOGON II and services offered by the Johannes Gutenberg University Mainz.Peer ReviewedPostprint (author's final draft

XtreemOS application execution management: a scalable approach

Designing a job management system for the Grid is a non-trivial task. While a complex middleware can give a lot of features, it often implies sacrificing performance. Such performance loss is especially noticeable for small jobs. A Job Manager’s design also affects the capabilities of the monitoring system. We believe that monitoring a job or asking for a job status should be fast and easy, like doing a simple ’ps’. In this paper, we present the job management of XtreemOS - a Linux-based operating system to support Virtual Organizations for Grid. This management is performed inside the Application Execution Manager (AEM). We evaluate its performance using only one job manager plus the built-in monitoring infrastructure. Furthermore, we present a set of real-world applications using AEM and its features. In XtreemOS we avoid reinventing the wheel and use the Linux paradigm as an abstraction.Peer ReviewedPostprint (published version

Simulating complex systems with a low-detail model

In this paper we show how modeling and simulating a complex system such as a web-server can help to evaluate di erent metrics and proposals to improve the performance without necessity of using a real system. Many times the system is unavailable or requires spending time and resources to generate results. With simulation and concretely with a coarse-grain simulation as we propose can solve with great success this problem. In this article we have been able to simulate the metric and behaviour of a web server with SSL security, the ellapsed time required by the simulation on a desktop machine is only 1/10 of real time. We have also been able to measure, for example,the performance enhancements with 8 CPUs without having an available machine of similar features.Postprint (published version

IOStack: Software-Defined Object Storage

The complexity and scale of today’s cloud storage systems is growing fast. In response to these challenges, Software- Defined Storage (SDS) has recently become a prime candidate to simplify storage management in the cloud. This article presents IOStack: The first SDS architecture for object stores (OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level DSL. Policies may target storage automation and/or specific SLA objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant’s requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Our experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multi-tenant object store.This work has been funded by the European Union through project H2020 “IOStack: Software-Defined Storage for Big Data” (644182) and by the Spanish Ministry of Science and Innovation through project “Servicios Cloud y Redes Comunitarias” (TIN-2013-47245-C2-2-R).Peer ReviewedPostprint (author's final draft

Conocimiento en 1000Genome y GWAS

Gran parte del conocimiento biológico se encuentra dividido en varias bases de datos. Gracias a los avances en la potencia de cálculo todos estos datos se pueden analizar utilizando técnicas basadas en minería de datos, estadística y machine learning. En este trabajo nos hemos centrado en dos grandes bases de datos que se pueden utilizar para encontrar relaciones entre poblaciones y distintos fenotipos utilizando SNPs (Single Nucleotide Polymorphism). En este caso, se utilizará información de la base de datos de 1000Genome, que contiene el genoma completo de más de 1000 humanos de distintas poblaciones y los datos de la base de datos GWAS que contiene los SNPs y su relación con distintos rasgos (asma, cáncer...) Mostraremos distintas formas para extraer información, incluyendo machine learning y posteriormente aplicaremos distintos métodos para mejorar su rendimiento tanto en el plano de la computación (añadiendo paralelismo) como mejorando la entrada/salida (mejorando la distribución y la utilización de los datos). Finalmente analizaremos la parte de aprendizaje y extracción de conocimiento comparando distintos algoritmos y métodos, realizando un análisis más detallado de los datos.The biological knowledge, or at least a big part of it, is divided in different databases. Thanks to the advances in the computation power, we can analyse all this data using data mining, statistical methods and machine learning techniques. In this work, we will focus in two important databases that can be used to find relations between populations and fenotypes using SNPs (Single Nucleotide Polymorphism) as features. For this work, we will use information from 1000Genome, a database containing the sequentiation of more than 1000 humans' genome and from GWAS, another database that contains the relation between SNPs and traits (i.e., asthma or cancer). Different ways of extracting information will be presented, including machine learning. After that, a performance analysis and optimization techniques will be applied both to computation speed (parallelism) and I/O (data distribution). Finally, a comparative analysis of machine learning algorithms will be presented.Gran part del coneixement biològic es troba dividit en diverses bases de dades. Gràcies als avanços en la potència de càlcul totes aquestes dades es poden analitzar utilitzant tècniques basades en mineria de dades, estadística i machine learning. En aquest treball ens hem centrat en dues grans bases de dadesque es poden utilitzar per trobar relacions entre poblacions i diferents fenotips utilitzant SNPs (Single Nucleotide Polymorphism). En aquest cas, s'utilitzarà informació de la base de dades de 1000Genome, que conté el genoma complet de més de 1000 humans de diferents poblacions i les dades de la base de dades GWAS que conté els SNPs i la seva relació amb diferents trets (asma, càncer...) Mostrarem diferents formes per extreure informació, incloent machine learning i posteriorment aplicarem diferents mètodes per millorar el seu rendiment tant en el plànol de la computació (afegint paral·lelisme) com millorant l'entrada/sortida (millorant la distribució i la utilització de les dades). Finalment analitzarem la part d'aprenentatge i extracció de coneixement comparant diferents algorismes i mètodes, realitzant una anàlisi més detallada de les dades

Heterogeneous QoS resource manager with prediction

As long as computers continue to get more CPU processing power, data centers need to optimize their power usage. We can do this and maintain the same complexity level as before by using virtualized environments. We can put a large number of small isolated servers, inside a large one and improve a large number of values like the wattage or power consumption, space usage, and resource usage. In this paper, we present a prototype with which we distribute resources between two virtualized servers, one with Tomcat and another with Globus, and both sharing the same host. The prototype is able to maintain the required SLA and QoS using prediction with simulation in real time. Our goal is to demonstrate that simulation can be used to improve resource managers decissions. In this paper, we use those simulations inside a shared server with several different applications using virtualization.Peer Reviewe