112 research outputs found
Bringing Introspection Into the BlobSeer Data-Management System Using the MonALISA Distributed Monitoring Framework
Held in conjunction with CISIS 2010 ConferenceInternational audienceIntrospection is the prerequisite of an autonomic behavior, the ïŹrst step towards a performance improvement and a resource-usage optimization for large-scale distributed systems. In grid environments, the task of observing the application behavior is assigned to monitoring systems. However, most of them are designed to provide general resource information and do not consider speciïŹc information for higher-level services. More specifically, in the context of data-intensive applications, a speciïŹc introspection layer is required in order to collect data about the usage of storage resources, about data access patterns, etc. This paper discusses the requirements for an introspection layer in a data-management system for large-scale distributed infrastructures. We focus on the case of BlobSeer, a large-scale distributed system for storing massive data. The paper explains why and how to enhance BlobSeer with introspective capabilities and proposes a three-layered architecture relying on the MonALISA monitoring framework. This approach has been evaluated on the Grid'5000 testbed, with experiments that prove the feasibility of generating relevant information related to the state and the behavior of the system
Spark versus Flink: Understanding Performance in Big Data Analytics Frameworks
International audienceBig Data analytics has recently gained increasing popularity as a tool to process large amounts of data on-demand. Spark and Flink are two Apache-hosted data analytics frameworks that facilitate the development of multi-step data pipelines using directly acyclic graph patterns. Making the most out of these frameworks is challenging because efficient executions strongly rely on complex parameter configurations and on an in-depth understanding of the underlying architectural choices. Although extensive research has been devoted to improving and evaluating the performance of such analytics frameworks, most of them benchmark the platforms against Hadoop, as a baseline, a rather unfair comparison considering the fundamentally different design principles. This paper aims to bring some justice in this respect, by directly evaluating the performance of Spark and Flink. Our goal is to identify and explain the impact of the different architectural choices and the parameter configurations on the perceived end-to-end performance. To this end, we develop a methodology for correlating the parameter settings and the operators execution plan with the resource usage. We use this methodology to dissect the performance of Spark and Flink with several representative batch and iterative workloads on up to 100 nodes. Our key finding is that there none of the two framework outperforms the other for all data types, sizes and job patterns. This paper performs a fine characterization of the cases when each framework is superior, and we highlight how this performance correlates to operators, to resource usage and to the specifics of the internal framework design
OverFlow: Multi-Site Aware Big Data Management for Scientific Workflows on Clouds
International audienceThe global deployment of cloud datacenters is enabling large scale scientific workflows to improve performance and deliver fast responses. This unprecedented geographical distribution of the computation is doubled by an increase in the scale of the data handled by such applications, bringing new challenges related to the efficient data management across sites. High throughput, low latencies or cost-related trade-offs are just a few concerns for both cloud providers and users when it comes to handling data across datacenters. Existing solutions are limited to cloud-provided storage, which offers low performance based on rigid cost schemes. In turn, workflow engines need to improvise substitutes, achieving performance at the cost of complex system configurations, maintenance overheads, reduced reliability and reusability. In this paper, we introduce OverFlow, a uniform data management system for scientific workflows running across geographically distributed sites, aiming to reap economic benefits from this geo-diversity. Our solution is environment-aware, as it monitors and models the global cloud infrastructure, offering high and predictable data handling performance for transfer cost and time, within and across sites. OverFlow proposes a set of pluggable services, grouped in a data scientist cloud kit. They provide the applications with the possibility to monitor the underlying infrastructure, to exploit smart data compression, deduplication and geo-replication, to evaluate data management costs, to set a tradeoff between money and time, and optimize the transfer strategy accordingly. The system was validated on the Microsoft Azure cloud across its 6 EU and US datacenters. The experiments were conducted on hundreds of nodes using synthetic benchmarks and real-life bio-informatics applications (A-Brain, BLAST). The results show that our system is able to model accurately the cloud performance and to leverage this for efficient data dissemination, being able to reduce the monetary costs and transfer time by up to 3 times
A-Brain: Using the Cloud to Understand the Impact of Genetic Variability on the Brain
International audienceJoint genetic and neuroimaging data analysis on large cohorts of subjects is a new approach used to assess and understand the variability that exists between individuals. This approach has remained poorly understood so far and brings forward very significant challenges, as progress in this field can open pioneering directions in biology and medicine. As both neuroimaging- and genetic-domain observations represent a huge amount of variables (of the order of 106 ), performing statistically rigorous analyses on such Big Data represents a computational challenge that cannot be addressed with conventional computational techniques. In the A-Brain project, we address this computational problem using cloud computing techniques on Microsoft Azure, relying on our complementary expertise in the area of scalable cloud data management and in the field of neuroimaging and genetics data analysis
KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments
Distributed infrastructures for computation and analytics are now evolving
towards an interconnected ecosystem allowing complex scientific workflows to be
executed across hybrid systems spanning from IoT Edge devices to Clouds, and
sometimes to supercomputers (the Computing Continuum). Understanding the
performance trade-offs of large-scale workflows deployed on such complex
Edge-to-Cloud Continuum is challenging. To achieve this, one needs to
systematically perform experiments, to enable their reproducibility and allow
other researchers to replicate the study and the obtained conclusions on
different infrastructures. This breaks down to the tedious process of
reconciling the numerous experimental requirements and constraints with
low-level infrastructure design choices.To address the limitations of the main
state-of-the-art approaches for distributed, collaborative experimentation,
such as Google Colab, Kaggle, and Code Ocean, we propose KheOps, a
collaborative environment specifically designed to enable cost-effective
reproducibility and replicability of Edge-to-Cloud experiments. KheOps is
composed of three core elements: (1) an experiment repository; (2) a notebook
environment; and (3) a multi-platform experiment methodology.We illustrate
KheOps with a real-life Edge-to-Cloud application. The evaluations explore the
point of view of the authors of an experiment described in an article (who aim
to make their experiments reproducible) and the perspective of their readers
(who aim to replicate the experiment). The results show how KheOps helps
authors to systematically perform repeatable and reproducible experiments on
the Grid5000 + FIT IoT LAB testbeds. Furthermore, KheOps helps readers to
cost-effectively replicate authors experiments in different infrastructures
such as Chameleon Cloud + CHI@Edge testbeds, and obtain the same conclusions
with high accuracies (> 88% for all performance metrics)
- âŠ