2,580 research outputs found
Evaluation and optimization of Big Data Processing on High Performance Computing Systems
Programa Oficial de Doutoramento en Investigación en Tecnoloxías da Información. 524V01[Resumo]
Hoxe en día, moitas organizacións empregan tecnoloxías Big Data para extraer
información de grandes volumes de datos. A medida que o tamaño destes volumes
crece, satisfacer as demandas de rendemento das aplicacións de procesamento
de datos masivos faise máis difícil. Esta Tese céntrase en avaliar e optimizar estas
aplicacións, presentando dúas novas ferramentas chamadas BDEv e Flame-MR. Por
unha banda, BDEv analiza o comportamento de frameworks de procesamento Big
Data como Hadoop, Spark e Flink, moi populares na actualidade. BDEv xestiona
a súa configuración e despregamento, xerando os conxuntos de datos de entrada
e executando cargas de traballo previamente elixidas polo usuario. Durante cada
execución, BDEv extrae diversas métricas de avaliación que inclúen rendemento,
uso de recursos, eficiencia enerxética e comportamento a nivel de microarquitectura.
Doutra banda, Flame-MR permite optimizar o rendemento de aplicacións Hadoop
MapReduce. En xeral, o seu deseño baséase nunha arquitectura dirixida por eventos
capaz de mellorar a eficiencia dos recursos do sistema mediante o solapamento da
computación coas comunicacións. Ademais de reducir o número de copias en memoria
que presenta Hadoop, emprega algoritmos eficientes para ordenar e mesturar os
datos. Flame-MR substitúe o motor de procesamento de datos MapReduce de xeito
totalmente transparente, polo que non é necesario modificar o código de aplicacións
xa existentes. A mellora de rendemento de Flame-MR foi avaliada de maneira exhaustiva
en sistemas clúster e cloud, executando tanto benchmarks estándar coma
aplicacións pertencentes a casos de uso reais. Os resultados amosan unha redución
de entre un 40% e un 90% do tempo de execución das aplicacións. Esta Tese proporciona
aos usuarios e desenvolvedores de Big Data dúas potentes ferramentas
para analizar e comprender o comportamento de frameworks de procesamento de
datos e reducir o tempo de execución das aplicacións sen necesidade de contar con
coñecemento experto para elo.[Resumen]
Hoy en día, muchas organizaciones utilizan tecnologías Big Data para extraer
información de grandes volúmenes de datos. A medida que el tamaño de estos volúmenes
crece, satisfacer las demandas de rendimiento de las aplicaciones de procesamiento
de datos masivos se vuelve más difícil. Esta Tesis se centra en evaluar y
optimizar estas aplicaciones, presentando dos nuevas herramientas llamadas BDEv
y Flame-MR. Por un lado, BDEv analiza el comportamiento de frameworks de procesamiento
Big Data como Hadoop, Spark y Flink, muy populares en la actualidad.
BDEv gestiona su configuración y despliegue, generando los conjuntos de datos de
entrada y ejecutando cargas de trabajo previamente elegidas por el usuario. Durante
cada ejecución, BDEv extrae diversas métricas de evaluación que incluyen rendimiento,
uso de recursos, eficiencia energética y comportamiento a nivel de microarquitectura.
Por otro lado, Flame-MR permite optimizar el rendimiento de aplicaciones
Hadoop MapReduce. En general, su diseño se basa en una arquitectura dirigida por
eventos capaz de mejorar la eficiencia de los recursos del sistema mediante el solapamiento
de la computación con las comunicaciones. Además de reducir el número
de copias en memoria que presenta Hadoop, utiliza algoritmos eficientes para ordenar
y mezclar los datos. Flame-MR reemplaza el motor de procesamiento de datos
MapReduce de manera totalmente transparente, por lo que no se necesita modificar
el código de aplicaciones ya existentes. La mejora de rendimiento de Flame-MR ha
sido evaluada de manera exhaustiva en sistemas clúster y cloud, ejecutando tanto
benchmarks estándar como aplicaciones pertenecientes a casos de uso reales. Los
resultados muestran una reducción de entre un 40% y un 90% del tiempo de ejecución
de las aplicaciones. Esta Tesis proporciona a los usuarios y desarrolladores de
Big Data dos potentes herramientas para analizar y comprender el comportamiento
de frameworks de procesamiento de datos y reducir el tiempo de ejecución de las
aplicaciones sin necesidad de contar con conocimiento experto para ello.[Abstract]
Nowadays, Big Data technologies are used by many organizations to extract
valuable information from large-scale datasets. As the size of these datasets increases,
meeting the huge performance requirements of data processing applications
becomes more challenging. This Thesis focuses on evaluating and optimizing these
applications by proposing two new tools, namely BDEv and Flame-MR. On the one
hand, BDEv allows to thoroughly assess the behavior of widespread Big Data processing
frameworks such as Hadoop, Spark and Flink. It manages the configuration
and deployment of the frameworks, generating the input datasets and launching the
workloads specified by the user. During each workload, it automatically extracts
several evaluation metrics that include performance, resource utilization, energy efficiency
and microarchitectural behavior. On the other hand, Flame-MR optimizes
the performance of existing Hadoop MapReduce applications. Its overall design is
based on an event-driven architecture that improves the efficiency of the system
resources by pipelining data movements and computation. Moreover, it avoids redundant
memory copies present in Hadoop, while also using efficient sort and merge
algorithms for data processing. Flame-MR replaces the underlying MapReduce data
processing engine in a transparent way and thus the source code of existing applications
does not require to be modified. The performance benefits provided by Flame-
MR have been thoroughly evaluated on cluster and cloud systems by using both
standard benchmarks and real-world applications, showing reductions in execution
time that range from 40% to 90%. This Thesis provides Big Data users with powerful
tools to analyze and understand the behavior of data processing frameworks and
reduce the execution time of the applications without requiring expert knowledge
BDEv 3.0: energy efficiency and microarchitectural characterization of Big Data processing frameworks
This is a post-peer-review, pre-copyedit version of an article published in Future Generation Computer Systems. The final authenticated version is available online at: https://doi.org/10.1016/j.future.2018.04.030[Abstract] As the size of Big Data workloads keeps increasing, the evaluation of distributed frameworks becomes a crucial task in order to identify potential performance bottlenecks that may delay the processing of large datasets. While most of the existing works generally focus only on execution time and resource utilization, analyzing other important metrics is key to fully understanding the behavior of these frameworks. For example, microarchitecture-level events can bring meaningful insights to characterize the interaction between frameworks and hardware. Moreover, energy consumption is also gaining increasing attention as systems scale to thousands of cores. This work discusses the current state of the art in evaluating distributed processing frameworks, while extending our Big Data Evaluator tool (BDEv) to extract energy efficiency and microarchitecture-level metrics from the execution of representative Big Data workloads. An experimental evaluation using BDEv demonstrates its usefulness to bring meaningful information from popular frameworks such as Hadoop, Spark and Flink.Ministerio de Economía, Industria y Competitividad; TIN2016-75845-PMinisterio de Educación; FPU14/02805Ministerio de Educación; FPU15/0338
An introduction to Graph Data Management
A graph database is a database where the data structures for the schema
and/or instances are modeled as a (labeled)(directed) graph or generalizations
of it, and where querying is expressed by graph-oriented operations and type
constructors. In this article we present the basic notions of graph databases,
give an historical overview of its main development, and study the main current
systems that implement them
Using Graph Properties to Speed-up GPU-based Graph Traversal: A Model-driven Approach
While it is well-known and acknowledged that the performance of graph
algorithms is heavily dependent on the input data, there has been surprisingly
little research to quantify and predict the impact the graph structure has on
performance. Parallel graph algorithms, running on many-core systems such as
GPUs, are no exception: most research has focused on how to efficiently
implement and tune different graph operations on a specific GPU. However, the
performance impact of the input graph has only been taken into account
indirectly as a result of the graphs used to benchmark the system.
In this work, we present a case study investigating how to use the properties
of the input graph to improve the performance of the breadth-first search (BFS)
graph traversal. To do so, we first study the performance variation of 15
different BFS implementations across 248 graphs. Using this performance data,
we show that significant speed-up can be achieved by combining the best
implementation for each level of the traversal. To make use of this
data-dependent optimization, we must correctly predict the relative performance
of algorithms per graph level, and enable dynamic switching to the optimal
algorithm for each level at runtime.
We use the collected performance data to train a binary decision tree, to
enable high-accuracy predictions and fast switching. We demonstrate empirically
that our decision tree is both fast enough to allow dynamic switching between
implementations, without noticeable overhead, and accurate enough in its
prediction to enable significant BFS speedup. We conclude that our model-driven
approach (1) enables BFS to outperform state of the art GPU algorithms, and (2)
can be adapted for other BFS variants, other algorithms, or more specific
datasets
A Comparative Study of Hadoop MapReduce, Apache Spark & Apache Flink for Data Science
Distributed data processing platforms for cloud computing are important tools for large-scale data analytics. Apache Hadoop MapReduce has become the de facto standard in this space, though its programming interface is relatively low-level, requiring many implementation steps even for simple analysis tasks. This has led to the development of advanced dataflow oriented platforms, most prominently Apache Spark and Apache Flink. Those not only aim to improve performance, but also provide high-level data processing functionality, such as filtering and join operators, which should make data analysis tasks easier to develop. But without comparison data available, how would data scientists know which system they should choose? This research compares: Apache Hadoop MapReduce; Apache Spark; and Apache Flink, from the perspectives of performance, usability and practicality for batch-oriented data analytics. We propose and apply a methodology which guides the conception of multidimensional software comparisons and the presentation of their results. The methodology was effective, providing direction and structure to the comparison, and should serve as helpful for future comparisons. The results confirm that Spark and Flink are superior to Hadoop MapReduce in performance and usability. Spark and Flink were similar in all three considerations, however as per the methodology, readers have the flexibility to adjust weightings to their needs, which could differentiate them. We also report on the design, execution and results of a large-scale usability study with a cohort of masters students, who learn and work with all three platforms, solving different use cases in data science contexts. Our findings show that Spark and Flink are preferred platforms over MapReduce. Among participants, there was no significant difference in perceived preference or development time between both Spark and Flink. These results were included in the usability component of the multidimensional comparison
- …