4 research outputs found
Model-driven development of data intensive applications over cloud resources
The proliferation of sensors over the last years has generated large amounts of raw data, forming data streams that need to be processed. In many cases, cloud resources are used for such processing, exploiting their flexibility, but these sensor streaming applications often need to support operational and control actions that have real-time and low-latency requirements that go beyond the cost effective and flexible solutions supported by existing cloud frameworks, such as Apache Kafka, Apache Spark Streaming, or Map-Reduce Streams. In this paper, we describe a model-driven and stepwise refinement methodological approach for streaming applications executed over clouds. The central role is assigned to a set of Petri Net models for specifying functional and non-functional requirements. They support model reuse, and a way to combine formal analysis, simulation, and approximate computation of minimal and maximal boundaries of non-functional requirements when the problem is either mathematically or computationally intractable. We show how our proposal can assist developers in their design and implementation decisions from a performance perspective. Our methodology allows to conduct performance analysis: The methodology is intended for all the engineering process stages, and we can (i) analyse how it can be mapped onto cloud resources, and (ii) obtain key performance indicators, including throughput or economic cost, so that developers are assisted in their development tasks and in their decision taking. In order to illustrate our approach, we make use of the pipelined wavefront array
Model-driven development of data intensive applications over cloud resources
The proliferation of sensors over the last years has generated large amounts
of raw data, forming data streams that need to be processed. In many cases,
cloud resources are used for such processing, exploiting their flexibility, but
these sensor streaming applications often need to support operational and
control actions that have real-time and low-latency requirements that go beyond
the cost effective and flexible solutions supported by existing cloud
frameworks, such as Apache Kafka, Apache Spark Streaming, or Map-Reduce
Streams. In this paper, we describe a model-driven and stepwise refinement
methodological approach for streaming applications executed over clouds. The
central role is assigned to a set of Petri Net models for specifying functional
and non-functional requirements. They support model reuse, and a way to combine
formal analysis, simulation, and approximate computation of minimal and maximal
boundaries of non-functional requirements when the problem is either
mathematically or computationally intractable. We show how our proposal can
assist developers in their design and implementation decisions from a
performance perspective. Our methodology allows to conduct performance
analysis: The methodology is intended for all the engineering process stages,
and we can (i) analyse how it can be mapped onto cloud resources, and (ii)
obtain key performance indicators, including throughput or economic cost, so
that developers are assisted in their development tasks and in their decision
taking. In order to illustrate our approach, we make use of the pipelined
wavefront array.Comment: Preprin
Autotuning wavefront patterns for heterogeneous architectures
Manual tuning of applications for heterogeneous parallel systems is tedious and complex.
Optimizations are often not portable, and the whole process must be repeated when moving
to a new system, or sometimes even to a different problem size.
Pattern based parallel programming models were originally designed to provide programmers
with an abstract layer, hiding tedious parallel boilerplate code, and allowing a focus on
only application specific issues. However, the constrained algorithmic model associated with
each pattern also enables the creation of pattern-specific optimization strategies. These can
capture more complex variations than would be accessible by analysis of equivalent unstructured
source code. These variations create complex optimization spaces. Machine learning
offers well established techniques for exploring such spaces.
In this thesis we use machine learning to create autotuning strategies for heterogeneous
parallel implementations of applications which follow the wavefront pattern. In a wavefront,
computation starts from one corner of the problem grid and proceeds diagonally like a wave
to the opposite corner in either two or three dimensions. Our framework partitions and
optimizes the work created by these applications across systems comprising multicore CPUs
and multiple GPU accelerators. The tuning opportunities for a wavefront include controlling
the amount of computation to be offloaded onto GPU accelerators, choosing the number of
CPU and GPU threads to process tasks, tiling for both CPU and GPU memory structures,
and trading redundant halo computation against communication for multiple GPUs.
Our exhaustive search of the problem space shows that these parameters are very sensitive
to the combination of architecture, wavefront instance and problem size. We design and
investigate a family of autotuning strategies, targeting single and multiple CPU + GPU
systems, and both two and three dimensional wavefront instances. These yield an average
of 87% of the performance found by offline exhaustive search, with up to 99% in some cases