196,589 research outputs found
Reusable, Extensible High-Level Data-Distribution Concept
A framework for high-level specification of data distributions in data-parallel application programs has been conceived. [As used here, distributions signifies means to express locality (more specifically, locations of specified pieces of data) in a computing system composed of many processor and memory components connected by a network.] Inasmuch as distributions exert a great effect on the performances of application programs, it is important that a distribution strategy be flexible, so that distributions can be adapted to the requirements of those programs. At the same time, for the sake of productivity in programming and execution, it is desirable that users be shielded from such error-prone, tedious details as those of communication and synchronization. As desired, the present framework enables a user to refine a distribution type and adjust it to optimize the performance of an application program and conceals, from the user, the low-level details of communication and synchronization. The framework provides for a reusable, extensible, data-distribution design, denoted the design pattern, that is independent of a concrete implementation. The design pattern abstracts over coding patterns that have been found to be commonly encountered in both manually and automatically generated distributed parallel programs. The following description of the present framework is necessarily oversimplified to fit within the space available for this article. Distributions are among the elements of a conceptual data-distribution machinery, some of the other elements being denoted domains, index sets, and data collections (see figure). Associated with each domain is one index set and one distribution. A distribution class interface (where "class" is used in the object-oriented-programming sense) includes operations that enable specification of the mapping of an index to a unit of locality. Thus, "Map(Index)" specifies a unit, while "LocalLayout(Index)" specifies the local address within that unit. The distribution class can be extended to enable specification of commonly used distributions or novel user-defined distributions. A data collection can be defined over a domain. The term "data collection" in this context signifies, more specifically, an abstraction of mappings from index sets to variables. Since the index set is distributed, the addresses of the variables are also distributed
Workflow Partitioning and Deployment on the Cloud using Orchestra
Orchestrating service-oriented workflows is typically based on a design model
that routes both data and control through a single point - the centralised
workflow engine. This causes scalability problems that include the unnecessary
consumption of the network bandwidth, high latency in transmitting data between
the services, and performance bottlenecks. These problems are highly prominent
when orchestrating workflows that are composed from services dispersed across
distant geographical locations. This paper presents a novel workflow
partitioning approach, which attempts to improve the scalability of
orchestrating large-scale workflows. It permits the workflow computation to be
moved towards the services providing the data in order to garner optimal
performance results. This is achieved by decomposing the workflow into smaller
sub workflows for parallel execution, and determining the most appropriate
network locations to which these sub workflows are transmitted and subsequently
executed. This paper demonstrates the efficiency of our approach using a set of
experimental workflows that are orchestrated over Amazon EC2 and across several
geographic network regions.Comment: To appear in Proceedings of the IEEE/ACM 7th International Conference
on Utility and Cloud Computing (UCC 2014
Grid-Brick Event Processing Framework in GEPS
Experiments like ATLAS at LHC involve a scale of computing and data
management that greatly exceeds the capability of existing systems, making it
necessary to resort to Grid-based Parallel Event Processing Systems (GEPS).
Traditional Grid systems concentrate the data in central data servers which
have to be accessed by many nodes each time an analysis or processing job
starts. These systems require very powerful central data servers and make
little use of the distributed disk space that is available in commodity
computers. The Grid-Brick system, which is described in this paper, follows a
different approach. The data storage is split among all grid nodes having each
one a piece of the whole information. Users submit queries and the system will
distribute the tasks through all the nodes and retrieve the result, merging
them together in the Job Submit Server. The main advantage of using this system
is the huge scalability it provides, while its biggest disadvantage appears in
the case of failure of one of the nodes. A workaround for this problem involves
data replication or backup.Comment: 6 pages; document for CHEP'03 conferenc
Energy-Aware Cloud Management through Progressive SLA Specification
Novel energy-aware cloud management methods dynamically reallocate
computation across geographically distributed data centers to leverage regional
electricity price and temperature differences. As a result, a managed VM may
suffer occasional downtimes. Current cloud providers only offer high
availability VMs, without enough flexibility to apply such energy-aware
management. In this paper we show how to analyse past traces of dynamic cloud
management actions based on electricity prices and temperatures to estimate VM
availability and price values. We propose a novel SLA specification approach
for offering VMs with different availability and price values guaranteed over
multiple SLAs to enable flexible energy-aware cloud management. We determine
the optimal number of such SLAs as well as their availability and price
guaranteed values. We evaluate our approach in a user SLA selection simulation
using Wikipedia and Grid'5000 workloads. The results show higher customer
conversion and 39% average energy savings per VM.Comment: 14 pages, conferenc
A Dataflow Language for Decentralised Orchestration of Web Service Workflows
Orchestrating centralised service-oriented workflows presents significant
scalability challenges that include: the consumption of network bandwidth,
degradation of performance, and single points of failure. This paper presents a
high-level dataflow specification language that attempts to address these
scalability challenges. This language provides simple abstractions for
orchestrating large-scale web service workflows, and separates between the
workflow logic and its execution. It is based on a data-driven model that
permits parallelism to improve the workflow performance. We provide a
decentralised architecture that allows the computation logic to be moved
"closer" to services involved in the workflow. This is achieved through
partitioning the workflow specification into smaller fragments that may be sent
to remote orchestration services for execution. The orchestration services rely
on proxies that exploit connectivity to services in the workflow. These proxies
perform service invocations and compositions on behalf of the orchestration
services, and carry out data collection, retrieval, and mediation tasks. The
evaluation of our architecture implementation concludes that our decentralised
approach reduces the execution time of workflows, and scales accordingly with
the increasing size of data sets.Comment: To appear in Proceedings of the IEEE 2013 7th International Workshop
on Scientific Workflows, in conjunction with IEEE SERVICES 201
P4CEP: Towards In-Network Complex Event Processing
In-network computing using programmable networking hardware is a strong trend
in networking that promises to reduce latency and consumption of server
resources through offloading to network elements (programmable switches and
smart NICs). In particular, the data plane programming language P4 together
with powerful P4 networking hardware has spawned projects offloading services
into the network, e.g., consensus services or caching services. In this paper,
we present a novel case for in-network computing, namely, Complex Event
Processing (CEP). CEP processes streams of basic events, e.g., stemming from
networked sensors, into meaningful complex events. Traditionally, CEP
processing has been performed on servers or overlay networks. However, we argue
in this paper that CEP is a good candidate for in-network computing along the
communication path avoiding detouring streams to distant servers to minimize
communication latency while also exploiting processing capabilities of novel
networking hardware. We show that it is feasible to express CEP operations in
P4 and also present a tool to compile CEP operations, formulated in our P4CEP
rule specification language, to P4 code. Moreover, we identify challenges and
problems that we have encountered to show future research directions for
implementing full-fledged in-network CEP systems.Comment: 6 pages. Author's versio
DART-MPI: An MPI-based Implementation of a PGAS Runtime System
A Partitioned Global Address Space (PGAS) approach treats a distributed
system as if the memory were shared on a global level. Given such a global view
on memory, the user may program applications very much like shared memory
systems. This greatly simplifies the tasks of developing parallel applications,
because no explicit communication has to be specified in the program for data
exchange between different computing nodes. In this paper we present DART, a
runtime environment, which implements the PGAS paradigm on large-scale
high-performance computing clusters. A specific feature of our implementation
is the use of one-sided communication of the Message Passing Interface (MPI)
version 3 (i.e. MPI-3) as the underlying communication substrate. We evaluated
the performance of the implementation with several low-level kernels in order
to determine overheads and limitations in comparison to the underlying MPI-3.Comment: 11 pages, International Conference on Partitioned Global Address
Space Programming Models (PGAS14
Discrete-event simulation of process control in low volume high value industries
This paper presents a new method of process control for set-up dominant processes. This new method known as Set-up Process Algorithm (SUPA) was compared with existing industrial practices and statistical techniques in the literature. To test the method’s robustness, a generic discrete-event simulation model was built. This model was used to test four different statistical approaches to process control. It was concluded that SUPA offers a method of process control for set-up dominant processes, which is easier to apply than classically derived SPC approaches, by using simple rules and a traffic light system based on design specification. Simulation analysis shows that SUPA: is more sensitive, at detecting an incapable process as it will monitor more units when a process is less capable; is more sensitive than PRE-Control at detecting mean shifts in a process. SUPA is also a nonparametric methodology and therefore robust against processes with non-Gaussian distributions
- …