3,340 research outputs found
Buffer Capacity Computation for Throughput Constrained Streaming Applications with Data-Dependent Inter-Task Communication
Streaming applications are often implemented as task graphs, in which data is communicated from task to task over buffers. Currently, techniques exist to compute buffer capacities that guarantee satisfaction of the throughput constraint if the amount of data produced and consumed by the tasks is known at design-time. However, applications such as audio and video decoders have tasks that produce and consume an amount of data that depends on the decoded stream. This paper introduces a dataflow model that allows for data-dependent communication, together with an algorithm that computes buffer capacities that guarantee satisfaction of a throughput constraint. The applicability of this algorithm is demonstrated by computing buffer capacities for an H.263 video decoder
Blazes: Coordination Analysis for Distributed Programs
Distributed consistency is perhaps the most discussed topic in distributed
systems today. Coordination protocols can ensure consistency, but in practice
they cause undesirable performance unless used judiciously. Scalable
distributed architectures avoid coordination whenever possible, but
under-coordinated systems can exhibit behavioral anomalies under fault, which
are often extremely difficult to debug. This raises significant challenges for
distributed system architects and developers. In this paper we present Blazes,
a cross-platform program analysis framework that (a) identifies program
locations that require coordination to ensure consistent executions, and (b)
automatically synthesizes application-specific coordination code that can
significantly outperform general-purpose techniques. We present two case
studies, one using annotated programs in the Twitter Storm system, and another
using the Bloom declarative language.Comment: Updated to include additional materials from the original technical
report: derivation rules, output stream label
Collaborative Reuse of Streaming Dataflows in IoT Applications
Distributed Stream Processing Systems (DSPS) like Apache Storm and Spark
Streaming enable composition of continuous dataflows that execute persistently
over data streams. They are used by Internet of Things (IoT) applications to
analyze sensor data from Smart City cyber-infrastructure, and make active
utility management decisions. As the ecosystem of such IoT applications that
leverage shared urban sensor streams continue to grow, applications will
perform duplicate pre-processing and analytics tasks. This offers the
opportunity to collaboratively reuse the outputs of overlapping dataflows,
thereby improving the resource efficiency. In this paper, we propose
\emph{dataflow reuse algorithms} that given a submitted dataflow, identifies
the intersection of reusable tasks and streams from a collection of running
dataflows to form a \emph{merged dataflow}. Similar algorithms to unmerge
dataflows when they are removed are also proposed. We implement these
algorithms for the popular Apache Storm DSPS, and validate their performance
and resource savings for 35 synthetic dataflows based on public OPMW workflows
with diverse arrival and departure distributions, and on 21 real IoT dataflows
from RIoTBench.Comment: To appear in IEEE eScience Conference 201
Modeling, Analysis, and Hard Real-time Scheduling of Adaptive Streaming Applications
In real-time systems, the application's behavior has to be predictable at
compile-time to guarantee timing constraints. However, modern streaming
applications which exhibit adaptive behavior due to mode switching at run-time,
may degrade system predictability due to unknown behavior of the application
during mode transitions. Therefore, proper temporal analysis during mode
transitions is imperative to preserve system predictability. To this end, in
this paper, we initially introduce Mode Aware Data Flow (MADF) which is our new
predictable Model of Computation (MoC) to efficiently capture the behavior of
adaptive streaming applications. Then, as an important part of the operational
semantics of MADF, we propose the Maximum-Overlap Offset (MOO) which is our
novel protocol for mode transitions. The main advantage of this transition
protocol is that, in contrast to self-timed transition protocols, it avoids
timing interference between modes upon mode transitions. As a result, any mode
transition can be analyzed independently from the mode transitions that
occurred in the past. Based on this transition protocol, we propose a hard
real-time analysis as well to guarantee timing constraints by avoiding
processor overloading during mode transitions. Therefore, using this protocol,
we can derive a lower bound and an upper bound on the earliest starting time of
the tasks in the new mode during mode transitions in such a way that hard
real-time constraints are respected.Comment: Accepted for presentation at EMSOFT 2018 and for publication in IEEE
Transactions on Computer-Aided Design of Integrated Circuits and Systems
(TCAD) as part of the ESWEEK-TCAD special issu
Computation of Buffer Capacities for Throughput Constrained and Data Dependent Inter-Task Communication
Streaming applications are often implemented as task graphs. Currently, techniques exist to derive buffer capacities that guarantee satisfaction of a throughput constraint for task graphs in which the inter-task communication is data-independent, i.e. the amount of data produced and consumed is independent of the data values in the processed stream. This paper presents a technique to compute buffer capacities that satisfy a throughput constraint for task graphs with data dependent inter-task communication, given that the task graph is a chain. We demonstrate the applicability of the approach by computing buffer capacities for an MP3 playback application, of which the MP3 decoder has a variable consumption rate. We are not aware of alternative approaches to compute buffer capacities that guarantee satisfaction of the throughput constraint for this application
- …