42,944 research outputs found
When Two Choices Are not Enough: Balancing at Scale in Distributed Stream Processing
Carefully balancing load in distributed stream processing systems has a
fundamental impact on execution latency and throughput. Load balancing is
challenging because real-world workloads are skewed: some tuples in the stream
are associated to keys which are significantly more frequent than others. Skew
is remarkably more problematic in large deployments: more workers implies fewer
keys per worker, so it becomes harder to "average out" the cost of hot keys
with cold keys.
We propose a novel load balancing technique that uses a heaving hitter
algorithm to efficiently identify the hottest keys in the stream. These hot
keys are assigned to choices to ensure a balanced load, where is
tuned automatically to minimize the memory and computation cost of operator
replication. The technique works online and does not require the use of routing
tables. Our extensive evaluation shows that our technique can balance
real-world workloads on large deployments, and improve throughput and latency
by and respectively over the previous
state-of-the-art when deployed on Apache Storm.Comment: 12 pages, 14 Figures, this paper is accepted and will be published at
ICDE 201
Optimal Net-Load Balancing in Smart Grids with High PV Penetration
Mitigating Supply-Demand mismatch is critical for smooth power grid
operation. Traditionally, load curtailment techniques such as Demand Response
(DR) have been used for this purpose. However, these cannot be the only
component of a net-load balancing framework for Smart Grids with high PV
penetration. These grids can sometimes exhibit supply surplus causing
over-voltages. Supply curtailment techniques such as Volt-Var Optimizations are
complex and computationally expensive. This increases the complexity of
net-load balancing systems used by the grid operator and limits their
scalability. Recently new technologies have been developed that enable the
rapid and selective connection of PV modules of an installation to the grid.
Taking advantage of these advancements, we develop a unified optimal net-load
balancing framework which performs both load and solar curtailment. We show
that when the available curtailment values are discrete, this problem is
NP-hard and develop bounded approximation algorithms for minimizing the
curtailment cost. Our algorithms produce fast solutions, given the tight timing
constraints required for grid operation. We also incorporate the notion of
fairness to ensure that curtailment is evenly distributed among all the nodes.
Finally, we develop an online algorithm which performs net-load balancing using
only data available for the current interval. Using both theoretical analysis
and practical evaluations, we show that our net-load balancing algorithms
provide solutions which are close to optimal in a small amount of time.Comment: 11 pages. To be published in the 4th ACM International Conference on
Systems for Energy-Efficient Built Environments (BuildSys 17) Changes from
previous version: Fixed a bug in Algorithm 1 which was causing some min cost
solutions to be misse
Avoiding Braess' Paradox through Collective Intelligence
In an Ideal Shortest Path Algorithm (ISPA), at each moment each router in a
network sends all of its traffic down the path that will incur the lowest cost
to that traffic. In the limit of an infinitesimally small amount of traffic for
a particular router, its routing that traffic via an ISPA is optimal, as far as
cost incurred by that traffic is concerned. We demonstrate though that in many
cases, due to the side-effects of one router's actions on another routers
performance, having routers use ISPA's is suboptimal as far as global aggregate
cost is concerned, even when only used to route infinitesimally small amounts
of traffic. As a particular example of this we present an instance of Braess'
paradox for ISPA's, in which adding new links to a network decreases overall
throughput. We also demonstrate that load-balancing, in which the routing
decisions are made to optimize the global cost incurred by all traffic
currently being routed, is suboptimal as far as global cost averaged across
time is concerned. This is also due to "side-effects", in this case of current
routing decision on future traffic.
The theory of COllective INtelligence (COIN) is concerned precisely with the
issue of avoiding such deleterious side-effects. We present key concepts from
that theory and use them to derive an idealized algorithm whose performance is
better than that of the ISPA, even in the infinitesimal limit. We present
experiments verifying this, and also showing that a machine-learning-based
version of this COIN algorithm in which costs are only imprecisely estimated (a
version potentially applicable in the real world) also outperforms the ISPA,
despite having access to less information than does the ISPA. In particular,
this COIN algorithm avoids Braess' paradox.Comment: 28 page
Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine
Load balancing, operator instance collocations and horizontal scaling are
critical issues in Parallel Stream Processing Engines to achieve low data
processing latency, optimized cluster utilization and minimized communication
cost respectively. In previous work, these issues are typically tackled
separately and independently. We argue that these problems are tightly coupled
in the sense that they all need to determine the allocations of workloads and
migrate computational states at runtime. Optimizing them independently would
result in suboptimal solutions. Therefore, in this paper, we investigate how
these three issues can be modeled as one integrated optimization problem. In
particular, we first consider jobs where workload allocations have little
effect on the communication cost, and model the problem of load balance as a
Mixed-Integer Linear Program. Afterwards, we present an extended solution
called ALBIC, which support general jobs. We implement the proposed techniques
on top of Apache Storm, an open-source Parallel Stream Processing Engine. The
extensive experimental results over both synthetic and real datasets show that
our techniques clearly outperform existing approaches
Quasirandom Load Balancing
We propose a simple distributed algorithm for balancing indivisible tokens on
graphs. The algorithm is completely deterministic, though it tries to imitate
(and enhance) a random algorithm by keeping the accumulated rounding errors as
small as possible.
Our new algorithm surprisingly closely approximates the idealized process
(where the tokens are divisible) on important network topologies. On
d-dimensional torus graphs with n nodes it deviates from the idealized process
only by an additive constant. In contrast to that, the randomized rounding
approach of Friedrich and Sauerwald (2009) can deviate up to Omega(polylog(n))
and the deterministic algorithm of Rabani, Sinclair and Wanka (1998) has a
deviation of Omega(n^{1/d}). This makes our quasirandom algorithm the first
known algorithm for this setting which is optimal both in time and achieved
smoothness. We further show that also on the hypercube our algorithm has a
smaller deviation from the idealized process than the previous algorithms.Comment: 25 page
- …