17,380 research outputs found
Using Dedicated and Opportunistic Networks in Synergy for a Cost-effective Distributed Stream Processing Platform
This paper presents a case for exploiting the synergy of dedicated and
opportunistic network resources in a distributed hosting platform for data
stream processing applications. Our previous studies have demonstrated the
benefits of combining dedicated reliable resources with opportunistic resources
in case of high-throughput computing applications, where timely allocation of
the processing units is the primary concern. Since distributed stream
processing applications demand large volume of data transmission between the
processing sites at a consistent rate, adequate control over the network
resources is important here to assure a steady flow of processing. In this
paper, we propose a system model for the hybrid hosting platform where stream
processing servers installed at distributed sites are interconnected with a
combination of dedicated links and public Internet. Decentralized algorithms
have been developed for allocation of the two classes of network resources
among the competing tasks with an objective towards higher task throughput and
better utilization of expensive dedicated resources. Results from extensive
simulation study show that with proper management, systems exploiting the
synergy of dedicated and opportunistic resources yield considerably higher task
throughput and thus, higher return on investment over the systems solely using
expensive dedicated resources.Comment: 9 page
Maximizing Service Reliability in Distributed Computing Systems with Random Node Failures: Theory and Implementation
In distributed computing systems (DCSs) where server nodes can fail permanently with nonzero probability, the system performance can be assessed by means of the service reliability, defined as the probability of serving all the tasks queued in the DCS before all the nodes fail. This paper presents a rigorous probabilistic framework to analytically characterize the service reliability of a DCS in the presence of communication uncertainties and stochastic topological changes due to node deletions. The framework considers a system composed of heterogeneous nodes with stochastic service and failure times and a communication network imposing random tangible delays. The framework also permits arbitrarily specified, distributed load-balancing actions to be taken by the individual nodes in order to improve the service reliability. The presented analysis is based upon a novel use of the concept of stochastic regeneration, which is exploited to derive a system of difference-differential equations characterizing the service reliability. The theory is further utilized to optimize certain load-balancing policies for maximal service reliability; the optimization is carried out by means of an algorithm that scales linearly with the number of nodes in the system. The analytical model is validated using both Monte Carlo simulations and experimental data collected from a DCS testbed
Distributed Algorithms for Scheduling on Line and Tree Networks
We have a set of processors (or agents) and a set of graph networks defined
over some vertex set. Each processor can access a subset of the graph networks.
Each processor has a demand specified as a pair of vertices , along
with a profit; the processor wishes to send data between and . Towards
that goal, the processor needs to select a graph network accessible to it and a
path connecting and within the selected network. The processor requires
exclusive access to the chosen path, in order to route the data. Thus, the
processors are competing for routes/channels. A feasible solution selects a
subset of demands and schedules each selected demand on a graph network
accessible to the processor owning the demand; the solution also specifies the
paths to use for this purpose. The requirement is that for any two demands
scheduled on the same graph network, their chosen paths must be edge disjoint.
The goal is to output a solution having the maximum aggregate profit. Prior
work has addressed the above problem in a distibuted setting for the special
case where all the graph networks are simply paths (i.e, line-networks).
Distributed constant factor approximation algorithms are known for this case.
The main contributions of this paper are twofold. First we design a
distributed constant factor approximation algorithm for the more general case
of tree-networks. The core component of our algorithm is a tree-decomposition
technique, which may be of independent interest. Secondly, for the case of
line-networks, we improve the known approximation guarantees by a factor of 5.
Our algorithms can also handle the capacitated scenario, wherein the demands
and edges have bandwidth requirements and capacities, respectively.Comment: Accepted to PODC 2012, full versio
QuickCast: Fast and Efficient Inter-Datacenter Transfers using Forwarding Tree Cohorts
Large inter-datacenter transfers are crucial for cloud service efficiency and
are increasingly used by organizations that have dedicated wide area networks
between datacenters. A recent work uses multicast forwarding trees to reduce
the bandwidth needs and improve completion times of point-to-multipoint
transfers. Using a single forwarding tree per transfer, however, leads to poor
performance because the slowest receiver dictates the completion time for all
receivers. Using multiple forwarding trees per transfer alleviates this
concern--the average receiver could finish early; however, if done naively,
bandwidth usage would also increase and it is apriori unclear how best to
partition receivers, how to construct the multiple trees and how to determine
the rate and schedule of flows on these trees. This paper presents QuickCast, a
first solution to these problems. Using simulations on real-world network
topologies, we see that QuickCast can speed up the average receiver's
completion time by as much as while only using more
bandwidth; further, the completion time for all receivers also improves by as
much as faster at high loads.Comment: [Extended Version] Accepted for presentation in IEEE INFOCOM 2018,
Honolulu, H
Online Resource Inference in Network Utility Maximization Problems
The amount of transmitted data in computer networks is expected to grow
considerably in the future, putting more and more pressure on the network
infrastructures. In order to guarantee a good service, it then becomes
fundamental to use the network resources efficiently. Network Utility
Maximization (NUM) provides a framework to optimize the rate allocation when
network resources are limited. Unfortunately, in the scenario where the amount
of available resources is not known a priori, classical NUM solving methods do
not offer a viable solution. To overcome this limitation we design an overlay
rate allocation scheme that attempts to infer the actual amount of available
network resources while coordinating the users rate allocation. Due to the
general and complex model assumed for the congestion measurements, a passive
learning of the available resources would not lead to satisfying performance.
The coordination scheme must then perform active learning in order to speed up
the resources estimation and quickly increase the system performance. By
adopting an optimal learning formulation we are able to balance the tradeoff
between an accurate estimation, and an effective resources exploitation in
order to maximize the long term quality of the service delivered to the users
Clustering Algorithms for Scale-free Networks and Applications to Cloud Resource Management
In this paper we introduce algorithms for the construction of scale-free
networks and for clustering around the nerve centers, nodes with a high
connectivity in a scale-free networks. We argue that such overlay networks
could support self-organization in a complex system like a cloud computing
infrastructure and allow the implementation of optimal resource management
policies.Comment: 14 pages, 8 Figurs, Journa
- …