27 research outputs found
Efficient Approximation Algorithms for Scheduling Coflows with Total Weighted Completion Time in Identical Parallel Networks
This paper addresses the scheduling problem of coflows in identical parallel
networks, which is a well-known -hard problem. Coflow is a relatively new
network abstraction used to characterize communication patterns in data
centers. We consider both flow-level scheduling and coflow-level scheduling
problems. In the flow-level scheduling problem, flows within a coflow can be
transmitted through different network cores. However, in the coflow-level
scheduling problem, flows within a coflow must be transmitted through the same
network core. The key difference between these two problems lies in their
scheduling granularity. Previous approaches relied on linear programming to
solve the scheduling order. In this paper, we enhance the efficiency of solving
by utilizing the primal-dual method. For the flow-level scheduling problem, we
propose a -approximation algorithm with arbitrary release
times and a -approximation algorithm without release time,
where represents the number of network cores. Additionally, for the
coflow-level scheduling problem, we introduce a -approximation
algorithm with arbitrary release times and a -approximation algorithm
without release time. To validate the effectiveness of our proposed algorithms,
we conduct simulations using both synthetic and real traffic traces. The
results demonstrate the superior performance of our algorithms compared to
previous approach, emphasizing their practical utility
Weighted Scheduling of Time-Sensitive Coflows
Datacenter networks commonly facilitate the transmission of data in
distributed computing frameworks through coflows, which are collections of
parallel flows associated with a common task. Most of the existing research has
concentrated on scheduling coflows to minimize the time required for their
completion, i.e., to optimize the average dispatch rate of coflows in the
network fabric. Nevertheless, modern applications often produce coflows that
are specifically intended for online services and mission-crucial computational
tasks, necessitating adherence to specific deadlines for their completion. In
this paper, we introduce \wdcoflow,~ a new algorithm to maximize the weighted
number of coflows that complete before their deadline. By combining a dynamic
programming algorithm along with parallel inequalities, our heuristic solution
performs at once coflow admission control and coflow prioritization, imposing a
-order on the set of coflows. With extensive simulation, we demonstrate
the effectiveness of our algorithm in improving up to more coflows
that meet their deadline in comparison the best SoA solution, namely
. Furthermore, when weights are used to differentiate
coflow classes, \wdcoflow~ is able to improve the admission per class up to
, while increasing the average weighted coflow admission rate.Comment: Submitted to IEEE Transactions on Cloud Computing. Parts of this work
have been presented at IFIP Networking 202
Scheduling Coflows for Minimizing the Makespan in Identical Parallel Networks
With the development of technology, parallel computing applications have been
commonly executed in large data centers. These parallel computing applications
include the computation phase and communication phase, and work is completed by
repeatedly executing these two phases. However, due to the ever-increasing
computing demands, large data centers are burdened with massive communication
demands. Coflow is a recently proposed networking abstraction to capture
communication patterns in data-parallel computing frameworks. This paper
focuses on the coflow scheduling problem in identical parallel networks, where
the goal is to minimize makespan, the maximum completion time of coflows. The
coflow scheduling problem in huge data center is considered one of the most
significant -hard problems. In this paper, coflow can be considered as
either a divisible or an indivisible case. Distinct flows in a divisible coflow
can be transferred through different network cores, while those in an
indivisible coflow can only be transferred through the same network core. In
the divisible coflow scheduling problem, this paper proposes a
-approximation algorithm, and a
-approximation algorithm, where is the number
of network cores. In the indivisible coflow scheduling problem, this paper
proposes a -approximation algorithm. Finally, we simulate our proposed
algorithm and Weaver's [Huang \textit{et al.}, In 2020 IEEE International
Parallel and Distributed Processing Symposium (IPDPS), pages 1071-1081, 2020.]
and compare the performance of our algorithms with that of Weaver's
Efficient Approximation Algorithms for Scheduling Coflows with Precedence Constraints in Identical Parallel Networks to Minimize Weighted Completion Time
This paper focuses on the problem of coflow scheduling with precedence
constraints in identical parallel networks, which is a well-known
-hard problem. Coflow is a relatively new network abstraction
used to characterize communication patterns in data centers. Both flow-level
scheduling and coflow-level scheduling problems are examined, with the key
distinction being the scheduling granularity. The proposed algorithm
effectively determines the scheduling order of coflows by employing the
primal-dual method. When considering workload sizes and weights that are
dependent on the network topology in the input instances, our proposed
algorithm for the flow-level scheduling problem achieves an approximation ratio
of where is the coflow number of the longest path in the
directed acyclic graph (DAG). Additionally, when taking into account workload
sizes that are topology-dependent, the algorithm achieves an approximation
ratio of , where represents the ratio of maximum weight to
minimum weight. For the coflow-level scheduling problem, the proposed algorithm
achieves an approximation ratio of , where is the number of
network cores, when considering workload sizes and weights that are
topology-dependent. Moreover, when considering workload sizes that are
topology-dependent, the algorithm achieves an approximation ratio of
. In the coflows of multi-stage job scheduling problem, the proposed
algorithm achieves an approximation ratio of . Although our
theoretical results are based on a limited set of input instances, experimental
findings show that the results for general input instances outperform the
theoretical results, thereby demonstrating the effectiveness and practicality
of the proposed algorithm.Comment: arXiv admin note: substantial text overlap with arXiv:2306.0229
Scheduling Coflows for Minimizing the Total Weighted Completion Time in Identical Parallel Networks
Coflow is a recently proposed network abstraction to capture communication
patterns in data centers. The coflow scheduling problem in large data centers
is one of the most important -hard problems. Previous research on coflow
scheduling focused mainly on the single-switch model. However, with recent
technological developments, this single-core model is no longer sufficient.
This paper considers the coflow scheduling problem in identical parallel
networks. The identical parallel network is an architecture based on multiple
network cores running in parallel. Coflow can be considered as divisible or
indivisible. Different flows in a divisible coflow can be transmitted through
different network cores. Considering the divisible coflow scheduling problem,
we propose a -approximation algorithm with arbitrary release
times, and a -approximation without release time, where is
the number of network cores. On the other hand, when coflow is indivisible, we
propose a -approximation algorithm with arbitrary release
times, and a -approximation without release time
Scheduling Coflows for Minimizing the Total Weighted Completion Time in Heterogeneous Parallel Networks
Coflow is a network abstraction used to represent communication patterns in
data centers. The coflow scheduling problem in large data centers is one of the
most important -hard problems. Many previous studies on coflow scheduling
mainly focus on the single-core model. However, with the growth of data
centers, this single-core model is no longer sufficient. This paper considers
the coflow scheduling problem in heterogeneous parallel networks. The
heterogeneous parallel network is an architecture based on multiple network
cores running in parallel. In this paper, two polynomial-time approximation
algorithms are developed for scheduling divisible and indivisible coflows in
heterogeneous parallel networks, respectively. Both algorithms achieve an
approximation ratio of with arbitrary release times.Comment: arXiv admin note: text overlap with arXiv:2204.0265
Recommended from our members
Resource Allocation In Large-Scale Distributed Systems
The focus of this dissertation is design and analysis of scheduling algorithms for distributed computer systems, i.e., data centers. Today’s data centers can contain thousands of servers and typically use a multi-tier switch network to provide connectivity among the servers. Data centers are the host for execution of various data-parallel applications. As an abstraction, a job in a data center can be thought of as a group of interdependent tasks, each with various requirements which need to be scheduled for execution on the servers and the data flows between the tasks that need to be scheduled in the switch network. In this thesis, we study both flow and task scheduling problems under the features of modern parallel computing frameworks.For the flow scheduling problem, we study three models.
The first model considers a general network topology where flows among the various source-destination pairs of servers are generated dynamically over time. The goal is to assign the end-to-end data flows among the available paths in order to efficiently balance the load in the network. We propose a myopic algorithm that is computationally efficient and prove that it asymptotically minimizes the total network cost using a convex optimization model, fluid limit and Lyapunov analysis. We further propose randomized versions of our myopic algorithm.
The second model consider the case that there is dependence among flows. Specifically, a coflow is defined as a collection of parallel flows whose completion time is determined by the completion time of the last flow in the collection. Our main result is a 5-approximation deterministic algorithm that schedule coflows in polynomial time so as to minimize the total weighted completion times. The key ingredient of our approach is an improved linear program formulation for sorting the coflows followed by a simple list scheduling policy.
Lastly, we study scheduling coflows of multi-stage jobs to minimize the jobs’ total weighted completion times. Each job is represented by a DAG (Directed Acyclic Graph) among its coflows that captures the dependencies among the coflows. We define g(m) = log(m)/log(log(m)) and h(m, μ) = log(mμ)/(log(log(mμ)), where m is number of servers, μ is the maximum number of coflows in a job. We develop two algorithms with approximation ratios O(√μg(m)) and O(√μg(m)h(m, μ)) for jobs with general DAGs and rooted trees, respectively. The algorithms rely on random delaying and merging optimal schedules of the coflows in the jobs’ DAG, followed by enforcing dependency among coflows and the links’ capacity constraints.
For the task scheduling problem, we study two models. We consider a setting where each job consists of a set of parallel tasks that need to be processed on different servers, and the job is completed once all its tasks finish processing. In the first model, each job is associated with a utility which is a decreasing function of its completion time. The objective is to schedule tasks in a way that achieves max-min fairness for jobs’ utilities. We first show a strong result regarding NP-hardness of this problem. We then proceed to define two notions of approximation solutions and develop scheduling algorithms that provide guarantees under these approximation notions, using dynamic programming and random perturbation of tasks’ processing times. In the second model, we further assume that processing times of tasks can be server dependent and a server can process (pack) multiple tasks at the same time subject to its capacity. We then propose three algorithms with approximation ratios of 4, (6 + ε), and 24 for different cases where preemption and migration of tasks among the servers are or are not allowed. Our algorithms use a combination of linear program relaxation and greedy packing techniques.
To demonstrate the gains in practice, we evaluate all the proposed algorithms and compare their performances with the prior approaches through extensive simulations using real and synthesized traffic traces. We hope this work inspires improvements to existing job management and scheduling in distributed computer systems
QuickCast: Fast and Efficient Inter-Datacenter Transfers using Forwarding Tree Cohorts
Large inter-datacenter transfers are crucial for cloud service efficiency and
are increasingly used by organizations that have dedicated wide area networks
between datacenters. A recent work uses multicast forwarding trees to reduce
the bandwidth needs and improve completion times of point-to-multipoint
transfers. Using a single forwarding tree per transfer, however, leads to poor
performance because the slowest receiver dictates the completion time for all
receivers. Using multiple forwarding trees per transfer alleviates this
concern--the average receiver could finish early; however, if done naively,
bandwidth usage would also increase and it is apriori unclear how best to
partition receivers, how to construct the multiple trees and how to determine
the rate and schedule of flows on these trees. This paper presents QuickCast, a
first solution to these problems. Using simulations on real-world network
topologies, we see that QuickCast can speed up the average receiver's
completion time by as much as while only using more
bandwidth; further, the completion time for all receivers also improves by as
much as faster at high loads.Comment: [Extended Version] Accepted for presentation in IEEE INFOCOM 2018,
Honolulu, H