28 research outputs found

    DCCast: Efficient Point to Multipoint Transfers Across Datacenters

    Full text link
    Using multiple datacenters allows for higher availability, load balancing and reduced latency to customers of cloud services. To distribute multiple copies of data, cloud providers depend on inter-datacenter WANs that ought to be used efficiently considering their limited capacity and the ever-increasing data demands. In this paper, we focus on applications that transfer objects from one datacenter to several datacenters over dedicated inter-datacenter networks. We present DCCast, a centralized Point to Multi-Point (P2MP) algorithm that uses forwarding trees to efficiently deliver an object from a source datacenter to required destination datacenters. With low computational overhead, DCCast selects forwarding trees that minimize bandwidth usage and balance load across all links. With simulation experiments on Google's GScale network, we show that DCCast can reduce total bandwidth usage and tail Transfer Completion Times (TCT) by up to 50%50\% compared to delivering the same objects via independent point-to-point (P2P) transfers.Comment: 9th USENIX Workshop on Hot Topics in Cloud Computing, https://www.usenix.org/conference/hotcloud17/program/presentation/noormohammadpou

    Energy Efficient Tapered Data Networks for Big Data Processing in IP/WDM Networks

    Get PDF
    Classically the data produced by Big Data applications is transferred through the access and core networks to be processed in data centers where the resulting data is stored. In this work we investigate improving the energy efficiency of transporting Big Data by processing the data in processing nodes of limited processing and storage capacity along its journey through the core network to the data center. The amount of data transported over the core network will be significantly reduced each time the data is processed therefore we refer to such a network as an Energy Efficient Tapered Data Network. The results of a Mixed Integer linear Programming (MILP), developed to optimize the processing of Big Data in the Energy Efficient Tapered Data Networks, show significant reduction in network power consumption up to 76%

    SDN-enabled Resource Provisioning Framework for Geo-Distributed Streaming Analytics

    Get PDF
    Geographically distributed (geo-distributed) datacenters for stream data processing typically comprise multiple edges and core datacenters connected through Wide-Area Network (WAN) with a master node responsible for allocating tasks to worker nodes. Since WAN links significantly impact the performance of distributed task execution, the existing task assignment approach is unsuitable for distributed stream data processing with low latency and high throughput demand. In this paper, we propose SAFA, a resource provisioning framework using the Software-Defined Networking (SDN) concept with an SDN controller responsible for monitoring the WAN, selecting an appropriate subset of worker nodes, and assigning tasks to the designated worker nodes. We implemented the data plane of the framework in P4 and the control plane components in Python. We tested the performance of the proposed system on Apache Spark, Apache Storm, and Apache Flink using the Yahoo! streaming benchmark on a set of custom topologies. The results of the experiments validate that the proposed approach is viable for distributed stream processing and confirm that it can improve at least 1.64Ă— the processing time of incoming events of the current stream processing systems.</p

    Profit-aware distributed online scheduling for data-oriented tasks in cloud datacenters

    Full text link
    As there is an increasing trend to deploy geographically distributed (geo-distributed) cloud datacenters (DCs), the scheduling of data-oriented tasks in such cloud DC systems becomes an appealing research topic. Specifically, it is challenging to achieve the distributed online scheduling that can handle the tasks\u27 acceptance, data-transfers, and processing jointly and efficiently. In this paper, by considering the store-and-forward and anycast schemes, we formulate an optimization problem to maximize the time-average profit from serving data-oriented tasks in a cloud DC system and then leverage the Lyapunov optimization techniques to propose an efficient scheduling algorithm, i.e., GlobalAny. We also extend the proposed algorithm by designing a data-transfer acceleration scheme to reduce the data-transfer latency. Extensive simulations verify that our algorithms can maximize the time-average profit in a distributed online manner. The results also indicate that GlobalAny and GlobalAnyExt (i.e., GlobalAny with data-transfer acceleration) outperform several existing algorithms in terms of both time-average profit and computation time

    BDS+: An Inter-Datacenter Data Replication System With Dynamic Bandwidth Separation

    Get PDF
    Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the rich inter-DC overlay paths that exist in geo-distributed DCs, as well as the remaining bandwidth reserved for online traffic under fixed bandwidth separation scheme. To take advantage of these opportunities, we present BDS+, a near-optimal network system for large-scale inter-DC data replication. BDS+ is an application-level multicast overlay network with a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. Furthermore, in each overlay path, it leverages dynamic bandwidth separation to make use of the remaining available bandwidth reserved for online traffic. By constantly estimating online traffic demand and rescheduling bulk-data transfers accordingly, BDS+ can further speed up the massive data multicast. Through a pilot deployment in one of the largest online service providers and large-scale real-trace simulations, we show that BDS+ can achieve 3-5 x speedup over the provider's existing system and several well-known overlay routing baselines of static bandwidth separation. Moreover, dynamic bandwidth separation can further reduce the completion time of bulk data transfers by 1.2 to 1.3 times

    On the dynamics of valley times and its application to bulk-transfer scheduling

    Full text link
    Periods of low load have been used for the scheduling of non-interactive tasks since the early stages of computing. Nowadays, the scheduling of bulk transfers—i.e., large-volume transfers without precise timing, such as database distribution, resources replication or backups—stands out among such tasks, given its direct effect on both the performance and billing of networks. Through visual inspection of traffic-demand curves of diverse points of presence (PoP), either a network, link, Internet service provider or Internet exchange point, it becomes apparent that low-use periods of bandwidth demands occur at early morning, showing a noticeable convex shape. Such observation led us to study and model the time when such demands reach their minimum, on what we have named valley time of a PoP, as an approximation to the ideal moment to carry out bulk transfers. After studying and modeling single-PoP scenarios both temporally and spatially seeking homogeneity in the phenomenon, as well as its extension to multi-PoP scenarios or paths—a meta-PoP constructed as the aggregation of several single PoPs—, we propose a final predictor system for the valley time. This tool works as an oracle for scheduling bulk transfers, with different versions according to time scales and the desired trade-off between precision and complexity. The evaluation of the system, named VTP, has proven its usefulness with errors below an hour on estimating the occurrence of valley times, as well as errors around 10% in terms of bandwidth between the prediction and actual valley trafficThis work has been partially supported by the European Commission under the project H2020 METRO-HAUL (Project ID: 761727

    Latency-bandwidth tradeoffs in Internet applications

    Get PDF
    Wide-area Internet links are slow, expensive, and unreliable. This affects applications in two distinct ways. Back-end data processing applications, which need to transfer large amounts of data between data centers across the world, are primarily constrained by the limited capacity of Internet links. Front-end user facing applications, on the other hand, are primarily latency-sensitive, and are bottlenecked by the high, unpredictably variable delays in the wide-area network. Our work exploits this asymmetry in applications' requirements by developing techniques that trade off one of bandwidth and latency to improve the other. We first consider the problem of supporting analytics over the large volumes of geographically dispersed data produced by global-scale organizations. Current solutions for analyzing this data as a whole operate by copying it to a single central data center, an approach that incurs substantial data transfer costs. We instead propose an alternative geo-distributed approach, orchestrating distributed execution across data centers. Our system, Geode, incorporates two key optimizations --- a low-level syntactic network redundancy elimination mechanism, and a high-level semantically aware workload optimization process --- both of which operate by trading off increased processing overhead (and computation latency) within data centers for a reduction in cross-data center bandwidth usage. In experiments we find that Geode achieves an up to 360x cost reduction compared to the current centralized baseline on a range of workloads, both real and synthetic. Next, we evaluate a simple, general purpose technique for trading off bandwidth for reduced latency: initiate redundant copies of latency sensitive operations and take the first copy to complete. While redundancy has been explored in some past systems, its use is typically avoided because of a fear of the overhead that it adds. We study the latency-bandwidth tradeoff due to redundancy and (i) show via empirical evaluation that its use is indeed a net positive in a number of important applications, and (ii) provide a theoretical characterization of its effect, identifying when it should and should not be used and how systems can tune their use of redundancy to maximum effect. Our results suggest that redundancy should be used much more widely than it currently is
    corecore