11,812 research outputs found
Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff
The relative ease of collaborative data science and analysis has led to a
proliferation of many thousands or millions of of the same datasets
in many scientific and commercial domains, acquired or constructed at various
stages of data analysis across many users, and often over long periods of time.
Managing, storing, and recreating these dataset versions is a non-trivial task.
The fundamental challenge here is the : the more
storage we use, the faster it is to recreate or retrieve versions, while the
less storage we use, the slower it is to recreate or retrieve versions. Despite
the fundamental nature of this problem, there has been a surprisingly little
amount of work on it. In this paper, we study this trade-off in a principled
manner: we formulate six problems under various settings, trading off these
quantities in various ways, demonstrate that most of the problems are
intractable, and propose a suite of inexpensive heuristics drawing from
techniques in delay-constrained scheduling, and spanning tree literature, to
solve these problems. We have built a prototype version management system, that
aims to serve as a foundation to our DATAHUB system for facilitating
collaborative data science. We demonstrate, via extensive experiments, that our
proposed heuristics provide efficient solutions in practical dataset versioning
scenarios
A framework for the joint placement of edge service infrastructure and User Plane Functions for 5G
Achieving less than 1 ms end-to-end communication latency, required for certain 5G services and use cases, is imposing severe technical challenges for the deployment of next-generation networks. To achieve such an ambitious goal, the service infrastructure and User Plane Function (UPF) placement at the network edge, is mandatory. However, this solution implies a substantial increase in deployment and operational costs. To cost-effectively solve this joint placement problem, this paper introduces a framework to jointly address the placement of edge nodes (ENs) and UPFs. Our framework proposal relies on Integer Linear Programming (ILP) and heuristic solutions. The main objective is to determine the ENs and UPFs’ optimal number and locations to minimize overall costs while satisfying the service requirements. To this aim, several parameters and factors are considered, such as capacity, latency, costs and site restrictions. The proposed solutions are evaluated based on different metrics and the obtained results showcase over 20% cost savings for the service infrastructure deployment. Moreover, the gap between the UPF placement heuristic and the optimal solution is equal to only one UPF in the worst cases, and a computation time reduction of over 35% is achieved in all the use cases studied.Postprint (author's final draft
Multi-capacity bin packing with dependent items and its application to the packing of brokered workloads in virtualized environments
Providing resource allocation with performance
predictability guarantees is increasingly important in cloud
platforms, especially for data-intensive applications, in which
performance depends greatly on the available rates of data
transfer between the various computing/storage hosts underlying
the virtualized resources assigned to the application. Existing
resource allocation solutions either assume that applications
manage their data transfer between their virtualized resources, or
that cloud providers manage their internal networking resources.
With the increased prevalence of brokerage services in cloud
platforms, there is a need for resource allocation solutions that
provides predictability guarantees in settings, in which neither
application scheduling nor cloud provider resources can be
managed/controlled by the broker. This paper addresses this
problem, as we define the Network-Constrained Packing (NCP)
problem of finding the optimal mapping of brokered resources
to applications with guaranteed performance predictability. We
prove that NCP is NP-hard, and we define two special instances
of the problem, for which exact solutions can be found efficiently.
We develop a greedy heuristic to solve the general instance of the
NCP problem , and we evaluate its efficiency using simulations
on various application workloads, and network models.This work was done while author was at Boston University. It was partially supported by NSF CISE awards #1430145, #1414119, #1239021 and #1012798. (1430145 - NSF CISE; 1414119 - NSF CISE; 1239021 - NSF CISE; 1012798 - NSF CISE
On the Benefit of Virtualization: Strategies for Flexible Server Allocation
Virtualization technology facilitates a dynamic, demand-driven allocation and
migration of servers. This paper studies how the flexibility offered by network
virtualization can be used to improve Quality-of-Service parameters such as
latency, while taking into account allocation costs. A generic use case is
considered where both the overall demand issued for a certain service (for
example, an SAP application in the cloud, or a gaming application) as well as
the origins of the requests change over time (e.g., due to time zone effects or
due to user mobility), and we present online and optimal offline strategies to
compute the number and location of the servers implementing this service. These
algorithms also allow us to study the fundamental benefits of dynamic resource
allocation compared to static systems. Our simulation results confirm our
expectations that the gain of flexible server allocation is particularly high
in scenarios with moderate dynamics
Route Planning in Transportation Networks
We survey recent advances in algorithms for route planning in transportation
networks. For road networks, we show that one can compute driving directions in
milliseconds or less even at continental scale. A variety of techniques provide
different trade-offs between preprocessing effort, space requirements, and
query time. Some algorithms can answer queries in a fraction of a microsecond,
while others can deal efficiently with real-time traffic. Journey planning on
public transportation systems, although conceptually similar, is a
significantly harder problem due to its inherent time-dependent and
multicriteria nature. Although exact algorithms are fast enough for interactive
queries on metropolitan transit systems, dealing with continent-sized instances
requires simplifications or heavy preprocessing. The multimodal route planning
problem, which seeks journeys combining schedule-based transportation (buses,
trains) with unrestricted modes (walking, driving), is even harder, relying on
approximate solutions even for metropolitan inputs.Comment: This is an updated version of the technical report MSR-TR-2014-4,
previously published by Microsoft Research. This work was mostly done while
the authors Daniel Delling, Andrew Goldberg, and Renato F. Werneck were at
Microsoft Research Silicon Valle
Network-constrained packing of brokered workloads in virtualized environments
Providing resource allocation with performance predictability guarantees is increasingly important in cloud platforms, especially for data-intensive applications, in which performance depends greatly on the available rates of data transfer between the various computing/storage hosts underlying the virtualized resources assigned to the application. Existing resource allocation solutions either assume that applications manage their data transfer between their virtualized resources, or that cloud providers manage their internal networking resources.With the increased prevalence of brokerage services in cloud platforms, there is a need for resource allocation solutions that provides predictability guarantees in settings, in which neither application scheduling nor cloud provider resources can be managed/controlled by the broker. This paper addresses this problem, as we define the Network-Constrained Packing (NCP)problem of finding the optimal mapping of brokered resources to applications with guaranteed performance predictability. We prove that NCP is NP-hard, and we define two special instances of the problem, for which exact solutions can be found efficiently. We develop a greedy heuristic to solve the general instance of the NCP problem, and we evaluate its efficiency using simulations on various application workloads, and network models.This work is supported by NSF CISE CNS Award #1347522, # 1239021, # 1012798
Survey on Combinatorial Register Allocation and Instruction Scheduling
Register allocation (mapping variables to processor registers or memory) and
instruction scheduling (reordering instructions to increase instruction-level
parallelism) are essential tasks for generating efficient assembly code in a
compiler. In the last three decades, combinatorial optimization has emerged as
an alternative to traditional, heuristic algorithms for these two tasks.
Combinatorial optimization approaches can deliver optimal solutions according
to a model, can precisely capture trade-offs between conflicting decisions, and
are more flexible at the expense of increased compilation time.
This paper provides an exhaustive literature review and a classification of
combinatorial optimization approaches to register allocation and instruction
scheduling, with a focus on the techniques that are most applied in this
context: integer programming, constraint programming, partitioned Boolean
quadratic programming, and enumeration. Researchers in compilers and
combinatorial optimization can benefit from identifying developments, trends,
and challenges in the area; compiler practitioners may discern opportunities
and grasp the potential benefit of applying combinatorial optimization
Low Latency Geo-distributed Data Analytics
Low latency analytics on geographically distributed dat-asets (across datacenters, edge clusters) is an upcoming and increasingly important challenge. The dominant approach of aggregating all the data to a single data-center significantly inflates the timeliness of analytics. At the same time, running queries over geo-distributed inputs using the current intra-DC analytics frameworks also leads to high query response times because these frameworks cannot cope with the relatively low and variable capacity of WAN links. We present Iridium, a system for low latency geo-distri-buted analytics. Iridium achieves low query response times by optimizing placement of both data and tasks of the queries. The joint data and task placement op-timization, however, is intractable. Therefore, Iridium uses an online heuristic to redistribute datasets among the sites prior to queries ’ arrivals, and places the tasks to reduce network bottlenecks during the query’s ex-ecution. Finally, it also contains a knob to budget WAN usage. Evaluation across eight worldwide EC2 re-gions using production queries show that Iridium speeds up queries by 3 × − 19 × and lowers WAN usage by 15% − 64 % compared to existing baselines
- …