3,624 research outputs found
Recommended from our members
The Design and Implementation of Low-Latency Prediction Serving Systems
Machine learning is being deployed in a growing number of applications which demand real- time, accurate, and cost-efficient predictions under heavy query load. These applications employ a variety of machine learning frameworks and models, often composing several models within the same application. However, most machine learning frameworks and systems are optimized for model training and not deployment.In this thesis, I discuss three prediction serving systems designed to meet the needs of modern interactive machine learning applications. The key idea in this work is to utilize a decoupled, layered design that interposes systems on top of training frameworks to build low-latency, scalable serving systems. Velox introduced this decoupled architecture to enable fast online learning and model personalization in response to feedback. Clipper generalized this system architecture to be framework-agnostic and introduced a set of optimizations to reduce and bound prediction latency and improve prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. And InferLine provisions and manages the individual stages of prediction pipelines to minimize cost while meeting end-to-end tail latency constraints
VIoLET: A Large-scale Virtual Environment for Internet of Things
IoT deployments have been growing manifold, encompassing sensors, networks,
edge, fog and cloud resources. Despite the intense interest from researchers
and practitioners, most do not have access to large-scale IoT testbeds for
validation. Simulation environments that allow analytical modeling are a poor
substitute for evaluating software platforms or application workloads in
realistic computing environments. Here, we propose VIoLET, a virtual
environment for defining and launching large-scale IoT deployments within cloud
VMs. It offers a declarative model to specify container-based compute resources
that match the performance of the native edge, fog and cloud devices using
Docker. These can be inter-connected by complex topologies on which
private/public networks, and bandwidth and latency rules are enforced. Users
can configure synthetic sensors for data generation on these devices as well.
We validate VIoLET for deployments with > 400 devices and > 1500 device-cores,
and show that the virtual IoT environment closely matches the expected compute
and network performance at modest costs. This fills an important gap between
IoT simulators and real deployments.Comment: To appear in the Proceedings of the 24TH International European
Conference On Parallel and Distributed Computing (EURO-PAR), August 27-31,
2018, Turin, Italy, europar2018.org. Selected as a Distinguished Paper for
presentation at the Plenary Session of the conferenc
- …