87 research outputs found
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
Machine Learning models are often composed of pipelines of transformations.
While this design allows to efficiently execute single model components at
training time, prediction serving has different requirements such as low
latency, high throughput and graceful performance degradation under heavy load.
Current prediction serving systems consider models as black boxes, whereby
prediction-time-specific optimizations are ignored in favor of ease of
deployment. In this paper, we present PRETZEL, a prediction serving system
introducing a novel white box architecture enabling both end-to-end and
multi-model optimizations. Using production-like model pipelines, our
experiments show that PRETZEL is able to introduce performance improvements
over different dimensions; compared to state-of-the-art approaches PRETZEL is
on average able to reduce 99th percentile latency by 5.5x while reducing memory
footprint by 25x, and increasing throughput by 4.7x.Comment: 16 pages, 14 figures, 13th USENIX Symposium on Operating Systems
Design and Implementation (OSDI), 201
Improving the Expressiveness of Deep Learning Frameworks with Recursion
Recursive neural networks have widely been used by researchers to handle
applications with recursively or hierarchically structured data. However,
embedded control flow deep learning frameworks such as TensorFlow, Theano,
Caffe2, and MXNet fail to efficiently represent and execute such neural
networks, due to lack of support for recursion. In this paper, we add recursion
to the programming model of existing frameworks by complementing their design
with recursive execution of dataflow graphs as well as additional APIs for
recursive definitions. Unlike iterative implementations, which can only
understand the topological index of each node in recursive data structures, our
recursive implementation is able to exploit the recursive relationships between
nodes for efficient execution based on parallel computation. We present an
implementation on TensorFlow and evaluation results with various recursive
neural network models, showing that our recursive implementation not only
conveys the recursive nature of recursive neural networks better than other
implementations, but also uses given resources more effectively to reduce
training and inference time.Comment: Appeared in EuroSys 2018. 13 pages, 11 figure
Data Analytics Service Composition and Deployment on Edge Devices
Data analytics on edge devices has gained rapid growth in research, industry, and different aspects of our daily life. This topic still faces many challenges such as limited computation resource on edge devices. In this paper, we further identify two main challenges: the composition and deployment of data analytics services on edge devices. We present the Zoo system to address these two challenge: on one hand, it provides simple and concise domain-specific language to enable easy and and type-safe composition of different data analytics services; on the other, it utilises multiple deployment backends, including Docker container, JavaScript, and MirageOS, to accommodate the heterogeneous edge deployment environment. We show the expressiveness of Zoo with a use case, and thoroughly compare the performance of different deployment backends in evaluation
- …