Skip to main content
Article thumbnail
Location of Repository

Jockey: Guaranteed Job Latency in Data Parallel Clusters

By Andrew D. Ferguson, Eric Boutin, Peter Bodik, Rodrigo Fonseca and Srikanth Kandula

Abstract

Data processing frameworks such as MapReduce [8] and Dryad [11] are used today in business environments where customers expect guaranteed performance. To date, however, these systems are not capable of providing guarantees on job latency because scheduling policies are based on fairsharing, and operators seek high cluster use through statistical multiplexing and over-subscription. With Jockey, we provide latency SLOs for data parallel jobs written in SCOPE. Jockey precomputes statistics using a simulator that captures the job’s complex internal dependencies, accurately and efficiently predicting the remaining run time at different resource allocations and in different stages of the job. Our control policy monitors a job’s performance, and dynamically adjusts resource allocation in the shared cluster in order to maximize the job’s economic utility while minimizing its impact on the rest of the cluster. In our experiments in Microsoft’s production Cosmos clusters, Jockey meets the specified job latency SLOs and responds to changes in cluster conditions

Topics: General Terms Algorithms, Performance Keywords deadline, scheduling, SLO, data parallel, dynamic adaptation, Dryad, MapReduce
Year: 2012
OAI identifier: oai:CiteSeerX.psu:10.1.1.353.1085
Provided by: CiteSeerX
Download PDF:
Sorry, we are unable to provide the full text but you may find it at the following location(s):
  • http://citeseerx.ist.psu.edu/v... (external link)
  • http://research.microsoft.com/... (external link)
  • Suggested articles


    To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.