Skip to main content
Article thumbnail
Location of Repository

SkewTune: Mitigating Skew in MapReduce Applications

By Yongchul Kwon, Magdalena Balazinska, Bill Howe and Jerome Rolia

Abstract

We present an automatic skew mitigation approach for userdefined MapReduce programs and present SkewTune, a system that implements this approach as a drop-in replacement for an existing MapReduce implementation. There are three key challenges: (a) require no extra input from the user yet work for all MapReduce applications, (b) be completely transparent, and (c) impose minimal overhead if there is no skew. The SkewTune approach addresses these challenges and works as follows: When a node in the cluster becomes idle, SkewTune identifies the task with the greatest expected remaining processing time. The unprocessed input data of this straggling task is then proactively repartitioned in a way that fully utilizes the nodes in the cluster and preserves the ordering of the input data so that the original output can be reconstructed by concatenation. We implement SkewTune as an extension to Hadoop and evaluate its effectiveness using several real applications. The results show that Skew-Tune can significantly reduce job runtime in the presence of skew and adds little to no overhead in the absence of skew. 1

Year: 2012
OAI identifier: oai:CiteSeerX.psu:10.1.1.363.231
Provided by: CiteSeerX
Download PDF:
Sorry, we are unable to provide the full text but you may find it at the following location(s):
  • http://citeseerx.ist.psu.edu/v... (external link)
  • http://homes.cs.washington.edu... (external link)
  • Suggested articles


    To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.