1 research outputs found
Detecting straggler MapReduce tasks in big data processing infrastructure by neural network
Straggler task detection is one of the main challenges in applying MapReduce
for parallelizing and distributing large-scale data processing. It is defined
as detecting running tasks on weak nodes. Considering two stages in the Map
phase copy, combine and three stages of Reduce shuffle, sort and reduce, the
total execution time is the total sum of the execution time of these five
stages. Estimating the correct execution time in each stage that results in
correct total execution time is the primary purpose of this paper. The proposed
method is based on the application of a backpropagation Neural Network NN on
the Hadoop for the detection of straggler tasks, to estimate the remaining
execution time of tasks that is very important in straggler task detection.
Results achieved have been compared with popular algorithms in this domain such
as LATE, ESAMR and the real remaining time for WordCount and Sort benchmarks,
and shown able to detect straggler tasks and estimate execution time
accurately. Besides, it supports to accelerate task execution time