Result Integrity Check for MapReduce Computation on Hybrid Clouds

Abstract — Large scale adoption of MapReduce computations on public clouds is hindered by the lack of trust on the participat-ing virtual machines, because misbehaving worker nodes can compromise the integrity of the computation result. In this paper, we propose a novel MapReduce framework, Cross Cloud MapRe-duce (CCMR), which overlays the MapReduce computation on top of a hybrid cloud: the master that is in control of the entire computation and guarantees result integrity runs on a private and trusted cloud, while normal workers run on a public cloud. In order to achieve high accuracy, CCMR proposes a result integrity check scheme on both the map phase and the reduce phase, which combines random task replication, random task verification, and credit accumulation; and CCMR strives to reduce the overhead by reducing cross-cloud communication. We implement our ap-proach based on Apache Hadoop MapReduce and evaluate our implementation on Amazon EC2. Both theoretical and experi-mental analysis show that our approach can guarantee high result integrity in a normal cloud environment while incurring non-negligible performance overhead (e.g., when 16.7 % workers are malicious, CCMR can guarantee at least 99.52 % of accuracy with 33.6 % of overhead when replication probability is 0.3 and the credit threshold is 50)