Parallel and pipelined processing of large-scale mobile comminucation data using hadoop open-source framework

Due to copyright restrictions, the access to the full text of this article is only available via subscription.The fast increase in mobile device and bandwidth usage is generating big workloads on the IT infrastructures of mobile service providers and increasing management costs. These providers collect log files continuously and use these logs for billing, operational and marketing purposes. In this paper, we describe the design, implementation and efficient parallel processing of large-scale mobile logs using the open-source Hadoop-based low-cost private cloud system for near real-time analytics. We find that batching of small files, parallel loading and pipelining of different workloads by overlapping their disk-and-CPU intensive phases can have significant performance benefits. Optimizations were performed in the light of these findings. Our web-based interface helps users explore progress and performance of their workloads.Avea Lab ; European Commission ; IBM Shared University Research ; TÜBİTA

Parallel and pipelined processing of large-scale mobile comminucation data using hadoop open-source framework

Due to copyright restrictions, the access to the full text of this article is only available via subscription.The fast increase in mobile device and bandwidth usage is generating big workloads on the IT infrastructures of mobile service providers and increasing management costs. These providers collect log files continuously and use these logs for billing, operational and marketing purposes. In this paper, we describe the design, implementation and efficient parallel processing of large-scale mobile logs using the open-source Hadoop-based low-cost private cloud system for near real-time analytics. We find that batching of small files, parallel loading and pipelining of different workloads by overlapping their disk-and-CPU intensive phases can have significant performance benefits. Optimizations were performed in the light of these findings. Our web-based interface helps users explore progress and performance of their workloads.Avea Lab ; European Commission ; IBM Shared University Research ; TÜBİTA