Performance Models of Data Parallel DAG Workflows for Large Scale Data Analytics

Directed Acyclic Graph (DAG) workflows are widely used for large-scale data analytics in cluster-based distributed computing systems. Building an accurate performance model for a DAG on data-parallel frameworks (e.g., MapReduce) is critical to implement autonomic self-management big data systems. An accurate performance model is challenging because the allocation of pre-emptable system resources among parallel jobs may dynamically vary during execution. This resource allocation variation during execution makes it difficult to accurately estimate the execution time. In this paper, we tackle this challenge by proposing a new cost model, called Bottleneck Oriented Estimation (BOE), to estimate the allocation of preemptable resources by identifying the bottleneck to accurately predict task execution time. For a DAG workflow, we propose a state-based approach to iteratively use the resource allocation property among stages to estimate the overall execution plan. Extensive experiments were performed to validate these cost models with HiBench and TPC-H workloads. The BOE model outperforms the state-of-the-art models by a factor of five for task execution time estimation.Peer reviewe

Performance models of data parallel DAG workflows for large scale data analytics

irected Acyclic Graph (DAG) workflows are widely used for large-scale data analytics in cluster-based distributed computing systems. The performance model for a DAG on data-parallel frameworks (e.g., MapReduce) is a research challenge because the allocation of preemptable system resources among parallel jobs may dynamically vary during execution. This resource allocation variation during execution makes it difficult to accurately estimate the execution time. In this paper, we tackle this challenge by proposing a new cost model, called Bottleneck Oriented Estimation (BOE), to estimate the allocation of preemptable resources by identifying the bottleneck to accurately predict task execution time. For a DAG workflow, we propose a state-based approach to iteratively use the resource allocation property among stages to estimate the overall execution plan. Furthermore, to handle the skewness of various jobs, we refine the model with the order statistics theory to improve estimation accuracy. Extensive experiments were performed to validate these cost models with HiBench and TPC-H workloads. The BOE model outperforms the state-of-the-art models by a factor of five for task execution time estimation. For the refined skew-aware model, the average prediction error is under 3\% when estimating the execution time of 51 hybrid analytics (HiBench) and query (TPC-H) DAG workflows.Peer reviewe

Quantitative Nondestructive Testing of Broken Wires for Wire Rope Based on Magnetic and Infrared Information

The lifetime of wire rope is crucial in industry manufacturing, mining, and so on. The damage can be detected by using appropriate nondestructive testing techniques or destructive tests by cutting the part. For broken wires classification problems, this work is aimed at improving the recognition accuracy. Facing the defects at the exterior of the rope, a novel method for recognition of broken wires is firstly developed based on magnetic and infrared information fusion. A denoising method, which is adopted for magnetic signal, is proposed for eliminating baseline signal and wave strand. An image segmentation method is employed for parting the defects of infrared images. Characteristic vectors are extracted from magnetic images and infrared images, then kernel extreme learning machine network is applied to implement recognition of broken wires. Experimental results show that the denoising method and image segmentation are effective and the information fusion can improve the classification accuracy, which can provide useful information for estimating the residual lifetime of wire rope

Gear tooth health state monitoring based on morphological filtering and Wavelet transition

Gear box is one of the most important transmission components in mechanical systems. Fault diagnosis and state monitoring techniques for gear box have been studied for years. But in practical, gear box usually works under varying work conditions which has not been considered in most studies. In this paper, a novel state monitoring method is proposed for monitor gear box work with varying conditions. The vibration signal is de-nosied by morphological filtering. Then gear-mesh frequency band is extracted by wavelet transform. Dimensionless time indexes are used as state monitoring features. A regularization method is proposed to calculate gear tooth health index. Simulate and experiment signal are presented to illustrate the effectiveness of the method. The result indicates that, morphological filtering is an efficient method to de-nosie the vibration signal; wavelet transform can extract the gear-mesh frequency band; gear tooth health index can monitor gear tooth state based on vibration signal