1 research outputs found
Optimizing ETL Dataflow Using Shared Caching and Parallelization Methods
Extract-Transform-Load (ETL) handles large amount of data and manages
workload through dataflows. ETL dataflows are widely regarded as complex and
expensive operations in terms of time and system resources. In order to
minimize the time and the resources required by ETL dataflows, this paper
presents a framework to optimize dataflows using shared cache and
parallelization techniques. The framework classifies the components in an ETL
dataflow into different categories based on their data operation properties.
The framework then partitions the dataflow based on the classification at
different granularities. Furthermore, the framework applies optimization
techniques such as cache re-using, pipelining and multi-threading to the
already-partitioned dataflows. The proposed techniques reduce system memory
footprint and the frequency of copying data between different components, and
also take full advantage of the computing power of multi-core processors. The
experimental results show that the proposed optimization framework is 4.7 times
faster than the ordinary ETL dataflows (without using the proposed optimization
techniques), and outperforms the similar tool (Kettle)