POP/FED: Progressive Query Optimization for Federated Queries in DB2

Federated queries are regular relational queries accessing data on one or more remote relational or non-relational data sources, possibly combining them with tables stored in the federated DBMS server. Their execution is typically divided between the federated server and the remote data sources. Outdated and incomplete statistics have a bigger impact on federated DBMS than on regular DBMS, as maintenance of federated statistics is unequally more complicated and expensive than the maintenance of the local statistics; consequently bad performance commonly occurs for federated queries due to the selection of a suboptimal query plan. To solve this problem we propose a progressive optimization technique for federated queries called POP/FED by extending the state of the art for progressive reoptimization for local source queries, POP [4]. POP/FED uses (a) an opportunistic, but risk controlled reoptimization technique for federated DBMS, (b) a technique for multiple reoptimizations during federated query processing with a strategy to discover redundant and eliminate partial results, and (c) a mechanism to eagerly procure statistics in a federated environment. In this demonstration we showcase POP/FED implemented in a prototype version of WebSphere Information Integrator for DB2 using the TPC-H benchmark database and its workload. For selected queries of the workload we show unique features including multi-round reoptimizations using both a new graphical reoptimization progress monitor POPMonitor and the DB2 graphical plan explain tool.1

Progressive Optimization in a Shared-Nothing Parallel Database

Commercial enterprise data warehouses are typically implemented on parallel databases due to the inherent scalability and performance limitation of a serial architecture. Queries used in such large data warehouses can contain complex predicates as well as multiple joins, and the resulting query execution plans generated by the optimizer may be sub-optimal due to mis-estimates of row cardinalities. Progressive optimization (POP) is an approach to detect cardinality estimation errors by monitoring actual cardinalities at run-time and to recover by triggering re-optimization with the actual cardinalities measured. However, the original serial POP solution is based on a serial processing architecture, and the core ideas cannot be readily applied to a parallel shared-nothing environment. Extending the serial POP to a parallel environment is a challenging problem since we need to determine when and how we can trigger re-optimization based on cardinalities collected from multiple independent nodes. In this paper, we present a comprehensive and practical solution to this problem, including several novel voting schemes whether to trigger re-optimization, a mechanism to reuse local intermediate results across nodes as a partitioned materialized view, several flavors of parallel checkpoint operators, and parallel checkpoint processing methods using efficient communication protocols. This solution has been prototyped in a leading commercial parallel DBMS. We have performed extensive experiments using the TPC-H benchmark and a real-world database. Experimental results show that our solution has negligible runtime overhead and accelerates the performance of complex OLAP queries by up to a factor of 22.1117scopu