A peer to peer approach to large scale information monitoring

Issued as final reportNational Science Foundation (U.S.

Optimizing Multiple Distributed Stream Queries Using Hierarchical Network Partitions

We consider the problem of query optimization in distributed data stream systems where multiple continuous queries may be executing simultaneously. In order to achieve the best performance, query planning (such as join ordering) must be considered in conjunction with deployment planning (e.g., assigning operators to physical nodes with optimal ordering). However, such a combination involves not only a large number of network nodes but also many query operators, resulting in an extremely large search space for optimal solutions. Our paper aims at addressing this problem by utilizing hierarchical network partitions. We propose two algorithms- the Top-Down algorithm and the Bottom-Up algorithm which utilize hierarchical network partitions to provide scalable query optimization. Formal analysis is presented to establish the bounds on the search-space and to show the sub-optimality of our algorithms. We have implemented both algorithms in the IFLOW [14] system, an adaptive distributed stream management system. Through simulations and experiments using a prototype deployed on Emulab [1] we demonstrate the effectiveness of our algorithms. 1