Resource management for data streaming applications

This dissertation investigates novel middleware mechanisms for building streaming applications. Developing streaming applications is a challenging task because (i) they are continuous in nature; (ii) they require fusion of data coming from multiple sources to derive higher level information; (iii) they require efficient transport of data from/to distributed sources and sinks; (iv) they need access to heterogeneous resources spanning sensor networks and high performance computing; and (v) they are time critical in nature. My thesis is that an intuitive programming abstraction will make it easier to build dynamic, distributed, and ubiquitous data streaming applications. Moreover, such an abstraction will enable an efficient allocation of shared and heterogeneous computational resources thereby making it easier for domain experts to build these applications. In support of the thesis, I present a novel programming abstraction, called DFuse, that makes it easier to develop these applications. A domain expert only needs to specify the input and output connections to fusion channels, and the fusion functions. The subsystems developed in this dissertation take care of instantiating the application, allocating resources for the application (via the scheduling heuristic developed in this dissertation) and dynamically managing the resources (via the dynamic scheduling algorithm presented in this dissertation). Through extensive performance evaluation, I demonstrate that the resources are allocated efficiently to optimize the throughput and latency constraints of an application.Ph.D.Committee Chair: Ramachandran, Umakishore; Committee Member: Chervenak, Ann; Committee Member: Cooper, Brian; Committee Member: Liu, Ling; Committee Member: Schwan, Karste

Automating Provisioning of Complete Software Stack in a Grid Environment

With scaling of data centers, clusters, and grids, it is going to be increasingly difficult to download, configure, install, update, and manage the software stack on the constituent nodes. Automation of this process is needed for the system to scale beyond a few hundred nodes. We present system architecture for a middleware service that performs this task in an automated manner. We propose that our architecture can be implemented as a grid service as part of the globus toolkit. Furthermore, we demonstrate the usefulness of the architecture through a prototype implementation which automates the whole process by integrating various existing solutions along with building intelligence into the middleware to handle such tasks

Interactive Grid Architecture for Application Service Providers

This paper presents our Interactive Grid architecture for Application Service Providers (I-GASP). We envision IGASP as a solution for making computers available primarily for interactive use in a grid computing environment. A user might access such a computer for running diverse applications such as graphics rendering, scientific visualization or mechanical CAD. I-GASP consists of a grid middleware for provisioning these computers, remote display technology that goes across firewalls and several techniques for making the computers suitable for use in a grid environment, namely controlled shell and desktop, dynamic accounts, admission control, and monitoring and management agents. To minimize the amount of user data that needs to migrate to the assigned computer before an interactive session begins, we present our affinity scheduling algorithm that favors a computer where the user has previously had an interactive session

Automating Provisioning of Complete Software Stack in a Grid Environment

With scaling of data centers, clusters, and grids, it is going to be increasingly difficult to download, configure, install, update, and manage the software stack on the constituent nodes. Automation of this process is needed for the system to scale beyond a few hundred nodes. We present system architecture for a middleware service that performs this task in an automated manner. We propose that our architecture can be implemented as a grid service as part of the globus toolkit. Furthermore, we demonstrate the usefulness of the architecture through a prototype implementation which automates the whole process by integrating various existing solutions along with building intelligence into the middleware to handle such tasks

Toward a standard ubiquitous computing framework

This paper surveys a variety of subsystems designed to be the building blocks from which sophisticated infrastructures for ubiquitous computing are assembled. Our experience shows that many of these building blocks fit neatly into one of five categories, each containing functionally-equivalent components. Effectively identifying the best-fit “lego pieces”, which in turn determines the composite functionality of the resulting infrastructure, is critical. The selection process, however, is impeded by the lack of convention for labeling these classes of building blocks. The lack of clarity with respect to what ready-made subsystems are available within each class often results in naive re-implementation of ready-made components, monolithic and clumsy implementations, and implementations that impose non-standard interfaces onto the applications above. This paper explores each class of subsystems in light of the experience gained over two years of active development of both ubiquitous computing applications and software infrastructures for their deployment

Streamline: a Scheduling Heuristic for Streaming Applications on the Grid

Scheduling a streaming application on high-performance computing (HPC) resources has to be sensitive to the computation and communication needs of each stage of the application dataflow graph to ensure QoS criteria such as latency and throughput. Since the grid has evolved out of traditional high-performance computing, the tools available for scheduling are more appropriate for batch-oriented applications. Our scheduler, called Streamline, considers the dynamic nature of the grid and runs periodically to adapt scheduling decisions using application requirements (per-stage computation and communication needs), application constraints (such as co-location of stages), and resource availability. The performance of Streamline is compared with an Optimal placement, Simulated Annealing (SA) approximations, and E-Condor, a streaming grid scheduler built using Condor. For kernels of streaming applications, we show that Streamline performs close to the Optimal and SA algorithms, and an order of magnitude better than E-Condor under non-uniform load conditions. We have also conducted scalability studies showing the advantage of Streamline over other approaches