ITR/SY: a distributed programming infrastructure for integrating smart sensors

Issued as final reportNational Science Foundation (U.S.

Memory Optimizations for Distributed Stream-based Applications

Distributed stream-based applications manage large quantities of data and exhibit unique production and consumption patterns that set them apart from general-purpose applications. This dissertation examines possible ways of creating more efficient memory management schemes. Specifically, it looks at the memory reclamation problem. It takes advantage of special traits of streaming applications to extend the definition of the garbage collection problem for those applications and include not only data items that are not reachable but also items that have no effect on the final outcome of the application. Streaming applications typically fully process only a portion of the data, and resources directed towards the remaining data items (i.e., those that dont affect the final outcome) can be viewed as wasted resources that should be minimized. Two complementary approaches are suggested: 1. Garbage Identification 2. Adaptive Resource Utilization Garbage Identification is concerned with an analysis of dynamic data dependencies to infer those items that the application is no longer going to access. Several garbage identification algorithms are examined. Each one of the algorithms uses a set of application properties (possibly distinct from one another) to reduce the memory consumption of the application. The performance of these garbage identification algorithms is compared to the performance of an ideal garbage collector, using a novel logging/post-mortem analyzer. The results indicate that the algorithms that achieve a low memory footprint (close to that of an ideal garbage collector) perform their garbage identification decisions locally; however, they base these decisions on best-effort global information obtained from other components of the distributed application. The Adaptive Resource Utilization (ARU) algorithm analyzes the dynamic relationships between the production and consumption of data items. It uses this information to infer the capacity of the system to process data items and adjusts data generation accordingly. The ARU algorithm makes local capacity decisions based on best-effort global information. This algorithm is found to be as effective as the most successful garbage identification algorithm in reducing the memory footprint of stream-based applications, thus confirming the observation that using best-effort global information to perform local decisions is fundamental in reducing memory consumption for stream-based applications.Ph.D.Committee Chair: Ramachandran, Umakishore; Committee Member: Ahamad, Mustaque; Committee Member: Fujimoto, Richard M.; Committee Member: Knobe, Kathleen; Committee Member: Prvulovic, Milo

Approved by: MEMORY OPTIMIZATIONS FOR DISTRIBUTED STREAM-BASED APPLICATIONS

To my parents, Edna and Haim. To my sister, Iris, and my brother, Igi. iii ACKNOWLEDGEMENTS Although this dissertation bears the name of a single author, it would not have been possible without the aid of many people. I would like to take this opportunity to express my gratitude to those individuals that helped me complete this work. First and foremost I would like to thank my advisor, Prof. Umakishore Ramachandran, for the optimism, enthusiasm, guidance, and the help he provided me and for the patience he showed through the many (too many...) years I spent at Georgia Tech as a graduate student. Without Prof. Ramachandran’s continuous support it would have been impossible to arrive at this point. I would also like to express my deep appreciation to Vasanthi Ra-machandran for the constant encouragement, and the many gatherings with all the delicious food. Yes, I will get married soon (I hope). I worked very closely with Dr. Kath Knobe. Her insight and advice were invaluabl