Continuous Monitoring of Distributed Data Streams over a Time-based Sliding Window

The past decade has witnessed many interesting algorithms for maintaining statistics over a data stream. This paper initiates a theoretical study of algorithms for monitoring distributed data streams over a time-based sliding window (which contains a variable number of items and possibly out-of-order items). The concern is how to minimize the communication between individual streams and the root, while allowing the root, at any time, to be able to report the global statistics of all streams within a given error bound. This paper presents communication-efficient algorithms for three classical statistics, namely, basic counting, frequent items and quantiles. The worst-case communication cost over a window is $O(\frac{k} {\epsilon} \log \frac{\epsilon N}{k})$ bits for basic counting and $O(\frac{k}{\epsilon} \log \frac{N}{k})$ words for the remainings, where $k$ is the number of distributed data streams, $N$ is the total number of items in the streams that arrive or expire in the window, and $\epsilon < 1$ is the desired error bound. Matching and nearly matching lower bounds are also obtained.Comment: 12 pages, to appear in the 27th International Symposium on Theoretical Aspects of Computer Science (STACS), 201

Blazes: Coordination Analysis for Distributed Programs

Distributed consistency is perhaps the most discussed topic in distributed systems today. Coordination protocols can ensure consistency, but in practice they cause undesirable performance unless used judiciously. Scalable distributed architectures avoid coordination whenever possible, but under-coordinated systems can exhibit behavioral anomalies under fault, which are often extremely difficult to debug. This raises significant challenges for distributed system architects and developers. In this paper we present Blazes, a cross-platform program analysis framework that (a) identifies program locations that require coordination to ensure consistent executions, and (b) automatically synthesizes application-specific coordination code that can significantly outperform general-purpose techniques. We present two case studies, one using annotated programs in the Twitter Storm system, and another using the Bloom declarative language.Comment: Updated to include additional materials from the original technical report: derivation rules, output stream label

Towards Ideal Semantics for Analyzing Stream Reasoning

The rise of smart applications has drawn interest to logical reasoning over data streams. Recently, different query languages and stream processing/reasoning engines were proposed in different communities. However, due to a lack of theoretical foundations, the expressivity and semantics of these diverse approaches are given only informally. Towards clear specifications and means for analytic study, a formal framework is needed to define their semantics in precise terms. To this end, we present a first step towards an ideal semantics that allows for exact descriptions and comparisons of stream reasoning systems.Comment: International Workshop on Reactive Concepts in Knowledge Representation (ReactKnow 2014), co-located with the 21st European Conference on Artificial Intelligence (ECAI 2014). Proceedings of the International Workshop on Reactive Concepts in Knowledge Representation (ReactKnow 2014), pages 17-22, technical report, ISSN 1430-3701, Leipzig University, 2014. http://nbn-resolving.de/urn:nbn:de:bsz:15-qucosa-150562 2014,

Some aspects of queueing and storage processes : a thesis in partial fulfilment of the requirements for the degree of Master of Science in Statistics at Massey University

In this study the nature of systems consisting of a single queue are first considered. Attention is then drawn to an analogy between such systems and storage systems. A development of the single queue viz queues with feedback is considered after first considering feedback processes in general. The behaviour of queues, some with feedback loops, combined into networks is then considered. Finally, the application of such networks to the analysis of interconnected reservoir systems is considered and the conclusion drawn that such analytic methods complement the more recently developed mathematical programming methods by providing analytic solutions for sub systems behaviour and thus guiding the development of a system model

Justifying Social Discounting: The Rank-Discounted Utilitarian Approach

The discounted utilitarian criterion for infinite horizon social choice has been criticized for treating generations unequally. We propose an extended rank-discounted utilitarian (ERDU) criterion instead. The criterion amounts to discounted utilitarianism on non-decreasing streams, but it treats all generations impartially: discounting becomes the mere expression of intergenerational inequality aversion. We show that more inequality averse ERDU societies have higher social discount rates when future generations are better-off. We apply the ERDU approach in two benchmark economic growth models and prove that it promotes sustainable policies that maximize discounted utilitarian welfare.intergenerational equity, social discounting, discounted utilitarianism, sustainability

Probabilistic Algorithms for Information and Technology Flows in the Networks

This thesis studies several probabilistic algorithms for information and technology flow in the networks. Information flow refers to the circulation of information in social or communication networks for the purpose of disseminating or aggregating knowledge. Technology flow refers to the process in the network in which nodes incrementally adopt a certain type of technological product such as networking protocols. In this thesis, we study the following problems. First, we consider the scenario where information flow acts as media to disseminate messages. The information flow here is considered as a mechanism of replicating a piece of information from one node to another in a network with a goal to “broadcast” the knowledge to everyone. Our studies focus on a broadcasting algorithm called the flooding algorithm. We give a tight characterization on the completion time of the flooding algorithm when we make natural stochastic assumptions on the evolution of the network. Second, we consider the problem that information flow acts as a device to aggregate statistics. We interpret information flow here as artifacts produced by algorithmic procedures that serve as statistical estimators for the networks. The goal is to maintain accurate estimators with minimal information flow overhead. We study these two problems: first, we consider the continual count tracking problem in a distributed environment where the input is an aggregate stream originating from $k$ distinct sites and the updates are allowed to be non-monotonic. We develop an optimal algorithm in communication cost that can continually track the count for a family of stochastic streams. Second, we study the effectiveness of using random walks to estimate the statistical properties of networks. Specifically, we give the first deviation bounds for random walks over finite state Markov chains based on mixing time properties of the chain. Finally, we study the problem where technology flow acts as a key to unlock innovative technology diffusion. Here, the technology flow shall be interpreted as a way to specify the circumstance, in which a node in the network will decide to adopt a new technology. Our studies focus on finding the most cost effective way to deploy networking protocols such as SecureBGP or IPv6 in the Internet. Our result is a near optimal strategy that leverages the patterns of technology flows to facilitate the new technology deployments.Engineering and Applied Science