Randomized Algorithms for Tracking Distributed Count, Frequencies, and Ranks

We show that randomization can lead to significant improvements for a few fundamental problems in distributed tracking. Our basis is the {\em count-tracking} problem, where there are $k$ players, each holding a counter $n_i$ that gets incremented over time, and the goal is to track an \eps-approximation of their sum $n=\sum_i n_i$ continuously at all times, using minimum communication. While the deterministic communication complexity of the problem is \Theta(k/\eps \cdot \log N), where $N$ is the final value of $n$ when the tracking finishes, we show that with randomization, the communication cost can be reduced to \Theta(\sqrt{k}/\eps \cdot \log N). Our algorithm is simple and uses only O(1) space at each player, while the lower bound holds even assuming each player has infinite computing power. Then, we extend our techniques to two related distributed tracking problems: {\em frequency-tracking} and {\em rank-tracking}, and obtain similar improvements over previous deterministic algorithms. Both problems are of central importance in large data monitoring and analysis, and have been extensively studied in the literature.Comment: 19 pages, 1 figur

Real-time adaptive algorithm for resource monitoring

In large scale systems, real-time monitoring of hardware and software resources is a crucial means for any management purpose. In architectures consisting of thousands of servers and hundreds of thousands of component resources, the amount of data monitored at high sampling frequencies represents an overhead on system performance and communication, while reducing sampling may cause quality degradation. We present a real-time adaptive algorithm for scalable data monitoring that is able to adapt the frequency of sampling and data updating for a twofold goal: to minimize computational and communication costs, to guarantee that reduced samples do not affect the accuracy of information about resources. Experiments carried out on heterogeneous data traces referring to synthetic and real environments confirm that the proposed adaptive approach reduces utilization and communication overhead without penalizing the quality of data with respect to existing monitoring algorithms

In-Network Placement of Security VNFs in Multi-Tenant Data Centers

Middleboxes are typically hardware-accelerated appliances such as firewalls, Proxies, WAN optimizers, and NATs that play an important role in service provisioning over today’s Data Centers. We focus on the placement of virtualised security services in multi-tenant Data Centers. Customised security services are provided to tenants as software VNF modules collocated with switches in the network. Our placement formulation satisfies the allocation constraints while maintaining efficient management of the infrastructure resources. We propose a Constraint Programming (CP) formulation and a CPLEX implementation. We also formulate a heuristic-based algorithm to solve larger instances of the placement problem. Extensive evaluation of the algorithms has been conducted, demonstrating that the VNF approach provides more than 50% reduction in resource consumption compared to other heuristic algorithms

Resource state monitoring of service transactions in cloud systems

In cloud systems, services constituting a transaction may spread over a large number of servers or clusters. Theoretically, these services could consume cloud resources unlimitedly. To avoid financial loss due to resource overuse, clouds have to monitor the state of resources consumed by the services – collect values of consumption, and evaluate whether the combined usage of resources has excessed a pre-defined upper bound or not. The distributed nature of the services introduces a challenge to the monitoring system on how to summarise distributed state information with low cost. We present our resource state monitoring solution to capture the challenge introduced by services hosted in clouds. Our solution tracks the resource consumed by each service constituting a transaction individually whilst ensures the whole transaction does not overuse the allocated resource. It improves availability by avoiding single points of failure, and achieves scalability by minimising message exchanges.We performed experimental analyses that indicate this work can provide an inexpensive resource monitoring solution for transactions in clouds

A Tree-based protocol for enforcing quotas in clouds

Services are increasingly being hosted on cloud nodes to enhance their performance and increase their availability. The virtually unlimited availability of cloud resources enables service owners to consume resources without quantitative restrictions, paying only for what they use. To avoid cost overruns, resource consumption must be controlled and capped when necessary. We present a distributed tree-based protocol for managing quotas in clouds that minimizes communication overheads and reduces the time required to determine whether a quota has been exhausted. Experimental evaluation shows that our protocol reduces communication costs by 42% relative to a distributed baseline solution and is up to 15 times faster

A Direct-Sum Theorem for Read-Once Branching Programs

We study a direct-sum question for read-once branching programs. If M(f) denotes the minimum average memory required to compute a function f(x_1,x_2, ..., x_n) how much memory is required to compute f on k independent inputs that arrive in parallel? We show that when the inputs are sampled independently from some domain X and M(f) = Omega(n), then computing the value of f on k streams requires average memory at least Omega(k * M(f)/n). Our results are obtained by defining new ways to measure the information complexity of read-once branching programs. We define two such measures: the transitional and cumulative information content. We prove that any read-once branching program with transitional information content I can be simulated using average memory O(n(I+1)). On the other hand, if every read-once branching program with cumulative information content I can be simulated with average memory O(I+1), then computing f on k inputs requires average memory at least Omega(k * (M(f)-1))