R3D3: A doubly opportunistic data structure for compressing and indexing massive data

Opportunistic data structures are used extensively in big data practice to break down the massive storage space requirements of processing large volumes of information. A data structure is called (singly) opportunistic if it takes advantage of the redundancy in the input in order to store it in informationtheoretically minimum space. Yet, efficient data processing requires a separate index alongside the data, whose size often substantially exceeds that of the compressed information. In this paper, we introduce doubly opportunistic data structures to not only attain best possible compression on the input data but also on the index. We present R3D3 that encodes a bitvector of length n and Shannon entropy H0 to nH0 bits and the accompanying index to nH0(1/2 + O(log C/C)) bits, thus attaining provably minimum space (up to small error terms) on both the data and the index, and supports a rich set of queries to arbitrary position in the compressed bitvector in O(C) time when C = o(log n). Our R3D3 prototype attains several times space reduction beyond known compression techniques on a wide range of synthetic and real data sets, while it supports operations on the compressed data at comparable speed

On a parity based group testing algorithm

In traditional Combinatorial Group Testing the problem is to identify up to d defective items from a set of n items on the basis of group tests. In this paper we describe a variant of the group testing problem above, which we call parity group testing. The problem is to identify up to d defective items from a set of n items as in the classical group test problem. The main difference is that we check the parity of the defective items in a subset. The test can be applied to an arbitrary subset of the n items with two possible outcomes. The test is positive if the number of defective items in the subset is odd, otherwise it is negative. In this paper we extend Hirschberg et al.’s method to the parity group testing scenario

On Signaling-Free Failure Dependent Restoration in All-Optical Mesh Networks

Failure dependent protection (FDP) is known to achieve optimal capacity efficiency among all types of protection, at the expense of longer recovery time and more complicated signaling overhead. This particularly hinders the usage of FDP in all-optical mesh networks. As a remedy, the paper investigates a new restoration framework that enables all-optical fault management and device configuration via state-of-the-art failure localization techniques, such that the FDP restoration process. It can be implemented without relying on any control plane signaling. With the proposed restoration framework, a novel spare capacity allocation problem is defined, and is further analyzed on circulant topologies for any single link failure, aiming to gain a solid understanding of the problem. By allowing reuse of monitoring resources for restoration capacity, we are particularly interested in the monitoring resource hidden property where less or even no monitoring resources are consumed as more working traffic is in place. To deal with general topologies, we introduce a novel heuristic approach to the proposed spare capacity allocation problem, which comprises a generic FDP survivable routing scheme followed by a novel monitoring resource allocation method. Extensive simulation is conducted to examine the proposed scheme and verify the proposed restoration framework

Neighborhood Failure Localization in All-Optical Networks via Monitoring Trails

Shared protection, such as failure dependent protection (FDP), is well recognized for its outstanding capacity efficiency in all-optical mesh networks, at the expense of lengthy restoration time due to multi-hop signaling mechanisms for failure localization, notification, and device configuration. This paper investigates a novel monitoring trail (m-trail) scenario, called Global Neighborhood Failure Localization (G-NFL), that aims to enable any shared protection scheme, including FDP, for achieving all-optical and ultra-fast failure restoration. We firstly define neighborhood of a node, which is a set of links whose failure states should be known to the node in restoration of the corresponding working lightpaths (W-LPs). By assuming every node can obtain the on-off status of traversing m-trails and W-LPs via lambda monitoring, the proposed G-NFL problem routes a set of m-trails such that each node can localize any failure in its neighborhood. Bound analysis is performed on the minimum bandwidth required for m-trails under the proposed G-NFL problem. Then a simple yet efficient heuristic approach is presented. Extensive simulation is conducted to verify the proposed G-NFL scenario under a number of different definitions of nodal neighborhood which concern the extent of dependency between the monitoring plane and data plane. The effect of reusing the spare capacity by FDP for supporting m-trails is examined. We conclude that the proposed G-NFL scenario enables a general shared protection scheme, toward signaling-free and ultra-fast failure restoration like p-Cycle, while achieving optimal capacity efficiency as FDP

Resilient flow decomposition of unicast connections with network coding

In this paper we close the gap between end-to-end diversity coding and intra-session network coding for unicast connections resilient against single link failures. In particular, we show that coding operations are sufficient to perform at the source and receiver if the user data can be split into at most two parts over the filed GF(2). Our proof is purely combinatorial and based on standard graph and network flow techniques. It is a linear time construction that defines the route of subflows A, B and A+B between the source and destination nodes. The proposed resilient flow decomposition method generalizes the 1+1 protection and the end-to-end diversity coding approaches while keeping both of their benefits. It provides a simple yet resource efficient protection method feasible in 2-connected backbone topologies. Since the core switches do not need to be modified, this result can bring benefits to current transport networks.Comment: submitted to IEEE International Symposium on Information Theory (ISIT) 201