Fast evaluation of union-intersection expressions

We show how to represent sets in a linear space data structure such that expressions involving unions and intersections of sets can be computed in a worst-case efficient way. This problem has applications in e.g. information retrieval and database systems. We mainly consider the RAM model of computation, and sets of machine words, but also state our results in the I/O model. On a RAM with word size $w$, a special case of our result is that the intersection of $m$ (preprocessed) sets, containing $n$ elements in total, can be computed in expected time $O(n (\log w)^2 / w + km)$, where $k$ is the number of elements in the intersection. If the first of the two terms dominates, this is a factor $w^{1-o(1)}$ faster than the standard solution of merging sorted lists. We show a cell probe lower bound of time $\Omega(n/(w m \log m)+ (1-\tfrac{\log k}{w}) k)$, meaning that our upper bound is nearly optimal for small $m$. Our algorithm uses a novel combination of approximate set representations and word-level parallelism

Design of a multiple bloom filter for distributed navigation routing

Unmanned navigation of vehicles and mobile robots can be greatly simplified by providing environmental intelligence with dispersed wireless sensors. The wireless sensors can work as active landmarks for vehicle localization and routing. However, wireless sensors are often resource scarce and require a resource-saving design. In this paper, a multiple Bloom-filter scheme is proposed to compress a global routing table for a wireless sensor. It is used as a lookup table for routing a vehicle to any destination but requires significantly less memory space and search effort. An error-expectation-based design for a multiple Bloom filter is proposed as an improvement to the conventional false-positive-rate-based design. The new design is shown to provide an equal relative error expectation for all branched paths, which ensures a better network load balance and uses less memory space. The scheme is implemented in a project for wheelchair navigation using wireless camera motes. © 2013 IEEE

The Case for Learned Index Structures

Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not. In this exploratory research paper, we start from this premise and posit that all existing index structures can be replaced with other types of models, including deep-learning models, which we term learned indexes. The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or existence of records. We theoretically analyze under which conditions learned indexes outperform traditional index structures and describe the main challenges in designing learned index structures. Our initial results show, that by using neural nets we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets. More importantly though, we believe that the idea of replacing core components of a data management system through learned models has far reaching implications for future systems designs and that this work just provides a glimpse of what might be possible

Dynamic Virtual Page-based Flash Translation Layer with Novel Hot Data Identification and Adaptive Parallelism Management

Solid-state disks (SSDs) tend to replace traditional motor-driven hard disks in high-end storage devices in past few decades. However, various inherent features, such as out-of-place update [resorting to garbage collection (GC)] and limited endurance (resorting to wear leveling), need to be reduced to a large extent before that day comes. Both the GC and wear leveling fundamentally depend on hot data identification (HDI). In this paper, we propose a hot data-aware flash translation layer architecture based on a dynamic virtual page (DVPFTL) so as to improve the performance and lifetime of NAND flash devices. First, we develop a generalized dual layer HDI (DL-HDI) framework, which is composed of a cold data pre-classifier and a hot data post-identifier. Those can efficiently follow the frequency and recency of information access. Then, we design an adaptive parallelism manager (APM) to assign the clustered data chunks to distinct resident blocks in the SSD so as to prolong its endurance. Finally, the experimental results from our realized SSD prototype indicate that the DVPFTL scheme has reliably improved the parallelizability and endurance of NAND flash devices with improved GC-costs, compared with related works.Peer reviewe

A privacy-preserving fuzzy interest matching protocol for friends finding in social networks

Nowadays, it is very popular to make friends, share photographs, and exchange news throughout social networks. Social networks widely expand the area of people’s social connections and make communication much smoother than ever before. In a social network, there are many social groups established based on common interests among persons, such as learning group, family group, and reading group. People often describe their profiles when registering as a user in a social network. Then social networks can organize these users into groups of friends according to their profiles. However, an important issue must be considered, namely many users’ sensitive profiles could have been leaked out during this process. Therefore, it is reasonable to design a privacy-preserving friends-finding protocol in social network. Toward this goal, we design a fuzzy interest matching protocol based on private set intersection. Concretely, two candidate users can first organize their profiles into sets, then use Bloom filters to generate new data structures, and finally find the intersection sets to decide whether being friends or not in the social network. The protocol is shown to be secure in the malicious model and can be useful for practical purposes.Peer ReviewedPostprint (author's final draft

Quality Assessment of Linked Datasets using Probabilistic Approximation

With the increasing application of Linked Open Data, assessing the quality of datasets by computing quality metrics becomes an issue of crucial importance. For large and evolving datasets, an exact, deterministic computation of the quality metrics is too time consuming or expensive. We employ probabilistic techniques such as Reservoir Sampling, Bloom Filters and Clustering Coefficient estimation for implementing a broad set of data quality metrics in an approximate but sufficiently accurate way. Our implementation is integrated in the comprehensive data quality assessment framework Luzzu. We evaluated its performance and accuracy on Linked Open Datasets of broad relevance.Comment: 15 pages, 2 figures, To appear in ESWC 2015 proceeding