In this work we study Invertible Bloom Lookup Tables (IBLTs) with small
failure probabilities. IBLTs are highly versatile data structures that have
found applications in set reconciliation protocols, error-correcting codes, and
even the design of advanced cryptographic primitives. For storing n elements
and ensuring correctness with probability at least 1−δ, existing IBLT
constructions require Ω(n(log(n)log(1/δ)+1)) space and
they crucially rely on fully random hash functions.
We present new constructions of IBLTs that are simultaneously more space
efficient and require less randomness. For storing n elements with a failure
probability of at most δ, our data structure only requires
O(n+log(1/δ)loglog(1/δ)) space and
O(log(log(n)/δ))-wise independent hash functions.
As a key technical ingredient we show that hashing n keys with any k-wise
independent hash function h:U→[Cn] for some sufficiently large constant
C guarantees with probability 1−2−Ω(k) that at least n/2 keys
will have a unique hash value. Proving this is highly non-trivial as k
approaches n. We believe that the techniques used to prove this statement may
be of independent interest