On randomness in Hash functions

In the talk, we shall discuss quality measures for hash functions used in data structures and algorithms, and survey positive and negative results. (This talk is not about cryptographic hash functions.) For the analysis of algorithms involving hash functions, it is often convenient to assume the hash functions used behave fully randomly; in some cases there is no analysis known that avoids this assumption. In practice, one needs to get by with weaker hash functions that can be generated by randomized algorithms. A well-studied range of applications concern realizations of dynamic dictionaries (linear probing, chained hashing, dynamic perfect hashing, cuckoo hashing and its generalizations) or Bloom filters and their variants. A particularly successful and useful means of classification are Carter and Wegman's universal or k-wise independent classes, introduced in 1977. A natural and widely used approach to analyzing an algorithm involving hash functions is to show that it works if a sufficiently strong universal class of hash functions is used, and to substitute one of the known constructions of such classes. This invites research into the question of just how much independence in the hash functions is necessary for an algorithm to work. Some recent analyses that gave impossibility results constructed rather artificial classes that would not work; other results pointed out natural, widely used hash classes that would not work in a particular application. Only recently it was shown that under certain assumptions on some entropy present in the set of keys even 2-wise independent hash classes will lead to strong randomness properties in the hash values. The negative results show that these results may not be taken as justification for using weak hash classes indiscriminately, in particular for key sets with structure. When stronger independence properties are needed for a theoretical analysis, one may resort to classic constructions. Only in 2003 it was found out how full randomness can be simulated using only linear space overhead (which is optimal). The "split-and-share" approach can be used to justify the full randomness assumption in some situations in which full randomness is needed for the analysis to go through, like in many applications involving multiple hash functions (e.g., generalized versions of cuckoo hashing with multiple hash functions or larger bucket sizes, load balancing, Bloom filters and variants, or minimal perfect hash function constructions). For practice, efficiency considerations beyond constant factors are important. It is not hard to construct very efficient 2-wise independent classes. Using k-wise independent classes for constant k bigger than 3 has become feasible in practice only by new constructions involving tabulation. This goes together well with the quite new result that linear probing works with 5-independent hash functions. Recent developments suggest that the classification of hash function constructions by their degree of independence alone may not be adequate in some cases. Thus, one may want to analyze the behavior of specific hash classes in specific applications, circumventing the concept of k-wise independence. Several such results were recently achieved concerning hash functions that utilize tabulation. In particular if the analysis of the application involves using randomness properties in graphs and hypergraphs (generalized cuckoo hashing, also in the version with a "stash", or load balancing), a hash class combining k-wise independence with tabulation has turned out to be very powerful

Simulating Uniform Hashing in Constant Time and Optimal Space

Many algorithms and data structures employing hashing have been analyzed under the uniform hashing assumption, i.e., the assumption that hash functions behave like truly random functions. In this paper it is shown how to implement hash functions that can be evaluated on a RAM in constant time, and behave like truly random functions on any set of n inputs, with high probability. The space needed to represent a function is O(n) words, which is the best possible (and a polynomial improvement compared to previous fast hash functions). As a consequence, a broad class of hashing schemes can be implemented to meet, with high probability, the performance guarantees of their uniform hashing analysis

Making Digital Artifacts on the Web Verifiable and Reliable

The current Web has no general mechanisms to make digital artifacts --- such as datasets, code, texts, and images --- verifiable and permanent. For digital artifacts that are supposed to be immutable, there is moreover no commonly accepted method to enforce this immutability. These shortcomings have a serious negative impact on the ability to reproduce the results of processes that rely on Web resources, which in turn heavily impacts areas such as science where reproducibility is important. To solve this problem, we propose trusty URIs containing cryptographic hash values. We show how trusty URIs can be used for the verification of digital artifacts, in a manner that is independent of the serialization format in the case of structured data files such as nanopublications. We demonstrate how the contents of these files become immutable, including dependencies to external digital artifacts and thereby extending the range of verifiability to the entire reference tree. Our approach sticks to the core principles of the Web, namely openness and decentralized architecture, and is fully compatible with existing standards and protocols. Evaluation of our reference implementations shows that these design goals are indeed accomplished by our approach, and that it remains practical even for very large files.Comment: Extended version of conference paper: arXiv:1401.577

CONSTRUCTION OF EFFICIENT AUTHENTICATION SCHEMES USING TRAPDOOR HASH FUNCTIONS

In large-scale distributed systems, where adversarial attacks can have widespread impact, authentication provides protection from threats involving impersonation of entities and tampering of data. Practical solutions to authentication problems in distributed systems must meet specific constraints of the target system, and provide a reasonable balance between security and cost. The goal of this dissertation is to address the problem of building practical and efficient authentication mechanisms to secure distributed applications. This dissertation presents techniques to construct efficient digital signature schemes using trapdoor hash functions for various distributed applications. Trapdoor hash functions are collision-resistant hash functions associated with a secret trapdoor key that allows the key-holder to find collisions between hashes of different messages. The main contributions of this dissertation are as follows: 1. A common problem with conventional trapdoor hash functions is that revealing a collision producing message pair allows an entity to compute additional collisions without knowledge of the trapdoor key. To overcome this problem, we design an efficient trapdoor hash function that prevents all entities except the trapdoor key-holder from computing collisions regardless of whether collision producing message pairs are revealed by the key-holder. 2. We design a technique to construct efficient proxy signatures using trapdoor hash functions to authenticate and authorize agents acting on behalf of users in agent-based computing systems. Our technique provides agent authentication, assurance of agreement between delegator and agent, security without relying on secure communication channels and control over an agent’s capabilities. 3. We develop a trapdoor hash-based signature amortization technique for authenticating real-time, delay-sensitive streams. Our technique provides independent verifiability of blocks comprising a stream, minimizes sender-side and receiver-side delays, minimizes communication overhead, and avoids transmission of redundant information. 4. We demonstrate the practical efficacy of our trapdoor hash-based techniques for signature amortization and proxy signature construction by presenting discrete log-based instantiations of the generic techniques that are efficient to compute, and produce short signatures. Our detailed performance analyses demonstrate that the proposed schemes outperform existing schemes in computation cost and signature size. We also present proofs for security of the proposed discrete-log based instantiations against forgery attacks under the discrete-log assumption