316,285 research outputs found
Bottom-k and Priority Sampling, Set Similarity and Subset Sums with Minimal Independence
We consider bottom-k sampling for a set X, picking a sample S_k(X) consisting
of the k elements that are smallest according to a given hash function h. With
this sample we can estimate the relative size f=|Y|/|X| of any subset Y as
|S_k(X) intersect Y|/k. A standard application is the estimation of the Jaccard
similarity f=|A intersect B|/|A union B| between sets A and B. Given the
bottom-k samples from A and B, we construct the bottom-k sample of their union
as S_k(A union B)=S_k(S_k(A) union S_k(B)), and then the similarity is
estimated as |S_k(A union B) intersect S_k(A) intersect S_k(B)|/k.
We show here that even if the hash function is only 2-independent, the
expected relative error is O(1/sqrt(fk)). For fk=Omega(1) this is within a
constant factor of the expected relative error with truly random hashing.
For comparison, consider the classic approach of kxmin-wise where we use k
hash independent functions h_1,...,h_k, storing the smallest element with each
hash function. For kxmin-wise there is an at least constant bias with constant
independence, and it is not reduced with larger k. Recently Feigenblat et al.
showed that bottom-k circumvents the bias if the hash function is 8-independent
and k is sufficiently large. We get down to 2-independence for any k. Our
result is based on a simply union bound, transferring generic concentration
bounds for the hashing scheme to the bottom-k sample, e.g., getting stronger
probability error bounds with higher independence.
For weighted sets, we consider priority sampling which adapts efficiently to
the concrete input weights, e.g., benefiting strongly from heavy-tailed input.
This time, the analysis is much more involved, but again we show that generic
concentration bounds can be applied.Comment: A short version appeared at STOC'1
AMS Without 4-Wise Independence on Product Domains
In their seminal work, Alon, Matias, and Szegedy introduced several sketching
techniques, including showing that 4-wise independence is sufficient to obtain
good approximations of the second frequency moment. In this work, we show that
their sketching technique can be extended to product domains by using
the product of 4-wise independent functions on . Our work extends that of
Indyk and McGregor, who showed the result for . Their primary motivation
was the problem of identifying correlations in data streams. In their model, a
stream of pairs arrive, giving a joint distribution ,
and they find approximation algorithms for how close the joint distribution is
to the product of the marginal distributions under various metrics, which
naturally corresponds to how close and are to being independent. By
using our technique, we obtain a new result for the problem of approximating
the distance between the joint distribution and the product of the
marginal distributions for -ary vectors, instead of just pairs, in a single
pass. Our analysis gives a randomized algorithm that is a
approximation (with probability ) that requires space logarithmic in
and and proportional to
Markov basis and Groebner basis of Segre-Veronese configuration for testing independence in group-wise selections
We consider testing independence in group-wise selections with some
restrictions on combinations of choices. We present models for frequency data
of selections for which it is easy to perform conditional tests by Markov chain
Monte Carlo (MCMC) methods. When the restrictions on the combinations can be
described in terms of a Segre-Veronese configuration, an explicit form of a
Gr\"obner basis consisting of moves of degree two is readily available for
performing a Markov chain. We illustrate our setting with the National Center
Test for university entrance examinations in Japan. We also apply our method to
testing independence hypotheses involving genotypes at more than one locus or
haplotypes of alleles on the same chromosome.Comment: 25 pages, 5 figure
Are a set of microarrays independent of each other?
Having observed an matrix whose rows are possibly correlated,
we wish to test the hypothesis that the columns are independent of each other.
Our motivation comes from microarray studies, where the rows of record
expression levels for different genes, often highly correlated, while the
columns represent individual microarrays, presumably obtained
independently. The presumption of independence underlies all the familiar
permutation, cross-validation and bootstrap methods for microarray analysis, so
it is important to know when independence fails. We develop nonparametric and
normal-theory testing methods. The row and column correlations of interact
with each other in a way that complicates test procedures, essentially by
reducing the accuracy of the relevant estimators.Comment: Published in at http://dx.doi.org/10.1214/09-AOAS236 the Annals of
Applied Statistics (http://www.imstat.org/aoas/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Bounded Independence Fools Degree-2 Threshold Functions
Let x be a random vector coming from any k-wise independent distribution over
{-1,1}^n. For an n-variate degree-2 polynomial p, we prove that E[sgn(p(x))] is
determined up to an additive epsilon for k = poly(1/epsilon). This answers an
open question of Diakonikolas et al. (FOCS 2009). Using standard constructions
of k-wise independent distributions, we obtain a broad class of explicit
generators that epsilon-fool the class of degree-2 threshold functions with
seed length log(n)*poly(1/epsilon).
Our approach is quite robust: it easily extends to yield that the
intersection of any constant number of degree-2 threshold functions is
epsilon-fooled by poly(1/epsilon)-wise independence. Our results also hold if
the entries of x are k-wise independent standard normals, implying for example
that bounded independence derandomizes the Goemans-Williamson hyperplane
rounding scheme.
To achieve our results, we introduce a technique we dub multivariate
FT-mollification, a generalization of the univariate form introduced by Kane et
al. (SODA 2010) in the context of streaming algorithms. Along the way we prove
a generalized hypercontractive inequality for quadratic forms which takes the
operator norm of the associated matrix into account. These techniques may be of
independent interest.Comment: Using v1 numbering: removed Lemma G.5 from the Appendix (it was
wrong). Net effect is that Theorem G.6 reduces the m^6 dependence of Theorem
8.1 to m^4, not m^
Pairwise Independent Random Walks Can Be Slightly Unbounded
A family of problems that have been studied in the context of various streaming algorithms are generalizations of the fact that the expected maximum distance of a 4-wise independent random walk on a line over n steps is O(sqrt{n}). For small values of k, there exist k-wise independent random walks that can be stored in much less space than storing n random bits, so these properties are often useful for lowering space bounds. In this paper, we show that for all of these examples, 4-wise independence is required by demonstrating a pairwise independent random walk with steps uniform in +/- 1 and expected maximum distance Omega(sqrt{n} lg n) from the origin. We also show that this bound is tight for the first and second moment, i.e. the expected maximum square distance of a 2-wise independent random walk is always O(n lg^2 n). Also, for any even k >= 4, we show that the kth moment of the maximum distance of any k-wise independent random walk is O(n^{k/2}). The previous two results generalize to random walks tracking insertion-only streams, and provide higher moment bounds than currently known. We also prove a generalization of Kolmogorov\u27s maximal inequality by showing an asymptotically equivalent statement that requires only 4-wise independent random variables with bounded second moments, which also generalizes a result of Blasiok
The universality of iterated hashing over variable-length strings
Iterated hash functions process strings recursively, one character at a time.
At each iteration, they compute a new hash value from the preceding hash value
and the next character. We prove that iterated hashing can be pairwise
independent, but never 3-wise independent. We show that it can be almost
universal over strings much longer than the number of hash values; we bound the
maximal string length given the collision probability
Recursive n-gram hashing is pairwise independent, at best
Many applications use sequences of n consecutive symbols (n-grams). Hashing
these n-grams can be a performance bottleneck. For more speed, recursive hash
families compute hash values by updating previous values. We prove that
recursive hash families cannot be more than pairwise independent. While hashing
by irreducible polynomials is pairwise independent, our implementations either
run in time O(n) or use an exponential amount of memory. As a more scalable
alternative, we make hashing by cyclic polynomials pairwise independent by
ignoring n-1 bits. Experimentally, we show that hashing by cyclic polynomials
is is twice as fast as hashing by irreducible polynomials. We also show that
randomized Karp-Rabin hash families are not pairwise independent.Comment: See software at https://github.com/lemire/rollinghashcp
- …