12,592 research outputs found
Bottom-k and Priority Sampling, Set Similarity and Subset Sums with Minimal Independence
We consider bottom-k sampling for a set X, picking a sample S_k(X) consisting
of the k elements that are smallest according to a given hash function h. With
this sample we can estimate the relative size f=|Y|/|X| of any subset Y as
|S_k(X) intersect Y|/k. A standard application is the estimation of the Jaccard
similarity f=|A intersect B|/|A union B| between sets A and B. Given the
bottom-k samples from A and B, we construct the bottom-k sample of their union
as S_k(A union B)=S_k(S_k(A) union S_k(B)), and then the similarity is
estimated as |S_k(A union B) intersect S_k(A) intersect S_k(B)|/k.
We show here that even if the hash function is only 2-independent, the
expected relative error is O(1/sqrt(fk)). For fk=Omega(1) this is within a
constant factor of the expected relative error with truly random hashing.
For comparison, consider the classic approach of kxmin-wise where we use k
hash independent functions h_1,...,h_k, storing the smallest element with each
hash function. For kxmin-wise there is an at least constant bias with constant
independence, and it is not reduced with larger k. Recently Feigenblat et al.
showed that bottom-k circumvents the bias if the hash function is 8-independent
and k is sufficiently large. We get down to 2-independence for any k. Our
result is based on a simply union bound, transferring generic concentration
bounds for the hashing scheme to the bottom-k sample, e.g., getting stronger
probability error bounds with higher independence.
For weighted sets, we consider priority sampling which adapts efficiently to
the concrete input weights, e.g., benefiting strongly from heavy-tailed input.
This time, the analysis is much more involved, but again we show that generic
concentration bounds can be applied.Comment: A short version appeared at STOC'1
An Optimal Algorithm for the Separating Common Tangents of two Polygons
We describe an algorithm for computing the separating common tangents of two
simple polygons using linear time and only constant workspace. A tangent of a
polygon is a line touching the polygon such that all of the polygon lies to the
same side of the line. A separating common tangent of two polygons is a tangent
of both polygons where the polygons are lying on different sides of the
tangent. Each polygon is given as a read-only array of its corners. If a
separating common tangent does not exist, the algorithm reports that.
Otherwise, two corners defining a separating common tangent are returned. The
algorithm is simple and implies an optimal algorithm for deciding if the convex
hulls of two polygons are disjoint or not. This was not known to be possible in
linear time and constant workspace prior to this paper.
An outer common tangent is a tangent of both polygons where the polygons are
on the same side of the tangent. In the case where the convex hulls of the
polygons are disjoint, we give an algorithm for computing the outer common
tangents in linear time using constant workspace.Comment: 12 pages, 6 figures. A preliminary version of this paper appeared at
SoCG 201
Classification of terminal simplicial reflexive d-polytopes with 3d-1 vertices
We classify terminal simplicial reflexive d-polytopes with 3d-1 vertices.
They turn out to be smooth Fano d-polytopes. When d is even there is 1 such
polytope up to isomorphism, while there are 2 when d is uneven.Comment: 13 page
Fast and Powerful Hashing using Tabulation
Randomized algorithms are often enjoyed for their simplicity, but the hash
functions employed to yield the desired probabilistic guarantees are often too
complicated to be practical. Here we survey recent results on how simple
hashing schemes based on tabulation provide unexpectedly strong guarantees.
Simple tabulation hashing dates back to Zobrist [1970]. Keys are viewed as
consisting of characters and we have precomputed character tables
mapping characters to random hash values. A key
is hashed to . This schemes is
very fast with character tables in cache. While simple tabulation is not even
4-independent, it does provide many of the guarantees that are normally
obtained via higher independence, e.g., linear probing and Cuckoo hashing.
Next we consider twisted tabulation where one input character is "twisted" in
a simple way. The resulting hash function has powerful distributional
properties: Chernoff-Hoeffding type tail bounds and a very small bias for
min-wise hashing. This also yields an extremely fast pseudo-random number
generator that is provably good for many classic randomized algorithms and
data-structures.
Finally, we consider double tabulation where we compose two simple tabulation
functions, applying one to the output of the other, and show that this yields
very high independence in the classic framework of Carter and Wegman [1977]. In
fact, w.h.p., for a given set of size proportional to that of the space
consumed, double tabulation gives fully-random hashing. We also mention some
more elaborate tabulation schemes getting near-optimal independence for given
time and space.
While these tabulation schemes are all easy to implement and use, their
analysis is not
Simple Tabulation, Fast Expanders, Double Tabulation, and High Independence
Simple tabulation dates back to Zobrist in 1970. Keys are viewed as c
characters from some alphabet A. We initialize c tables h_0, ..., h_{c-1}
mapping characters to random hash values. A key x=(x_0, ..., x_{c-1}) is hashed
to h_0[x_0] xor...xor h_{c-1}[x_{c-1}]. The scheme is extremely fast when the
character hash tables h_i are in cache. Simple tabulation hashing is not
4-independent, but we show that if we apply it twice, then we get high
independence. First we hash to intermediate keys that are 6 times longer than
the original keys, and then we hash the intermediate keys to the final hash
values.
The intermediate keys have d=6c characters from A. We can view the hash
function as a degree d bipartite graph with keys on one side, each with edges
to d output characters. We show that this graph has nice expansion properties,
and from that we get that with another level of simple tabulation on the
intermediate keys, the composition is a highly independent hash function. The
independence we get is |A|^{Omega(1/c)}.
Our space is O(c|A|) and the hash function is evaluated in O(c) time. Siegel
[FOCS'89, SICOMP'04] proved that with this space, if the hash function is
evaluated in o(c) time, then the independence can only be o(c), so our
evaluation time is best possible for Omega(c) independence---our independence
is much higher if c=|A|^{o(1)}.
Siegel used O(c)^c evaluation time to get the same independence with similar
space. Siegel's main focus was c=O(1), but we are exponentially faster when
c=omega(1).
Applying our scheme recursively, we can increase our independence to
|A|^{Omega(1)} with o(c^{log c}) evaluation time. Compared with Siegel's scheme
this is both faster and higher independence.
Our scheme is easy to implement, and it does provide realistic
implementations of 100-independent hashing for, say, 32 and 64-bit keys
Regional Differences in Food Consumption in Urban Mozambique: A Censored Demand System Approach
A nationwide household survey for Mozambique is used to estimate a large censored food demand system with 12 food groups for the sample of urban households. Using the translog indirect utility approach, the censored nature of the data is addressed by estimating a system of Tobit equations with a recently suggested quasi maximum likelihood estimator. Augmenting the system with demographic and geographical variables in a theoretically consistent way, I find that differences in elasticities between regions are significant. The results show that regional variation has to be taken into account when evaluating policies or employing CGE models. Further, the approach employed here can be applied to a number of developing countries with varying geographic conditions.censored demand system; elasticities; Mozambique; food demand; regional differences
- …