Get the Most out of Your Sample: Optimal Unbiased Estimators using Partial Information

Random sampling is an essential tool in the processing and transmission of data. It is used to summarize data too large to store or manipulate and meet resource constraints on bandwidth or battery power. Estimators that are applied to the sample facilitate fast approximate processing of queries posed over the original data and the value of the sample hinges on the quality of these estimators. Our work targets data sets such as request and traffic logs and sensor measurements, where data is repeatedly collected over multiple {\em instances}: time periods, locations, or snapshots. We are interested in queries that span multiple instances, such as distinct counts and distance measures over selected records. These queries are used for applications ranging from planning to anomaly and change detection. Unbiased low-variance estimators are particularly effective as the relative error decreases with the number of selected record keys. The Horvitz-Thompson estimator, known to minimize variance for sampling with "all or nothing" outcomes (which reveals exacts value or no information on estimated quantity), is not optimal for multi-instance operations for which an outcome may provide partial information. We present a general principled methodology for the derivation of (Pareto) optimal unbiased estimators over sampled instances and aim to understand its potential. We demonstrate significant improvement in estimate accuracy of fundamental queries for common sampling schemes.Comment: This is a full version of a PODS 2011 pape

Effective Field Theory for Nuclear Physics

I summarize the motivation for the effective field theory approach to nuclear physics, and highlight some of its recent accomplishments. The results are compared with those computed in potential models.Comment: Talk delivered at Baryons '98, Bonn, Sept. 22, 1998. 15 pages, 9 figure

What you can do with Coordinated Samples

Sample coordination, where similar instances have similar samples, was proposed by statisticians four decades ago as a way to maximize overlap in repeated surveys. Coordinated sampling had been since used for summarizing massive data sets. The usefulness of a sampling scheme hinges on the scope and accuracy within which queries posed over the original data can be answered from the sample. We aim here to gain a fundamental understanding of the limits and potential of coordination. Our main result is a precise characterization, in terms of simple properties of the estimated function, of queries for which estimators with desirable properties exist. We consider unbiasedness, nonnegativity, finite variance, and bounded estimates. Since generally a single estimator can not be optimal (minimize variance simultaneously) for all data, we propose {\em variance competitiveness}, which means that the expectation of the square on any data is not too far from the minimum one possible for the data. Surprisingly perhaps, we show how to construct, for any function for which an unbiased nonnegative estimator exists, a variance competitive estimator.Comment: 4 figures, 21 pages, Extended Abstract appeared in RANDOM 201

Systematic Power Counting in Cutoff Effective Field Theories for Nucleon-Nucleon Interactions and the Equivalence With PDS

An analytic expression for the ${}^1S_0$ phase shifts in nucleon-nucleon scattering is derived in the context of the Schr\"odinger equation in configuration space with a short distance cutoff and with a consistent power counting scheme including pionic effects. The scheme treats the pion mass and the inverse scattering length over the intrinsic short distance scale as small parameters. Working at next-to-leading order in this scheme, we show that the expression obtained is identical to one obtained using the recently introduced PDS approach which is based on dimensional regularization with a novel subtraction scheme. This strongly supports the conjecture that the schemes are equivalent provided one works to the same order in the power counting.Comment: 6 pages; replaced version has corrected typos (We thank Mike Birse for pointing them out to u

Average Distance Queries through Weighted Samples in Graphs and Metric Spaces: High Scalability with Tight Statistical Guarantees

The average distance from a node to all other nodes in a graph, or from a query point in a metric space to a set of points, is a fundamental quantity in data analysis. The inverse of the average distance, known as the (classic) closeness centrality of a node, is a popular importance measure in the study of social networks. We develop novel structural insights on the sparsifiability of the distance relation via weighted sampling. Based on that, we present highly practical algorithms with strong statistical guarantees for fundamental problems. We show that the average distance (and hence the centrality) for all nodes in a graph can be estimated using $O(\epsilon^{-2})$ single-source distance computations. For a set $V$ of $n$ points in a metric space, we show that after preprocessing which uses $O(n)$ distance computations we can compute a weighted sample $S\subset V$ of size $O(\epsilon^{-2})$ such that the average distance from any query point $v$ to $V$ can be estimated from the distances from $v$ to $S$. Finally, we show that for a set of points $V$ in a metric space, we can estimate the average pairwise distance using $O(n+\epsilon^{-2})$ distance computations. The estimate is based on a weighted sample of $O(\epsilon^{-2})$ pairs of points, which is computed using $O(n)$ distance computations. Our estimates are unbiased with normalized mean square error (NRMSE) of at most $\epsilon$. Increasing the sample size by a $O(\log n)$ factor ensures that the probability that the relative error exceeds $\epsilon$ is polynomially small.Comment: 21 pages, will appear in the Proceedings of RANDOM 201

Sample Complexity Bounds for Influence Maximization

Influence maximization (IM) is the problem of finding for a given s ? 1 a set S of |S|=s nodes in a network with maximum influence. With stochastic diffusion models, the influence of a set S of seed nodes is defined as the expectation of its reachability over simulations, where each simulation specifies a deterministic reachability function. Two well-studied special cases are the Independent Cascade (IC) and the Linear Threshold (LT) models of Kempe, Kleinberg, and Tardos [Kempe et al., 2003]. The influence function in stochastic diffusion is unbiasedly estimated by averaging reachability values over i.i.d. simulations. We study the IM sample complexity: the number of simulations needed to determine a (1-?)-approximate maximizer with confidence 1-?. Our main result is a surprising upper bound of O(s ? ?^{-2} ln (n/?)) for a broad class of models that includes IC and LT models and their mixtures, where n is the number of nodes and ? is the number of diffusion steps. Generally ? ? n, so this significantly improves over the generic upper bound of O(s n ?^{-2} ln (n/?)). Our sample complexity bounds are derived from novel upper bounds on the variance of the reachability that allow for small relative error for influential sets and additive error when influence is small. Moreover, we provide a data-adaptive method that can detect and utilize fewer simulations on models where it suffices. Finally, we provide an efficient greedy design that computes an (1-1/e-?)-approximate maximizer from simulations and applies to any submodular stochastic diffusion model that satisfies the variance bounds

Echo spectroscopy and Atom Optics Billiards

We discuss a recently demonstrated type of microwave spectroscopy of trapped ultra-cold atoms known as "echo spectroscopy" [M.F. Andersen et. al., Phys. Rev. Lett., in press (2002)]. Echo spectroscopy can serve as an extremely sensitive experimental tool for investigating quantum dynamics of trapped atoms even when a large number of states are thermally populated. We show numerical results for the stability of eigenstates of an atom-optics billiard of the Bunimovich type, and discuss its behavior under different types of perturbations. Finally, we propose to use special geometrical constructions to make a dephasing free dipole trap

The Large N_c Baryon-Meson I_t = J_t Rule Holds for Three Flavors

It has long been known that nonstrange baryon-meson scattering in the 1/N_c expansion of QCD greatly simplifies when expressed in terms of t-channel exchanges: The leading-order amplitudes satisfy the selection rule I_t = J_t. We show that I_t = J_t, as well as Y_t = 0, also hold for the leading amplitudes when the baryon and/or meson contain strange quarks, and also characterize their 1/N_c corrections, thus opening a new front in the phenomenological study of baryon-meson scattering and baryon resonances.Comment: 12 pages, 0 figures, ReVTe