Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing

An important estimation problem that is closely related to large-scale multiple testing is that of estimating the null density and the proportion of nonnull effects. A few estimators have been introduced in the literature; however, several important problems, including the evaluation of the minimax rate of convergence and the construction of rate-optimal estimators, remain open. In this paper, we consider optimal estimation of the null density and the proportion of nonnull effects. Both minimax lower and upper bounds are derived. The lower bound is established by a two-point testing argument, where at the core is the novel construction of two least favorable marginal densities $f_1$ and $f_2$. The density $f_1$ is heavy tailed both in the spatial and frequency domains and $f_2$ is a perturbation of $f_1$ such that the characteristic functions associated with $f_1$ and $f_2$ match each other in low frequencies. The minimax upper bound is obtained by constructing estimators which rely on the empirical characteristic function and Fourier analysis. The estimator is shown to be minimax rate optimal. Compared to existing methods in the literature, the proposed procedure not only provides more precise estimates of the null density and the proportion of the nonnull effects, but also yields more accurate results when used inside some multiple testing procedures which aim at controlling the False Discovery Rate (FDR). The procedure is easy to implement and numerical results are given.Comment: Published in at http://dx.doi.org/10.1214/09-AOS696 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Estimating the Null and the Proportion of non-Null effects in Large-scale Multiple Comparisons

An important issue raised by Efron in the context of large-scale multiple comparisons is that in many applications the usual assumption that the null distribution is known is incorrect, and seemingly negligible differences in the null may result in large differences in subsequent studies. This suggests that a careful study of estimation of the null is indispensable. In this paper, we consider the problem of estimating a null normal distribution, and a closely related problem, estimation of the proportion of non-null effects. We develop an approach based on the empirical characteristic function and Fourier analysis. The estimators are shown to be uniformly consistent over a wide class of parameters. Numerical performance of the estimators is investigated using both simulated and real data. In particular, we apply our procedure to the analysis of breast cancer and HIV microarray data sets. The estimators perform favorably in comparison to existing methods.Comment: 42 pages, 6 figure

Holant Problems for Regular Graphs with Complex Edge Functions

We prove a complexity dichotomy theorem for Holant Problems on 3-regular graphs with an arbitrary complex-valued edge function. Three new techniques are introduced: (1) higher dimensional iterations in interpolation; (2) Eigenvalue Shifted Pairs, which allow us to prove that a pair of combinatorial gadgets in combination succeed in proving #P-hardness; and (3) algebraic symmetrization, which significantly lowers the symbolic complexity of the proof for computational complexity. With holographic reductions the classification theorem also applies to problems beyond the basic model.Comment: 19 pages, 4 figures, added proofs for full versio

Finite 33-connected homogeneous graphs

A finite graph \G is said to be {\em $(G,3)$-$($connected$)$ homogeneous} if every isomorphism between any two isomorphic (connected) subgraphs of order at most $3$ extends to an automorphism $g\in G$ of the graph, where $G$ is a group of automorphisms of the graph. In 1985, Cameron and Macpherson determined all finite $(G, 3)$-homogeneous graphs. In this paper, we develop a method for characterising $(G,3)$-connected homogeneous graphs. It is shown that for a finite $(G,3)$-connected homogeneous graph \G=(V, E), either G_v^{\G(v)} is $2$--transitive or G_v^{\G(v)} is of rank $3$ and \G has girth $3$, and that the class of finite $(G,3)$-connected homogeneous graphs is closed under taking normal quotients. This leads us to study graphs where $G$ is quasiprimitive on $V$. We determine the possible quasiprimitive types for $G$ in this case and give new constructions of examples for some possible types

Automorphisms of surfaces of general type with q>=2 acting trivially in cohomology

A compact complex manifold X is said to be rationally cohomologically rigidified if its automorphism group Aut(X) acts faithfully on the cohomology ring H*(X,Q). In this note, we prove that, surfaces of general type with irregularity q>2 are rationally cohomologically rigidified, and so are minimal surfaces S with q=2 unless K^2=8X. This answers a question of Fabrizio Catanese in part. As examples we give a complete classification of surfaces isogenous to a product with q=2 that are not rationally cohomologically rigidified. These surfaces turn out however to be rigidified.Comment: 18 pages; a remark and a closely relevant reference are adde

New Planar P-time Computable Six-Vertex Models and a Complete Complexity Classification

We discover new P-time computable six-vertex models on planar graphs beyond Kasteleyn's algorithm for counting planar perfect matchings. We further prove that there are no more: Together, they exhaust all P-time computable six-vertex models on planar graphs, assuming #P is not P. This leads to the following exact complexity classification: For every parameter setting in ${\mathbb C}$ for the six-vertex model, the partition function is either (1) computable in P-time for every graph, or (2) #P-hard for general graphs but computable in P-time for planar graphs, or (3) #P-hard even for planar graphs. The classification has an explicit criterion. The new P-time cases in (2) provably cannot be subsumed by Kasteleyn's algorithm. They are obtained by a non-local connection to #CSP, defined in terms of a "loop space". This is the first substantive advance toward a planar Holant classification with not necessarily symmetric constraints. We introduce M\"obius transformation on ${\mathbb C}$ as a powerful new tool in hardness proofs for counting problems.Comment: 61 pages, 16 figures. An extended abstract appears in SODA 202