4 research outputs found
The xyz algorithm for fast interaction search in high-dimensional data
When performing regression on a data set with p variables, it is often of interest to go beyond using main linear effects and include interactions as products between individual variables. For small-scale problems, these interactions can be computed explicitly but this leads to a computational complexity of at least O(p2) if done naively. This cost can be prohibitive if p is very large. We introduce a new randomised algorithm that is able to discover interactions with high probability and under mild conditions has a runtime that is subquadratic in p. We show that strong interactions can be discovered in almost linear time, whilst finding weaker interactions requires O(pα) operations for 1 < α < 2 depending on their strength. The underlying idea is to transform interaction search into a closest pair problem which can be solved efficiently in subquadratic time. The algorithm is called xyz and is implemented in the language R. We demonstrate its efficiency for application to genome-wide association studies, where more than 1011 interactions can be screened in under 280 seconds with a single-core 1:2 GHz CPU.Isaac Newton Trust Early Career Support Schem
Right singular vector projection graphs: fast high dimensional covariance matrix estimation under latent confounding
In this work we consider the problem of estimating a high-dimensional covariance matrix , given observations of confounded data
with covariance , where is an unknown matrix of latent factor loadings. We propose a simple and scalable
estimator based on the projection on to the right singular vectors of the
observed data matrix, which we call RSVP. Our theoretical analysis of this
method reveals that in contrast to PCA-based approaches, RSVP is able to cope
well with settings where the smallest eigenvalue of is close
to the largest eigenvalue of , as well as settings where the
eigenvalues of are diverging fast. It is also able to handle
data that may have heavy tails and only requires that the data has an
elliptical distribution. RSVP does not require knowledge or estimation of the
number of latent factors , but only recovers up to an unknown
positive scale factor. We argue this suffices in many applications, for example
if an estimate of the correlation matrix is desired. We also show that by using
subsampling, we can further improve the performance of the method. We
demonstrate the favourable performance of RSVP through simulation experiments
and an analysis of gene expression datasets collated by the GTEX consortium.Supported by an EPSRC First Grant and the Alan Turing Institute under the EPSRC grant EP/N510129/1
Recommended from our members
The xyz algorithm for fast interaction search in high-dimensional data
When performing regression on a data set with p variables, it is often of interest to go beyond using main linear effects and include interactions as products between individual variables. For small-scale problems, these interactions can be computed explicitly but this leads to a computational complexity of at least O(p2) if done naively. This cost can be prohibitive if p is very large. We introduce a new randomised algorithm that is able to discover interactions with high probability and under mild conditions has a runtime that is subquadratic in p. We show that strong interactions can be discovered in almost linear time, whilst finding weaker interactions requires O(pα) operations for 1 < α < 2 depending on their strength. The underlying idea is to transform interaction search into a closest pair problem which can be solved efficiently in subquadratic time. The algorithm is called xyz and is implemented in the language R. We demonstrate its efficiency for application to genome-wide association studies, where more than 1011 interactions can be screened in under 280 seconds with a single-core 1:2 GHz CPU.Isaac Newton Trust Early Career Support Schem