Randomized Dimension Reduction on Massive Data

Scalability of statistical estimators is of increasing importance in modern applications and dimension reduction is often used to extract relevant information from data. A variety of popular dimension reduction approaches can be framed as symmetric generalized eigendecomposition problems. In this paper we outline how taking into account the low rank structure assumption implicit in these dimension reduction approaches provides both computational and statistical advantages. We adapt recent randomized low-rank approximation algorithms to provide efficient solutions to three dimension reduction methods: Principal Component Analysis (PCA), Sliced Inverse Regression (SIR), and Localized Sliced Inverse Regression (LSIR). A key observation in this paper is that randomization serves a dual role, improving both computational and statistical performance. This point is highlighted in our experiments on real and simulated data.Comment: 31 pages, 6 figures, Key Words:dimension reduction, generalized eigendecompositon, low-rank, supervised, inverse regression, random projections, randomized algorithms, Krylov subspace method

Chandra Study of the Cepheus B Star Forming Region: Stellar Populations and the Initial Mass Function

Cepheus B (Cep B) molecular cloud and a portion of the nearby Cep OB3b OB association, one of the most active regions of star formation within 1 kpc, has been observed with the ACIS detector on board the Chandra X-ray Observatory. We detect 431 X-ray sources, of which 89% are confidently identified as clustered pre-main sequence stars. Two main results are obtained. First, we provide the best census to date for the stellar population of the region. We identify many members of two rich stellar clusters: the lightly obscured Cep OB3b association, and the deeply embedded cluster in Cep B whose existence was previously traced only by a handful of radio sources and T Tauri stars. Second, we find a discrepancy between the X-ray Luminosity Functions of the Cep OB3b and the Orion Nebula Cluster. This may be due to different Initial Mass Functions of two regions (excess of ~0.3 solar mass stars), or different age distributions. Several other results are obtained. A diffuse X-ray component seen in the field is attributed to the integrated emission of unresolved low mass PMS stars. The X-ray emission from HD 217086 (O7n), the principle ionizing source of the region, follows the standard model involving many small shocks in an unmagnetized radiatively accelerated wind. The X-ray source #294 joins a number of similar superflare PMS stars where long magnetic structures may connect the protoplanetary disk to the stellar surface.Comment: 72 pages, 31 figures, 8 tables. Accepted for publication in Ap

A new algorithm for estimating the effective dimension-reduction subspace

The statistical problem of estimating the effective dimension-reduction (EDR) subspace in the multi-index regression model with deterministic design and additive noise is considered. A new procedure for recovering the directions of the EDR subspace is proposed. Under mild assumptions, $\sqrt n$-consistency of the proposed procedure is proved (up to a logarithmic factor) in the case when the structural dimension is not larger than 4. The empirical behavior of the algorithm is studied through numerical simulations

Variable-Dependent Partial Dimension Reduction

Sufficient dimension reduction reduces the dimension of a regression model without loss of information by replacing the original predictor with its lower-dimensional linear combinations. Partial (sufficient) dimension reduction arises when the predictors naturally fall into two sets X and W, and pursues a partial dimension reduction of X. Though partial dimension reduction is a very general problem, only very few research results are available when W is continuous. To the best of our knowledge, none can deal with the situation where the reduced lower-dimensional subspace of X varies with W. To address such issue, we in this paper propose a novel variable-dependent partial dimension reduction framework and adapt classical sufficient dimension reduction methods into this general paradigm. The asymptotic consistency of our method is investigated. Extensive numerical studies and real data analysis show that our variable-dependent partial dimension reduction method has superior performance compared to the existing methods

Variable-Dependent Partial Dimension Reduction

Sufficient dimension reduction reduces the dimension of a regression model without loss of information by replacing the original predictor with its lower-dimensional linear combinations. Partial (sufficient) dimension reduction arises when the predictors naturally fall into two sets X and W, and pursues a partial dimension reduction of X. Though partial dimension reduction is a very general problem, only very few research results are available when W is continuous. To the best of our knowledge, none can deal with the situation where the reduced lower-dimensional subspace of X varies with W. To address such issue, we in this paper propose a novel variable-dependent partial dimension reduction framework and adapt classical sufficient dimension reduction methods into this general paradigm. The asymptotic consistency of our method is investigated. Extensive numerical studies and real data analysis show that our variable-dependent partial dimension reduction method has superior performance compared to the existing methods

The retail revolution and food-price mismeasurement

If a product sells for $3 this week at the local supermarket and$2 next week, what is the "real" price? What if that same product has a different price at a different store? Thanks to scanner technology, food prices differ a lot these days because they can be changed quickly and easily. How do our official statistics take these price movements into account? Not too well, according to Leonard Nakamura. In this article, he describes the retail revolution of recent years and how it has led to mismeasurement of food pricesConsumer price indexes ; Food prices