New results in dimension reduction and model selection

Dimension reduction is a vital tool in many areas of applied statistics in which the dimensionality of the predictors can be large. In such cases, many statistical methods will fail or yield unsatisfactory results. However, many data sets of high dimensionality actually contain a much simpler, low-dimensional structure. Classical methods such as principal components analysis are able to detect linear structures very effectively, but fail in the presence of nonlinear structures. In the first part of this thesis, we investigate the asymptotic behavior of two nonlinear dimensionality reduction algorithms, LTSA and HLLE. In particular, we show that both algorithms, under suitable conditions, asymptotically recover the true generating coordinates up to an isometry. We also discuss the relative merits of the two algorithms, and the effects of the underlying probability distributions of the coordinates on their performance. Model selection is a fundamental problem in nearly all areas of applied statistics. In particular, a balance must be achieved between good in-sample performance and out-of-sample prediction. It is typically very easy to achieve good fit in the sample data, but empirically we often find that such models will generalize poorly. In the second part of the thesis, we propose a new procedure for the model selection problem which generalizes traditional methods. Our algorithm allows the combination of existing model selection criteria via a ranking procedure, leading to the creation of new criteria which are able to combine measures of in-sample fit and out-of-sample prediction performance into a single value. We then propose an algorithm which provably finds the optimal combination with a specified probability. We demonstrate through simulations that these new combined criteria can be substantially more powerful than any individual criterion.Ph.D.Committee Chair: Huo, Xiaoming; Committee Member: Serban, Nicoleta; Committee Member: Shapiro, Alexander; Committee Member: Yuan, Ming; Committee Member: Zha, Hongyua

A Reduced Order Modeling Methodology for the Multidisciplinary Design Analysis of Hypersonic Aerial Systems

Recent years have seen a significant increase in the demand for an advance and diverse fleet of hypersonic aerial systems. As computational power has increased, high-fidelity physics-based numerical analyses have emerged as feasible alternatives to physical experimentation, especially during early design phases. Due to the complexity of the underlying physics that govern hypersonic aerodynamics, these numerical tools can be very costly and not practical for systems engineering tasks that require many queries. To overcome these challenges, Reduced Order Models (ROMs) have been implemented that are capable of replacing expensive numerical analyses with inexpensive field surrogate models that can accurately predict aerodynamic flow features. This dissertation puts forth a methodology for the development of accurate, efficient, data-driven ROMs capable of predicting complex off-body hypersonic flow features. This methodology uses both linear and nonlinear Dimensionality Reduction (DR) to reduce high-dimensional aerodynamic field data into low-dimensional representations that faithfully represent the original data set. After this reduction, state-of-the-art surrogate modeling techniques are used to map parametric design inputs into this low-dimensional space to enable predictions. Manifold Alignment (MA), has also been implemented to enable models to leverage data from multiple fidelity sources. The performance of this method is evaluated experimentally using a number of test problems that exhibit a range of size and feature complexity. It is observed in many of these experiments that nonlinear ROMs outperform equivalent linear ROMs when the underlying fields exhibit complex discontinuous behavior. Furthermore, nonlinear ROMs consistently reduce data to lower dimensional forms than equivalent linear models, which results in nonlinear ROMs having lower evaluation costs and being more resilient to the “curse of dimensionality” then their linear counterparts. Similar trends are observed with multi-fidelity ROMs. When implemented into a coupled analysis, ROMs trained using the proposed methodology are able to achieve superior performance to state-of-the-at scalar models when predicting integrated force coefficients. Moreover, the proposed ROMs offer the novel capability of providing parametric flow-field data within a coupled analysis, which enables more sophisticated assessments of system-level performance, objectives, and constraints.Ph.D

Manifold Elastic Net: A Unified Framework for Sparse Dimension Reduction

It is difficult to find the optimal sparse solution of a manifold learning based dimensionality reduction algorithm. The lasso or the elastic net penalized manifold learning based dimensionality reduction is not directly a lasso penalized least square problem and thus the least angle regression (LARS) (Efron et al. \cite{LARS}), one of the most popular algorithms in sparse learning, cannot be applied. Therefore, most current approaches take indirect ways or have strict settings, which can be inconvenient for applications. In this paper, we proposed the manifold elastic net or MEN for short. MEN incorporates the merits of both the manifold learning based dimensionality reduction and the sparse learning based dimensionality reduction. By using a series of equivalent transformations, we show MEN is equivalent to the lasso penalized least square problem and thus LARS is adopted to obtain the optimal sparse solution of MEN. In particular, MEN has the following advantages for subsequent classification: 1) the local geometry of samples is well preserved for low dimensional data representation, 2) both the margin maximization and the classification error minimization are considered for sparse projection calculation, 3) the projection matrix of MEN improves the parsimony in computation, 4) the elastic net penalty reduces the over-fitting problem, and 5) the projection matrix of MEN can be interpreted psychologically and physiologically. Experimental evidence on face recognition over various popular datasets suggests that MEN is superior to top level dimensionality reduction algorithms.Comment: 33 pages, 12 figure

Recent Advances of Manifold Regularization

Semi-supervised learning (SSL) that can make use of a small number of labeled data with a large number of unlabeled data to produce significant improvement in learning performance has been received considerable attention. Manifold regularization is one of the most popular works that exploits the geometry of the probability distribution that generates the data and incorporates them as regularization terms. There are many representative works of manifold regularization including Laplacian regularization (LapR), Hessian regularization (HesR) and p-Laplacian regularization (pLapR). Based on the manifold regularization framework, many extensions and applications have been reported. In the chapter, we review the LapR and HesR, and we introduce an approximation algorithm of graph p-Laplacian. We study several extensions of this framework for pairwise constraint, p-Laplacian learning, hypergraph learning, etc