Regression on manifolds: Estimation of the exterior derivative

Collinearity and near-collinearity of predictors cause difficulties when doing regression. In these cases, variable selection becomes untenable because of mathematical issues concerning the existence and numerical stability of the regression coefficients, and interpretation of the coefficients is ambiguous because gradients are not defined. Using a differential geometric interpretation, in which the regression coefficients are interpreted as estimates of the exterior derivative of a function, we develop a new method to do regression in the presence of collinearities. Our regularization scheme can improve estimation error, and it can be easily modified to include lasso-type regularization. These estimators also have simple extensions to the "large $p$, small $n$" context.Comment: Published in at http://dx.doi.org/10.1214/10-AOS823 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

On the interpretation and identification of dynamic Takagi-Sugenofuzzy models

Dynamic Takagi-Sugeno fuzzy models are not always easy to interpret, in particular when they are identified from experimental data. It is shown that there exists a close relationship between dynamic Takagi-Sugeno fuzzy models and dynamic linearization when using affine local model structures, which suggests that a solution to the multiobjective identification problem exists. However, it is also shown that the affine local model structure is a highly sensitive parametrization when applied in transient operating regimes. Due to the multiobjective nature of the identification problem studied here, special considerations must be made during model structure selection, experiment design, and identification in order to meet both objectives. Some guidelines for experiment design are suggested and some robust nonlinear identification algorithms are studied. These include constrained and regularized identification and locally weighted identification. Their usefulness in the present context is illustrated by examples

Calibration and Rescaling Principles for Nonlinear Inverse Heat Conduction and Parameter Estimation Problems

This dissertation provides a systematic method for resolving nonlinear inverse heat conduction problems based on a calibration formulation and its accompanying principles. It is well-known that inverse heat conduction problems are ill-posed and hence subject to stability and uniqueness issues. Regularization methods are required to extract the best prediction based on a family of solutions. To date, most studies require sophisticated and combined numerical methods and regularization schemes for producing predictions. All thermophysical and geometrical properties must be provided in the simulations. The successful application of the numerical methods relies on the accuracy of the related system parameters as previously described. Due to the existence of uncertainties in the system parameters, these numerical methods possess bias of varying magnitudes. The calibration based approaches are proposed to minimize the systematic errors since system parameters are implicitly included in the mathematical formulation based on several calibration tests. To date, most calibration inverse studies have been based on the assumption of constant thermophysical properties. In contrast, this dissertation focuses on accounting for temperature-dependent thermophysical properties that produces a nonlinear heat equation. A novel rescaling principle is introduced for linearzing the system. This concept generates a mathematical framework similar to that of the linear formulation. Unlike the linear formulation, the present approach does require knowledge of thermophysical properties. However, all geometrical properties and sensor characterization are completely removed from the system. In this dissertation, a linear one-probe calibration method is first introduced as background. After that, the calibration method is generalized to the one-probe and two-probe, one-dimensional thermal system based on the assumption of temperature-dependent thermophysical properties. All previously proposed calibration equations are expressed in terms of a Volterra integral equation of the first kind for the unknown surface (net) heat flux and hence requires regularization owning to the ill-posed nature of first kind equations. A new strategy is proposed for determining the optimal regularization parameter that is independent of the applied regularization approach. As a final application, the described calibration principle is used for estimating unknown thermophysical properties above room temperature

Towards Efficient Maximum Likelihood Estimation of LPV-SS Models

How to efficiently identify multiple-input multiple-output (MIMO) linear parameter-varying (LPV) discrete-time state-space (SS) models with affine dependence on the scheduling variable still remains an open question, as identification methods proposed in the literature suffer heavily from the curse of dimensionality and/or depend on over-restrictive approximations of the measured signal behaviors. However, obtaining an SS model of the targeted system is crucial for many LPV control synthesis methods, as these synthesis tools are almost exclusively formulated for the aforementioned representation of the system dynamics. Therefore, in this paper, we tackle the problem by combining state-of-the-art LPV input-output (IO) identification methods with an LPV-IO to LPV-SS realization scheme and a maximum likelihood refinement step. The resulting modular LPV-SS identification approach achieves statical efficiency with a relatively low computational load. The method contains the following three steps: 1) estimation of the Markov coefficient sequence of the underlying system using correlation analysis or Bayesian impulse response estimation, then 2) LPV-SS realization of the estimated coefficients by using a basis reduced Ho-Kalman method, and 3) refinement of the LPV-SS model estimate from a maximum-likelihood point of view by a gradient-based or an expectation-maximization optimization methodology. The effectiveness of the full identification scheme is demonstrated by a Monte Carlo study where our proposed method is compared to existing schemes for identifying a MIMO LPV system

High-order regularized regression in Electrical Impedance Tomography

We present a novel approach for the inverse problem in electrical impedance tomography based on regularized quadratic regression. Our contribution introduces a new formulation for the forward model in the form of a nonlinear integral transform, that maps changes in the electrical properties of a domain to their respective variations in boundary data. Using perturbation theory the transform is approximated to yield a high-order misfit unction which is then used to derive a regularized inverse problem. In particular, we consider the nonlinear problem to second-order accuracy, hence our approximation method improves upon the local linearization of the forward mapping. The inverse problem is approached using Newton's iterative algorithm and results from simulated experiments are presented. With a moderate increase in computational complexity, the method yields superior results compared to those of regularized linear regression and can be implemented to address the nonlinear inverse problem

Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future

Regularization and Bayesian methods for system identification have been repopularized in the recent years, and proved to be competitive w.r.t. classical parametric approaches. In this paper we shall make an attempt to illustrate how the use of regularization in system identification has evolved over the years, starting from the early contributions both in the Automatic Control as well as Econometrics and Statistics literature. In particular we shall discuss some fundamental issues such as compound estimation problems and exchangeability which play and important role in regularization and Bayesian approaches, as also illustrated in early publications in Statistics. The historical and foundational issues will be given more emphasis (and space), at the expense of the more recent developments which are only briefly discussed. The main reason for such a choice is that, while the recent literature is readily available, and surveys have already been published on the subject, in the author's opinion a clear link with past work had not been completely clarified.Comment: Plenary Presentation at the IFAC SYSID 2015. Submitted to Annual Reviews in Contro

Low Complexity Regularization of Linear Inverse Problems

Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it. A now standard method for recovering the unknown signal is to solve a convex optimization problem that enforces some prior knowledge about its structure. This has proved efficient in many problems routinely encountered in imaging sciences, statistics and machine learning. This chapter delivers a review of recent advances in the field where the regularization prior promotes solutions conforming to some notion of simplicity/low-complexity. These priors encompass as popular examples sparsity and group sparsity (to capture the compressibility of natural signals and images), total variation and analysis sparsity (to promote piecewise regularity), and low-rank (as natural extension of sparsity to matrix-valued data). Our aim is to provide a unified treatment of all these regularizations under a single umbrella, namely the theory of partial smoothness. This framework is very general and accommodates all low-complexity regularizers just mentioned, as well as many others. Partial smoothness turns out to be the canonical way to encode low-dimensional models that can be linear spaces or more general smooth manifolds. This review is intended to serve as a one stop shop toward the understanding of the theoretical properties of the so-regularized solutions. It covers a large spectrum including: (i) recovery guarantees and stability to noise, both in terms of $\ell^2$-stability and model (manifold) identification; (ii) sensitivity analysis to perturbations of the parameters involved (in particular the observations), with applications to unbiased risk estimation ; (iii) convergence properties of the forward-backward proximal splitting scheme, that is particularly well suited to solve the corresponding large-scale regularized optimization problem