On the Approximation of Toeplitz Operators for Nonparametric H∞\mathcal{H}_\infty-norm Estimation

Given a stable SISO LTI system $G$, we investigate the problem of estimating the $\mathcal{H}_\infty$-norm of $G$, denoted $||G||_\infty$, when $G$ is only accessible via noisy observations. Wahlberg et al. recently proposed a nonparametric algorithm based on the power method for estimating the top eigenvalue of a matrix. In particular, by applying a clever time-reversal trick, Wahlberg et al. implement the power method on the top left $n \times n$ corner $T_n$ of the Toeplitz (convolution) operator associated with $G$. In this paper, we prove sharp non-asymptotic bounds on the necessary length $n$ needed so that $||T_n||$ is an $\varepsilon$-additive approximation of $||G||_\infty$. Furthermore, in the process of demonstrating the sharpness of our bounds, we construct a simple family of finite impulse response (FIR) filters where the number of timesteps needed for the power method is arbitrarily worse than the number of timesteps needed for parametric FIR identification via least-squares to achieve the same $\varepsilon$-additive approximation

Estimation of trend in state-space models: Asymptotic mean square error and rate of convergence

The focus of this paper is on trend estimation for a general state-space model $Y_t=\mu_t+\varepsilon_t$, where the $d$th difference of the trend $\{\mu_t\}$ is assumed to be i.i.d., and the error sequence $\{\varepsilon_t\}$ is assumed to be a mean zero stationary process. A fairly precise asymptotic expression of the mean square error is derived for the estimator obtained by penalizing the $d$th order differences. Optimal rate of convergence is obtained, and it is shown to be "asymptotically equivalent" to a nonparametric estimator of a fixed trend model of smoothness of order $d-0.5$. The results of this paper show that the optimal rate of convergence for the stochastic and nonstochastic cases are different. A criterion for selecting the penalty parameter and degree of difference $d$ is given, along with an application to the global temperature data, which shows that a longer term history has nonlinearities that are important to take into consideration.Comment: Published in at http://dx.doi.org/10.1214/08-AOS675 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

On the Spectral Properties of Matrices Associated with Trend Filters

This paper is concerned with the spectral properties of matrices associated with linear filters for the estimation of the underlying trend of a time series. The interest lies in the fact that the eigenvectors can be interpreted as the latent components of any time series that the filter smooths through the corresponding eigenvalues. A difficulty arises because matrices associated with trend filters are finite approximations of Toeplitz operators and therefore very little is known about their eigenstructure, which also depends on the boundary conditions or, equivalently, on the filters for trend estimation at the end of the sample. Assuming reflecting boundary conditions, we derive a time series decomposition in terms of periodic latent components and corresponding smoothing eigenvalues. This decomposition depends on the local polynomial regression estimator chosen for the interior. Otherwise, the eigenvalue distribution is derived with an approximation measured by the size of the perturbation that different boundary conditions apport to the eigenvalues of matrices belonging to algebras with known spectral properties, such as the Circulant or the Cosine. The analytical form of the eigenvectors is then derived with an approximation that involves the extremes only. A further topic investigated in the paper concerns a strategy for a filter design in the time domain. Based on cut-off eigenvalues, new estimators are derived, that are less variable and almost equally biased as the original estimator, based on all the eigenvalues. Empirical examples illustrate the effectiveness of the method

Kernel Density Estimation with Linked Boundary Conditions

Kernel density estimation on a finite interval poses an outstanding challenge because of the well-recognized bias at the boundaries of the interval. Motivated by an application in cancer research, we consider a boundary constraint linking the values of the unknown target density function at the boundaries. We provide a kernel density estimator (KDE) that successfully incorporates this linked boundary condition, leading to a non-self-adjoint diffusion process and expansions in non-separable generalized eigenfunctions. The solution is rigorously analyzed through an integral representation given by the unified transform (or Fokas method). The new KDE possesses many desirable properties, such as consistency, asymptotically negligible bias at the boundaries, and an increased rate of approximation, as measured by the AMISE. We apply our method to the motivating example in biology and provide numerical experiments with synthetic data, including comparisons with state-of-the-art KDEs (which currently cannot handle linked boundary constraints). Results suggest that the new method is fast and accurate. Furthermore, we demonstrate how to build statistical estimators of the boundary conditions satisfied by the target function without apriori knowledge. Our analysis can also be extended to more general boundary conditions that may be encountered in applications

Nonparametric inference for fractional diffusion

A non-parametric diffusion model with an additive fractional Brownian motion noise is considered in this work. The drift is a non-parametric function that will be estimated by two methods. On one hand, we propose a locally linear estimator based on the local approximation of the drift by a linear function. On the other hand, a Nadaraya-Watson kernel type estimator is studied. In both cases, some non-asymptotic results are proposed by means of deviation probability bound. The consistency property of the estimators are obtained under a one sided dissipative Lipschitz condition on the drift that insures the ergodic property for the stochastic differential equation. Our estimators are first constructed under continuous observations. The drift function is then estimated with discrete time observations that is of the most importance for practical applications.Comment: Published in at http://dx.doi.org/10.3150/13-BEJ509 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page