Online Density Estimation of Nonstationary Sources Using Exponential Family of Distributions

We investigate online probability density estimation (or learning) of nonstationary (and memoryless) sources using exponential family of distributions. To this end, we introduce a truly sequential algorithm that achieves Hannan-consistent log-loss regret performance against true probability distribution without requiring any information about the observation sequence (e.g., the time horizon T and the drift of the underlying distribution C) to optimize its parameters. Our results are guaranteed to hold in an individual sequence manner. Our log-loss performance with respect to the true probability density has regret bounds of O((CT)1/2), where C is the total change (drift) in the natural parameters of the underlying distribution. To achieve this, we design a variety of probability density estimators with exponentially quantized learning rates and merge them with a mixture-of-experts notion. Hence, we achieve this square-root regret with computational complexity only logarithmic in the time horizon. Thus, our algorithm can be efficiently used in big data applications. Apart from the regret bounds, through synthetic and real-life experiments, we demonstrate substantial performance gains with respect to the state-of-the-art probability density estimation algorithms in the literature. IEE

Bayesian separation of spectral sources under non-negativity and full additivity constraints

This paper addresses the problem of separating spectral sources which are linearly mixed with unknown proportions. The main difficulty of the problem is to ensure the full additivity (sum-to-one) of the mixing coefficients and non-negativity of sources and mixing coefficients. A Bayesian estimation approach based on Gamma priors was recently proposed to handle the non-negativity constraints in a linear mixture model. However, incorporating the full additivity constraint requires further developments. This paper studies a new hierarchical Bayesian model appropriate to the non-negativity and sum-to-one constraints associated to the regressors and regression coefficients of linear mixtures. The estimation of the unknown parameters of this model is performed using samples generated using an appropriate Gibbs sampler. The performance of the proposed algorithm is evaluated through simulation results conducted on synthetic mixture models. The proposed approach is also applied to the processing of multicomponent chemical mixtures resulting from Raman spectroscopy.Comment: v4: minor grammatical changes; Signal Processing, 200

On dimension reduction in Gaussian filters

A priori dimension reduction is a widely adopted technique for reducing the computational complexity of stationary inverse problems. In this setting, the solution of an inverse problem is parameterized by a low-dimensional basis that is often obtained from the truncated Karhunen-Loeve expansion of the prior distribution. For high-dimensional inverse problems equipped with smoothing priors, this technique can lead to drastic reductions in parameter dimension and significant computational savings. In this paper, we extend the concept of a priori dimension reduction to non-stationary inverse problems, in which the goal is to sequentially infer the state of a dynamical system. Our approach proceeds in an offline-online fashion. We first identify a low-dimensional subspace in the state space before solving the inverse problem (the offline phase), using either the method of "snapshots" or regularized covariance estimation. Then this subspace is used to reduce the computational complexity of various filtering algorithms - including the Kalman filter, extended Kalman filter, and ensemble Kalman filter - within a novel subspace-constrained Bayesian prediction-and-update procedure (the online phase). We demonstrate the performance of our new dimension reduction approach on various numerical examples. In some test cases, our approach reduces the dimensionality of the original problem by orders of magnitude and yields up to two orders of magnitude in computational savings

A review of RFI mitigation techniques in microwave radiometry

Radio frequency interference (RFI) is a well-known problem in microwave radiometry (MWR). Any undesired signal overlapping the MWR protected frequency bands introduces a bias in the measurements, which can corrupt the retrieved geophysical parameters. This paper presents a literature review of RFI detection and mitigation techniques for microwave radiometry from space. The reviewed techniques are divided between real aperture and aperture synthesis. A discussion and assessment of the application of RFI mitigation techniques is presented for each type of radiometer.Peer ReviewedPostprint (published version

Knowledge-aided STAP in heterogeneous clutter using a hierarchical bayesian algorithm

This paper addresses the problem of estimating the covariance matrix of a primary vector from heterogeneous samples and some prior knowledge, under the framework of knowledge-aided space-time adaptive processing (KA-STAP). More precisely, a Gaussian scenario is considered where the covariance matrix of the secondary data may differ from the one of interest. Additionally, some knowledge on the primary data is supposed to be available and summarized into a prior matrix. Two KA-estimation schemes are presented in a Bayesian framework whereby the minimum mean square error (MMSE) estimates are derived. The first scheme is an extension of a previous work and takes into account the non-homogeneity via an original relation. {In search of simplicity and to reduce the computational load, a second estimation scheme, less complex, is proposed and omits the fact that the environment may be heterogeneous.} Along the estimation process, not only the covariance matrix is estimated but also some parameters representing the degree of \emph{a priori} and/or the degree of heterogeneity. Performance of the two approaches are then compared using STAP synthetic data. STAP filter shapes are analyzed and also compared with a colored loading technique

Functional Regression

Functional data analysis (FDA) involves the analysis of data whose ideal units of observation are functions defined on some continuous domain, and the observed data consist of a sample of functions taken from some population, sampled on a discrete grid. Ramsay and Silverman's 1997 textbook sparked the development of this field, which has accelerated in the past 10 years to become one of the fastest growing areas of statistics, fueled by the growing number of applications yielding this type of data. One unique characteristic of FDA is the need to combine information both across and within functions, which Ramsay and Silverman called replication and regularization, respectively. This article will focus on functional regression, the area of FDA that has received the most attention in applications and methodological development. First will be an introduction to basis functions, key building blocks for regularization in functional regression methods, followed by an overview of functional regression methods, split into three types: [1] functional predictor regression (scalar-on-function), [2] functional response regression (function-on-scalar) and [3] function-on-function regression. For each, the role of replication and regularization will be discussed and the methodological development described in a roughly chronological manner, at times deviating from the historical timeline to group together similar methods. The primary focus is on modeling and methodology, highlighting the modeling structures that have been developed and the various regularization approaches employed. At the end is a brief discussion describing potential areas of future development in this field

Dirichlet process mixture models for non-stationary data streams

In recent years, we have seen a handful of work on inference algorithms over non-stationary data streams. Given their flexibility, Bayesian non-parametric models are a good candidate for these scenarios. However, reliable streaming inference under the concept drift phenomenon is still an open problem for these models. In this work, we propose a variational inference algorithm for Dirichlet process mixture models. Our proposal deals with the concept drift by including an exponential forgetting over the prior global parameters. Our algorithm allows adapting the learned model to the concept drifts automatically. We perform experiments in both synthetic and real data, showing that the proposed model outperforms state-of-the-art variational methods in density estimation, clustering and parameter tracking

Wind Energy: Forecasting Challenges for its Operational Management

Renewable energy sources, especially wind energy, are to play a larger role in providing electricity to industrial and domestic consumers. This is already the case today for a number of European countries, closely followed by the US and high growth countries, for example, Brazil, India and China. There exist a number of technological, environmental and political challenges linked to supplementing existing electricity generation capacities with wind energy. Here, mathematicians and statisticians could make a substantial contribution at the interface of meteorology and decision-making, in connection with the generation of forecasts tailored to the various operational decision problems involved. Indeed, while wind energy may be seen as an environmentally friendly source of energy, full benefits from its usage can only be obtained if one is able to accommodate its variability and limited predictability. Based on a short presentation of its physical basics, the importance of considering wind power generation as a stochastic process is motivated. After describing representative operational decision-making problems for both market participants and system operators, it is underlined that forecasts should be issued in a probabilistic framework. Even though, eventually, the forecaster may only communicate single-valued predictions. The existing approaches to wind power forecasting are subsequently described, with focus on single-valued predictions, predictive marginal densities and space-time trajectories. Upcoming challenges related to generating improved and new types of forecasts, as well as their verification and value to forecast users, are finally discussed.Comment: Published in at http://dx.doi.org/10.1214/13-STS445 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Deep Statistical Models with Application to Environmental Data

When analyzing environmental data, constructing a realistic statistical model is important, not only to fully characterize the physical phenomena, but also to provide valid and useful predictions. Gaussian process models are amongst the most popular tools used for this purpose. However, many assumptions are usually made when using Gaussian processes, such as stationarity of the covariance function. There are several approaches to construct nonstationary spatial and spatio-temporal Gaussian processes, including the deformation approach. In the deformation approach, the geographical domain is warped into a new domain, on which the Gaussian process is modeled to be stationary. One of the main challenges with this approach is how to construct a deformation function that is complicated enough to adequately capture the nonstationarity in the process, but simple enough to facilitate statistical inference and prediction. In this thesis, by using ideas from deep learning, we construct deformation functions that are compositions of simple warping units. In particular, deformation functions that are composed of aligning functions and warping functions are introduced to model nonstationary and asymmetric multivariate spatial processes, while spatial and temporal warping functions are used to model nonstationary spatio-temporal processes. Similarly to the traditional deformation approach, familiar stationary models are used on the warped domain. It is shown that this new approach to model nonstationarity is computationally efficient, and that it can lead to predictions that are superior to those from stationary models. We show the utility of these models on both simulated data and real-world environmental data: ocean temperatures and surface-ice elevation. The developed warped nonstationary processes can also be used for emulation. We show that a warped, gradient-enhanced Gaussian process surrogate model can be embedded in algorithms such as importance sampling and delayed-acceptance Markov chain Monte Carlo. Our surrogate models can provide more accurate emulation than other traditional surrogate models, and can help speed up Bayesian inference in problems with exponential-family likelihoods with intractable normalizing constants, for example when analyzing satellite images using the Potts model