H∞ optimality of the LMS algorithm

We show that the celebrated least-mean squares (LMS) adaptive algorithm is H∞ optimal. The LMS algorithm has been long regarded as an approximate solution to either a stochastic or a deterministic least-squares problem, and it essentially amounts to updating the weight vector estimates along the direction of the instantaneous gradient of a quadratic cost function. We show that the LMS can be regarded as the exact solution to a minimization problem in its own right. Namely, we establish that it is a minimax filter: it minimizes the maximum energy gain from the disturbances to the predicted errors, whereas the closely related so-called normalized LMS algorithm minimizes the maximum energy gain from the disturbances to the filtered errors. Moreover, since these algorithms are central H∞ filters, they minimize a certain exponential cost function and are thus also risk-sensitive optimal. We discuss the various implications of these results and show how they provide theoretical justification for the widely observed excellent robustness properties of the LMS filter

Estimation-based synthesis of H∞-optimal adaptive FIR filtersfor filtered-LMS problems

This paper presents a systematic synthesis procedure for H∞-optimal adaptive FIR filters in the context of an active noise cancellation (ANC) problem. An estimation interpretation of the adaptive control problem is introduced first. Based on this interpretation, an H∞ estimation problem is formulated, and its finite horizon prediction (filtering) solution is discussed. The solution minimizes the maximum energy gain from the disturbances to the predicted (filtered) estimation error and serves as the adaptation criterion for the weight vector in the adaptive FIR filter. We refer to this adaptation scheme as estimation-based adaptive filtering (EBAF). We show that the steady-state gain vector in the EBAF algorithm approaches that of the classical (normalized) filtered-X LMS algorithm. The error terms, however, are shown to be different. Thus, these classical algorithms can be considered to be approximations of our algorithm. We examine the performance of the proposed EBAF algorithm (both experimentally and in simulation) in an active noise cancellation problem of a one-dimensional (1-D) acoustic duct for both narrowband and broadband cases. Comparisons to the results from a conventional filtered-LMS (FxLMS) algorithm show faster convergence without compromising steady-state performance and/or robustness of the algorithm to feedback contamination of the reference signal

Least quantile regression via modern optimization

We address the Least Quantile of Squares (LQS) (and in particular the Least Median of Squares) regression problem using modern optimization methods. We propose a Mixed Integer Optimization (MIO) formulation of the LQS problem which allows us to find a provably global optimal solution for the LQS problem. Our MIO framework has the appealing characteristic that if we terminate the algorithm early, we obtain a solution with a guarantee on its sub-optimality. We also propose continuous optimization methods based on first-order subdifferential methods, sequential linear optimization and hybrid combinations of them to obtain near optimal solutions to the LQS problem. The MIO algorithm is found to benefit significantly from high quality solutions delivered by our continuous optimization based methods. We further show that the MIO approach leads to (a) an optimal solution for any dataset, where the data-points $(y_i,\mathbf{x}_i)$'s are not necessarily in general position, (b) a simple proof of the breakdown point of the LQS objective value that holds for any dataset and (c) an extension to situations where there are polyhedral constraints on the regression coefficient vector. We report computational results with both synthetic and real-world datasets showing that the MIO algorithm with warm starts from the continuous optimization methods solve small ($n=100$) and medium ($n=500$) size problems to provable optimality in under two hours, and outperform all publicly available methods for large-scale ($n={}$10,000) LQS problems.Comment: Published in at http://dx.doi.org/10.1214/14-AOS1223 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

A Stochastic Interpretation of Stochastic Mirror Descent: Risk-Sensitive Optimality

Stochastic mirror descent (SMD) is a fairly new family of algorithms that has recently found a wide range of applications in optimization, machine learning, and control. It can be considered a generalization of the classical stochastic gradient algorithm (SGD), where instead of updating the weight vector along the negative direction of the stochastic gradient, the update is performed in a "mirror domain" defined by the gradient of a (strictly convex) potential function. This potential function, and the mirror domain it yields, provides considerable flexibility in the algorithm compared to SGD. While many properties of SMD have already been obtained in the literature, in this paper we exhibit a new interpretation of SMD, namely that it is a risk-sensitive optimal estimator when the unknown weight vector and additive noise are non-Gaussian and belong to the exponential family of distributions. The analysis also suggests a modified version of SMD, which we refer to as symmetric SMD (SSMD). The proofs rely on some simple properties of Bregman divergence, which allow us to extend results from quadratics and Gaussians to certain convex functions and exponential families in a rather seamless way

Adaptive filtering techniques for gravitational wave interferometric data: Removing long-term sinusoidal disturbances and oscillatory transients

It is known by the experience gained from the gravitational wave detector proto-types that the interferometric output signal will be corrupted by a significant amount of non-Gaussian noise, large part of it being essentially composed of long-term sinusoids with slowly varying envelope (such as violin resonances in the suspensions, or main power harmonics) and short-term ringdown noise (which may emanate from servo control systems, electronics in a non-linear state, etc.). Since non-Gaussian noise components make the detection and estimation of the gravitational wave signature more difficult, a denoising algorithm based on adaptive filtering techniques (LMS methods) is proposed to separate and extract them from the stationary and Gaussian background noise. The strength of the method is that it does not require any precise model on the observed data: the signals are distinguished on the basis of their autocorrelation time. We believe that the robustness and simplicity of this method make it useful for data preparation and for the understanding of the first interferometric data. We present the detailed structure of the algorithm and its application to both simulated data and real data from the LIGO 40meter proto-type.Comment: 16 pages, 9 figures, submitted to Phys. Rev.

Routing Unmanned Vehicles in GPS-Denied Environments

Most of the routing algorithms for unmanned vehicles, that arise in data gathering and monitoring applications in the literature, rely on the Global Positioning System (GPS) information for localization. However, disruption of GPS signals either intentionally or unintentionally could potentially render these algorithms not applicable. In this article, we present a novel method to address this difficulty by combining methods from cooperative localization and routing. In particular, the article formulates a fundamental combinatorial optimization problem to plan routes for an unmanned vehicle in a GPS-restricted environment while enabling localization for the vehicle. We also develop algorithms to compute optimal paths for the vehicle using the proposed formulation. Extensive simulation results are also presented to corroborate the effectiveness and performance of the proposed formulation and algorithms.Comment: Publised in International Conference on Umanned Aerial System

A Linear Multi-User Detector for STBC MC-CDMA Systems based on the Adaptive Implementation of the Minimum-Conditional Bit-Error-Rate Criterion and on Genetic Algorithm-assisted MMSE Channel Estimation

The implementation of efficient baseband receivers characterized by affordable computational load is a crucial point in the development of transmission systems exploiting diversity in different domains. In this paper, we are proposing a linear multi-user detector for MIMO MC-CDMA systems with Alamouti’s Space-Time Block Coding, inspired by the concept of Minimum Conditional Bit-Error-Rate (MCBER) and relying on Genetic-Algorithm (GA)-assisted MMSE channel estimation. The MCBER combiner has been implemented in adaptive way by using Least-Mean-Square (LMS) optimization. Firstly, we shall analyze the proposed adaptive MCBER MUD receiver with ideal knowledge of Channel Status Information (CSI). Afterwards, we shall consider the complete receiver structure, encompassing also the non-ideal GA-assisted channel estimation. Simulation results evidenced that the proposed MCBER receiver always outperforms state-of-the-art receiver schemes based on EGC and MMSE criterion exploiting the same degree of channel knowledge (i.e. ideal or estimated CSI)

Underdetermined-order recursive least-squares adaptive filtering: The concept and algorithms

Published versio