Least Squares for Practitioners

In experimental science and engineering, least squares are ubiquitous in analysis and digital data processing applications. Minimizing sums of squares of some quantities can be interpreted in very different ways and confusion can arise in practice, especially concerning the optimality and reliability of the results. Interpretations of least squares in terms of norms and likelihoods need to be considered to provide guidelines for general users. Assuming minimal prerequisites, the following expository discussion is intended to elaborate on some of the mathematical characteristics of the least-squares methodology and some closely related questions in the analysis of the results, model identification, and reliability for practical applications. Examples of simple applications are included to illustrate some of the advantages, disadvantages, and limitations of least squares in practice. Concluding remarks summarize the situation and provide some indications of practical areas of current research and development

Enforcing Statistical Orthogonality in Massive MIMO Systems via Covariance Shaping

This paper tackles the problem of downlink transmission in massive multiple-input multiple-output(MIMO) systems where the equipments (UEs) exhibit high spatial correlation and the channel estimation is limited by strong pilot contamination. Signal subspace separation among the UEs is, in fact, rarely realized in practice and is generally beyond the control of the network designer (as it is dictated by the physical scattering environment). In this context, we propose a novel statistical beamforming technique, referred to asMIMO covariance shaping, that exploits multiple antennas at the UEs and leverages the realistic non-Kronecker structure of massive MIMO channels to target a suitable shaping of the channel statistics performed at the UE-side. To optimize the covariance shaping strategies, we propose a low-complexity block coordinate descent algorithm that is proved to converge to a limit point of the original nonconvex problem. For the two-UE case, this is shown to converge to a stationary point of the original problem. Numerical results illustrate the sum-rate performance gains of the proposed method with respect to reference scenarios employing the multiple antennas at the UE for spatial multiplexing.Comment: Submitted for journal publicatio

Optimal linear shrinkage corrections of sample LMMSE and MVDR estimators

La proposició d'estimadors shrinkage òptims que corregeixen la degradació dels mètodes sample LMMSE i sample MUDR en el règim on el número de mostres és petit en comparació a la dimensió de les observacions.[ANGLÈS] This master thesis proposes optimal shrinkage estimators that counteract the performance degradation of the sample LMMSE and sample MVDR methods in the regime where the sample size is small compared to the observation dimension.[CASTELLÀ] Esta máster tesis propone estimadores shrinkage óptimos que corrigen la degradación de los métodos sample LMMSE y sample MVDR en el régimen donde el número de muestras es pequeño en comparación con la dimensión de las observaciones.[CATALÀ] Aquesta màster tesi proposa estimadors shrinkage òptims que corregeixen la degradació dels mètodes sample LMMSE i sample MVDR en el règim on el número de mostres és petit en comparació a la dimensió de les observacions

Shrinkage corrections of sample linear estimators in the small sample size regime

We are living in a data deluge era where the dimensionality of the data gathered by inexpensive sensors is growing at a fast pace, whereas the availability of independent samples of the observed data is limited. Thus, classical statistical inference methods relying on the assumption that the sample size is large, compared to the observation dimension, are suffering a severe performance degradation. Within this context, this thesis focus on a popular problem in signal processing, the estimation of a parameter, observed through a linear model. This inference is commonly based on a linear filtering of the data. For instance, beamforming in array signal processing, where a spatial filter steers the beampattern of the antenna array towards a direction to obtain the signal of interest (SOI). In signal processing the design of the optimal filters relies on the optimization of performance measures such as the Mean Square Error (MSE) and the Signal to Interference plus Noise Ratio (SINR). When the first two moments of the SOI are known, the optimization of the MSE leads to the Linear Minimum Mean Square Error (LMMSE). When such statistical information is not available one may force a no distortion constraint towards the SOI in the optimization of the MSE, which is equivalent to maximize the SINR. This leads to the Minimum Variance Distortionless Response (MVDR) method. The LMMSE and MVDR are optimal, though unrealizable in general, since they depend on the inverse of the data correlation, which is not known. The common approach to circumvent this problem is to substitute it for the inverse of the sample correlation matrix (SCM), leading to the sample LMMSE and sample MVDR. This approach is optimal when the number of available statistical samples tends to infinity for a fixed observation dimension. This large sample size scenario hardly holds in practice and the sample methods undergo large performance degradations in the small sample size regime, which may be due to short stationarity constraints or to a system with a high observation dimension. The aim of this thesis is to propose corrections of sample estimators, such as the sample LMMSE and MVDR, to circumvent their performance degradation in the small sample size regime. To this end, two powerful tools are used, shrinkage estimation and random matrix theory (RMT). Shrinkage estimation introduces a structure on the filters that forces some corrections in small sample size situations. They improve sample based estimators by optimizing a bias variance tradeoff. As direct optimization of these shrinkage methods leads to unrealizable estimators, then a consistent estimate of these optimal shrinkage estimators is obtained, within the general asymptotics where both the observation dimension and the sample size tend to infinity, but at a fixed rate. That is, RMT is used to obtain consistent estimates within an asymptotic regime that deals naturally with the small sample size. This RMT approach does not require any assumptions about the distribution of the observations. The proposed filters deal directly with the estimation of the SOI, which leads to performance gains compared to related work methods based on optimizing a metric related to the data covariance estimate or proposing rather ad-hoc regularizations of the SCM. Compared to related work methods which also treat directly the estimation of the SOI and which are based on a shrinkage of the SCM, the proposed filter structure is more general. It contemplates corrections of the inverse of the SCM and considers the related work methods as particular cases. This leads to performance gains which are notable when there is a mismatch in the signature vector of the SOI. This mismatch and the small sample size are the main sources of degradation of the sample LMMSE and MVDR. Thus, in the last part of this thesis, unlike the previous proposed filters and the related work, we propose a filter which treats directly both sources of degradation.Estamos viviendo en una era en la que la dimensión de los datos, recogidos por sensores de bajo precio, está creciendo a un ritmo elevado, pero la disponibilidad de muestras estadísticamente independientes de los datos es limitada. Así, los métodos clásicos de inferencia estadística sufren una degradación importante, ya que asumen un tamaño muestral grande comparado con la dimensión de los datos. En este contexto, esta tesis se centra en un problema popular en procesado de señal, la estimación lineal de un parámetro observado mediante un modelo lineal. Por ejemplo, la conformación de haz en procesado de agrupaciones de antenas, donde un filtro enfoca el haz hacia una dirección para obtener la señal asociada a una fuente de interés (SOI). El diseño de los filtros óptimos se basa en optimizar una medida de prestación como el error cuadrático medio (MSE) o la relación señal a ruido más interferente (SINR). Cuando hay información sobre los momentos de segundo orden de la SOI, la optimización del MSE lleva a obtener el estimador lineal de mínimo error cuadrático medio (LMMSE). Cuando esa información no está disponible, se puede forzar la restricción de no distorsión de la SOI en la optimización del MSE, que es equivalente a maximizar la SINR. Esto conduce al estimador de Capon (MVDR). El LMMSE y MVDR son óptimos, pero no son realizables, ya que dependen de la inversa de la matriz de correlación de los datos, que no es conocida. El procedimiento habitual para solventar este problema es sustituirla por la inversa de la correlación muestral (SCM), esto lleva al LMMSE y MVDR muestral. Este procedimiento es óptimo cuando el tamaño muestral tiende a infinito y la dimensión de los datos es fija. En la práctica este tamaño muestral elevado no suele producirse y los métodos LMMSE y MVDR muestrales sufren una degradación importante en este régimen de tamaño muestral pequeño. Éste se puede deber a periodos cortos de estacionariedad estadística o a sistemas cuya dimensión sea elevada. El objetivo de esta tesis es proponer correcciones de los estimadores LMMSE y MVDR muestrales que permitan combatir su degradación en el régimen de tamaño muestral pequeño. Para ello se usan dos herramientas potentes, la estimación shrinkage y la teoría de matrices aleatorias (RMT). La estimación shrinkage introduce una estructura de los estimadores que mejora los estimadores muestrales mediante la optimización del compromiso entre media y varianza del estimador. La optimización directa de los métodos shrinkage lleva a métodos no realizables. Por eso luego se propone obtener una estimación consistente de ellos en el régimen asintótico en el que tanto la dimensión de los datos como el tamaño muestral tienden a infinito, pero manteniendo un ratio constante. Es decir RMT se usa para obtener estimaciones consistentes en un régimen asintótico que trata naturalmente las situaciones de tamaño muestral pequeño. Esta metodología basada en RMT no requiere suposiciones sobre el tipo de distribución de los datos. Los filtros propuestos tratan directamente la estimación de la SOI, esto lleva a ganancias de prestaciones en comparación a otros métodos basados en optimizar una métrica relacionada con la estimación de la covarianza de los datos o regularizaciones ad hoc de la SCM. La estructura de filtro propuesta es más general que otros métodos que también tratan directamente la estimación de la SOI y que se basan en un shrinkage de la SCM. Contemplamos correcciones de la inversa de la SCM y los métodos del estado del arte son casos particulares. Esto lleva a ganancias de prestaciones que son notables cuando hay una incertidumbre en el vector de firma asociado a la SOI. Esa incertidumbre y el tamaño muestral pequeño son las fuentes de degradación de los LMMSE y MVDR muestrales. Así, en la última parte de la tesis, a diferencia de métodos propuestos previamente en la tesis y en la literatura, se propone un filtro que trata de forma directa ambas fuentes de degradación

Resource Constrained Adaptive Sensing.

RESOURCE CONSTRAINED ADAPTIVE SENSING by Raghuram Rangarajan Chair: Alfred O. Hero III Many signal processing methods in applications such as radar imaging, communication systems, and wireless sensor networks can be presented in an adaptive sensing context. The goal in adaptive sensing is to control the acquisition of data measurements through adaptive design of the input parameters, e.g., waveforms, energies, projections, and sensors for optimizing performance. This dissertation develops new methods for resource constrained adaptive sensing in the context of parameter estimation and detection, sensor management, and target tracking. We begin by investigating the advantages of adaptive waveform amplitude design for estimating parameters of an unknown channel/medium under average energy constraints. We present a statistical framework for sequential design (e.g., design of waveforms in adaptive sensing) of experiments that improves parameter estimation (e.g., scatter coefficients for radar imaging, channel coefficients for channel estimation) performance in terms of reduction in mean-squared error (MSE). We derive optimal adaptive energy allocation strategies that achieve an MSE improvement of more than 5dB over non adaptive methods. As a natural extension to the problem of estimation, we derive optimal energy allocation strategies for binary hypotheses testing under the frequentist and Bayesian frameworks which yield at least 2dB improvement in performance. We then shift our focus towards spatial design of waveforms by considering the problem of optimal waveform selection from a large waveform library for a state estimation problem. Since the optimal solution to this subset selection problem is combinatorially complex, we propose a convex relaxation to the problem and provide a low complexity suboptimal solution that achieves near optimal performance. Finally, we address the problem of sensor and target localization in wireless sensor networks. We develop a novel sparsity penalized multidimensional scaling algorithm for blind target tracking, i.e., a sensor network which can simultaneously track targets and obtain sensor location estimates.Ph.D.Electrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/57621/2/rangaraj_1.pd