Proximal Markov chain Monte Carlo algorithms

This paper presents a new Metropolis-adjusted Langevin algorithm (MALA) that uses convex analysis to simulate efficiently from high-dimensional densities that are log-concave, a class of probability distributions that is widely used in modern high-dimensional statistics and data analysis. The method is based on a new first-order approximation for Langevin diffusions that exploits log-concavity to construct Markov chains with favourable convergence properties. This approximation is closely related to Moreau--Yoshida regularisations for convex functions and uses proximity mappings instead of gradient mappings to approximate the continuous-time process. The proposed method complements existing MALA methods in two ways. First, the method is shown to have very robust stability properties and to converge geometrically for many target densities for which other MALA are not geometric, or only if the step size is sufficiently small. Second, the method can be applied to high-dimensional target densities that are not continuously differentiable, a class of distributions that is increasingly used in image processing and machine learning and that is beyond the scope of existing MALA and HMC algorithms. To use this method it is necessary to compute or to approximate efficiently the proximity mappings of the logarithm of the target density. For several popular models, including many Bayesian models used in modern signal and image processing and machine learning, this can be achieved with convex optimisation algorithms and with approximations based on proximal splitting techniques, which can be implemented in parallel. The proposed method is demonstrated on two challenging high-dimensional and non-differentiable models related to image resolution enhancement and low-rank matrix estimation that are not well addressed by existing MCMC methodology

First-order Convex Optimization Methods for Signal and Image Processing

In this thesis we investigate the use of first-order convex optimization methods applied to problems in signal and image processing. First we make a general introduction to convex optimization, first-order methods and their iteration com-plexity. Then we look at different techniques, which can be used with first-order methods such as smoothing, Lagrange multipliers and proximal gradient meth-ods. We continue by presenting different applications of convex optimization and notable convex formulations with an emphasis on inverse problems and sparse signal processing. We also describe the multiple-description problem. We finally present the contributions of the thesis. The remaining parts of the thesis consist of five research papers. The first paper addresses non-smooth first-order convex optimization and the trade-off between accuracy and smoothness of the approximating smooth function. The second and third papers concern discrete linear inverse problems and reliable numerical reconstruction software. The last two papers present a convex opti-mization formulation of the multiple-description problem and a method to solve it in the case of large-scale instances. i i

Low-rank and sparse reconstruction in dynamic magnetic resonance imaging via proximal splitting methods

Dynamic magnetic resonance imaging (MRI) consists of collecting multiple MR images in time, resulting in a spatio-temporal signal. However, MRI intrinsically suffers from long acquisition times due to various constraints. This limits the full potential of dynamic MR imaging, such as obtaining high spatial and temporal resolutions which are crucial to observe dynamic phenomena. This dissertation addresses the problem of the reconstruction of dynamic MR images from a limited amount of samples arising from a nuclear magnetic resonance experiment. The term limited can be explained by the approach taken in this thesis to speed up scan time, which is based on violating the Nyquist criterion by skipping measurements that would be normally acquired in a standard MRI procedure. The resulting problem can be classified in the general framework of linear ill-posed inverse problems. This thesis shows how low-dimensional signal models, specifically lowrank and sparsity, can help in the reconstruction of dynamic images from partial measurements. The use of these models are justified by significant developments in signal recovery techniques from partial data that have emerged in recent years in signal processing. The major contributions of this thesis are the development and characterisation of fast and efficient computational tools using convex low-rank and sparse constraints via proximal gradient methods, the development and characterisation of a novel joint reconstruction–separation method via the low-rank plus sparse matrix decomposition technique, and the development and characterisation of low-rank based recovery methods in the context of dynamic parallel MRI. Finally, an additional contribution of this thesis is to formulate the various MR image reconstruction problems in the context of convex optimisation to develop algorithms based on proximal splitting methods

International Conference on Continuous Optimization (ICCOPT) 2019 Conference Book

The Sixth International Conference on Continuous Optimization took place on the campus of the Technical University of Berlin, August 3-8, 2019. The ICCOPT is a flagship conference of the Mathematical Optimization Society (MOS), organized every three years. ICCOPT 2019 was hosted by the Weierstrass Institute for Applied Analysis and Stochastics (WIAS) Berlin. It included a Summer School and a Conference with a series of plenary and semi-plenary talks, organized and contributed sessions, and poster sessions. This book comprises the full conference program. It contains, in particular, the scientific program in survey style as well as with all details, and information on the social program, the venue, special meetings, and more

Advanced tensor based signal processing techniques for wireless communication systems and biomedical signal processing

Many observed signals in signal processing applications including wireless communications, biomedical signal processing, image processing, and machine learning are multi-dimensional. Tensors preserve the multi-dimensional structure and provide a natural representation of these signals/data. Moreover, tensors provide often an improved identifiability. Therefore, we benefit from using tensor algebra in the above mentioned applications and many more. In this thesis, we present the benefits of utilizing tensor algebra in two signal processing areas. These include signal processing for MIMO (Multiple-Input Multiple-Output) wireless communication systems and biomedical signal processing. Moreover, we contribute to the theoretical aspects of tensor algebra by deriving new properties and ways of computing tensor decompositions. Often, we only have an element-wise or a slice-wise description of the signal model. This representation of the signal model does not reveal the explicit tensor structure. Therefore, the derivation of all tensor unfoldings is not always obvious. Consequently, exploiting the multi-dimensional structure of these models is not always straightforward. We propose an alternative representation of the element-wise multiplication or the slice-wise multiplication based on the generalized tensor contraction operator. Later in this thesis, we exploit this novel representation and the properties of the contraction operator such that we derive the final tensor models. There exist a number of different tensor decompositions that describe different signal models such as the HOSVD (Higher Order Singular Value Decomposition), the CP/PARAFAC (Canonical Polyadic / PARallel FACtors) decomposition, the BTD (Block Term Decomposition), the PARATUCK2 (PARAfac and TUCker2) decomposition, and the PARAFAC2 (PARAllel FACtors2) decomposition. Among these decompositions, the CP decomposition is most widely spread and used. Therefore, the development of algorithms for the efficient computation of the CP decomposition is important for many applications. The SECSI (Semi-Algebraic framework for approximate CP decomposition via SImultaneaous matrix diagonalization) framework is an efficient and robust tool for the calculation of the approximate low-rank CP decomposition via simultaneous matrix diagonalizations. In this thesis, we present five extensions of the SECSI framework that reduce the computational complexity of the original framework and/or introduce constraints to the factor matrices. Moreover, the PARAFAC2 decomposition and the PARATUCK2 decomposition are usually described using a slice-wise notation that can be expressed in terms of the generalized tensor contraction as proposed in this thesis. We exploit this novel representation to derive explicit tensor models for the PARAFAC2 decomposition and the PARATUCK2 decomposition. Furthermore, we use the PARAFAC2 model to derive an ALS (Alternating Least-Squares) algorithm for the computation of the PARAFAC2 decomposition. Moreover, we exploit the novel contraction properties for element wise and slice-wise multiplications to model MIMO multi-carrier wireless communication systems. We show that this very general model can be used to derive the tensor model of the received signal for MIMO-OFDM (Multiple-Input Multiple-Output - Orthogonal Frequency Division Multiplexing), Khatri-Rao coded MIMO-OFDM, and randomly coded MIMO-OFDM systems. We propose the transmission techniques Khatri-Rao coding and random coding in order to impose an additional tensor structure of the transmit signal tensor that otherwise does not have a particular structure. Moreover, we show that this model can be extended to other multi-carrier techniques such as GFDM (Generalized Frequency Division Multiplexing). Utilizing these models at the receiver side, we design several types for receivers for these systems that outperform the traditional matrix based solutions in terms of the symbol error rate. In the last part of this thesis, we show the benefits of using tensor algebra in biomedical signal processing by jointly decomposing EEG (ElectroEncephaloGraphy) and MEG (MagnetoEncephaloGraphy) signals. EEG and MEG signals are usually acquired simultaneously, and they capture aspects of the same brain activity. Therefore, EEG and MEG signals can be decomposed using coupled tensor decompositions such as the coupled CP decomposition. We exploit the proposed coupled SECSI framework (one of the proposed extensions of the SECSI framework) for the computation of the coupled CP decomposition to first validate and analyze the photic driving effect. Moreover, we validate the effects of scull defects on the measurement EEG and MEG signals by means of a joint EEG-MEG decomposition using the coupled SECSI framework. Both applications show that we benefit from coupled tensor decompositions and the coupled SECSI framework is a very practical tool for the analysis of biomedical data.Zahlreiche messbare Signale in verschiedenen Bereichen der digitalen Signalverarbeitung, z.B. in der drahtlosen Kommunikation, im Mobilfunk, biomedizinischen Anwendungen, der Bild- oder akustischen Signalverarbeitung und dem maschinellen Lernen sind mehrdimensional. Tensoren erhalten die mehrdimensionale Struktur und stellen eine natürliche Darstellung dieser Signale/Daten dar. Darüber hinaus bieten Tensoren oft eine verbesserte Trennbarkeit von enthaltenen Signalkomponenten. Daher profitieren wir von der Verwendung der Tensor-Algebra in den oben genannten Anwendungen und vielen mehr. In dieser Arbeit stellen wir die Vorteile der Nutzung der Tensor-Algebra in zwei Bereichen der Signalverarbeitung vor: drahtlose MIMO (Multiple-Input Multiple-Output) Kommunikationssysteme und biomedizinische Signalverarbeitung. Darüber hinaus tragen wir zu theoretischen Aspekten der Tensor-Algebra bei, indem wir neue Eigenschaften und Berechnungsmethoden für die Tensor-Zerlegung ableiten. Oftmals verfügen wir lediglich über eine elementweise oder ebenenweise Beschreibung des Signalmodells, welche nicht die explizite Tensorstruktur zeigt. Daher ist die Ableitung aller Tensor-Unfoldings nicht offensichtlich, wodurch die multidimensionale Struktur dieser Modelle nicht trivial nutzbar ist. Wir schlagen eine alternative Darstellung der elementweisen Multiplikation oder der ebenenweisen Multiplikation auf der Grundlage des generalisierten Tensor-Kontraktionsoperators vor. Weiterhin nutzen wir diese neuartige Darstellung und deren Eigenschaften zur Ableitung der letztendlichen Tensor-Modelle. Es existieren eine Vielzahl von Tensor-Zerlegungen, die verschiedene Signalmodelle beschreiben, wie die HOSVD (Higher Order Singular Value Decomposition), CP/PARAFAC (Canonical Polyadic/ PARallel FACtors) Zerlegung, die BTD (Block Term Decomposition), die PARATUCK2-(PARAfac und TUCker2) und die PARAFAC2-Zerlegung (PARAllel FACtors2). Dabei ist die CP-Zerlegung am weitesten verbreitet und wird findet in zahlreichen Gebieten Anwendung. Daher ist die Entwicklung von Algorithmen zur effizienten Berechnung der CP-Zerlegung von besonderer Bedeutung. Das SECSI (Semi-Algebraic Framework for approximate CP decomposition via Simultaneaous matrix diagonalization) Framework ist ein effizientes und robustes Werkzeug zur Berechnung der approximierten Low-Rank CP-Zerlegung durch simultane Matrixdiagonalisierung. In dieser Arbeit stellen wir fünf Erweiterungen des SECSI-Frameworks vor, welche die Rechenkomplexität des ursprünglichen Frameworks reduzieren bzw. Einschränkungen für die Faktormatrizen einführen. Darüber hinaus werden die PARAFAC2- und die PARATUCK2-Zerlegung in der Regel mit einer ebenenweisen Notation beschrieben, die sich in Form der allgemeinen Tensor-Kontraktion, wie sie in dieser Arbeit vorgeschlagen wird, ausdrücken lässt. Wir nutzen diese neuartige Darstellung, um explizite Tensormodelle für diese beiden Zerlegungen abzuleiten. Darüber hinaus verwenden wir das PARAFAC2-Modell, um einen ALS-Algorithmus (Alternating Least-Squares) für die Berechnung der PARAFAC2-Zerlegungen abzuleiten. Weiterhin nutzen wir die neuartigen Kontraktionseigenschaften für elementweise und ebenenweise Multiplikationen, um MIMO Multi-Carrier-Mobilfunksysteme zu modellieren. Wir zeigen, dass dieses sehr allgemeine Modell verwendet werden kann, um das Tensor-Modell des empfangenen Signals für MIMO-OFDM- (Multiple- Input Multiple-Output - Orthogonal Frequency Division Multiplexing), Khatri-Rao codierte MIMO-OFDM- und zufällig codierte MIMO-OFDM-Systeme abzuleiten. Wir schlagen die Übertragungstechniken der Khatri-Rao-Kodierung und zufällige Kodierung vor, um eine zusätzliche Tensor-Struktur des Sendesignal-Tensors einzuführen, welcher gewöhnlich keine bestimmte Struktur aufweist. Darüber hinaus zeigen wir, dass dieses Modell auf andere Multi-Carrier-Techniken wie GFDM (Generalized Frequency Division Multiplexing) erweitert werden kann. Unter Verwendung dieser Modelle auf der Empfängerseite entwerfen wir verschiedene Typen von Empfängern für diese Systeme, die die traditionellen matrixbasierten Lösungen in Bezug auf die Symbolfehlerrate übertreffen. Im letzten Teil dieser Arbeit zeigen wir die Vorteile der Verwendung von Tensor-Algebra in der biomedizinischen Signalverarbeitung durch die gemeinsame Zerlegung von EEG-(ElectroEncephaloGraphy) und MEG- (MagnetoEncephaloGraphy) Signalen. Diese werden in der Regel gleichzeitig erfasst, wobei sie gemeinsame Aspekte derselben Gehirnaktivität beschreiben. Daher können EEG- und MEG-Signale mit gekoppelten Tensor-Zerlegungen wie der gekoppelten CP Zerlegung analysiert werden. Wir nutzen das vorgeschlagene gekoppelte SECSI-Framework (eine der vorgeschlagenen Erweiterungen des SECSI-Frameworks) für die Berechnung der gekoppelten CP Zerlegung, um zunächst den photic driving effect zu validieren und zu analysieren. Darüber hinaus validieren wir die Auswirkungen von Schädeldefekten auf die Messsignale von EEG und MEG durch eine gemeinsame EEG-MEG-Zerlegung mit dem gekoppelten SECSI-Framework. Beide Anwendungen zeigen, dass wir von gekoppelten Tensor-Zerlegungen profitieren, wobei die Methoden des gekoppelten SECSI-Frameworks erfolgreich zur Analyse biomedizinischer Daten genutzt werden können

Proximal methods for structured group features and correlation matrix nearness

Tesis doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Ingeniería Informática. Fecha de lectura: junio de 2014Optimization is ubiquitous in real life as many of the strategies followed both by nature and by humans aim to minimize a certain cost, or maximize a certain benefit. More specifically, numerous strategies in engineering are designed according to a minimization problem, although usually the problems tackled are convex with a di erentiable objective function, since these problems have no local minima and they can be solved with gradient-based techniques. Nevertheless, many interesting problems are not di erentiable, such as, for instance, projection problems or problems based on non-smooth norms. An approach to deal with them can be found in the theory of Proximal Methods (PMs), which are based on iterative local minimizations using the Proximity Operator (ProxOp) of the terms that compose the objective function. This thesis begins with a general introduction and a brief motivation of the work done. The state of the art in PMs is thoroughly reviewed, defining the basic concepts from the very beginning and describing the main algorithms, as far as possible, in a simple and self-contained way. After that, the PMs are employed in the field of supervised regression, where regularized models play a prominent role. In particular, some classical linear sparse models are reviewed and unified under the point of view of regularization, namely the Lasso, the Elastic–Network, the Group Lasso and the Group Elastic–Network. All these models are trained by minimizing an error term plus a regularization term, and thus they fit nicely in the domain of PMs, as the structure of the problem can be exploited by minimizing alternatively the di erent expressions that compose the objective function, in particular using the Fast Iterative Shrinkage–Thresholding Algorithm (FISTA). As a real-world application, it is shown how these models can be used to forecast wind energy, where they yield both good predictions in terms of the error and, more importantly, valuable information about the structure and distribution of the relevant features. Following with the regularized learning approach, a new regularizer is proposed, called the Group Total Variation, which is a group extension of the classical Total Variation regularizer and thus it imposes constancy over groups of features. In order to deal with it, an approach to compute its ProxOp is derived. Moreover, it is shown that this regularizer can be used directly to clean noisy multidimensional signals (such as colour images) or to define a new linear model, the Group Fused Lasso (GFL), which can be then trained using FISTA. It is also exemplified how this model, when applied to regression problems, is able to provide solutions that identify the underlying problem structure. As an additional result of this thesis, a public software implementation of the GFL model is provided. The PMs are also applied to the Nearest Correlation Matrix problem under observation uncertainty. The original problem consists in finding the correlation matrix which is nearest to the true empirical one. Some variants introduce weights to adapt the confidence given to each entry of the matrix; with a more general perspective, in this thesis the problem is explored directly considering uncertainty on the observations, which is formalized as a set of intervals where the measured matrices lie. Two di erent variants are defined under this framework: a robust approach called the Robust Nearest Correlation Matrix (which aims to minimize the worst-case scenario) and an exploratory approach, the Exploratory Nearest Correlation Matrix (which focuses on the best-case scenario). It is shown how both optimization problems can be solved using the Douglas–Rachford PM with a suitable splitting of the objective functions. The thesis ends with a brief overall discussion and pointers to further work.La optimización está presente en todas las facetas de la vida, de hecho muchas de las estrategias tanto de la naturaleza como del ser humano pretenden minimizar un cierto coste, o maximizar un cierto beneficio. En concreto, multitud de estrategias en ingeniería se diseñan según problemas de minimización, que habitualmente son problemas convexos con una función objetivo diferenciable, puesto que en ese caso no hay mínimos locales y los problemas pueden resolverse mediante técnicas basadas en gradiente. Sin embargo, hay muchos problemas interesantes que no son diferenciables, como por ejemplo problemas de proyección o basados en normas no suaves. Una aproximación para abordar estos problemas son los Métodos Proximales (PMs), que se basan en minimizaciones locales iterativas utilizando el Operador de Proximidad (ProxOp) de los términos de la función objetivo. La tesis comienza con una introducción general y una breve motivación del trabajo hecho. Se revisa en profundidad el estado del arte en PMs, definiendo los conceptos básicos y describiendo los algoritmos principales, dentro de lo posible, de forma simple y auto-contenida. Tras ello, se emplean los PMs en el campo de la regresión supervisada, donde los modelos regularizados tienen un papel prominente. En particular, se revisan y unifican bajo esta perspectiva de regularización algunos modelos lineales dispersos clásicos, a saber, Lasso, Elastic–Network, Lasso Grupal y Elastic–Network Grupal. Todos estos modelos se entrenan minimizando un término de error y uno de regularización, y por tanto encajan perfectamente en el dominio de los PMs, ya que la estructura del problema puede ser aprovechada minimizando alternativamente las diferentes expresiones que componen la función objetivo, en particular mediante el Algoritmo Fast Iterative Shrinkage–Thresholding (FISTA). Como aplicación al mundo real, se muestra que estos modelos pueden utilizarse para predecir energía eólica, donde proporcionan tanto buenos resultados en términos del error como información valiosa sobre la estructura y distribución de las características relevantes. Siguiendo con esta aproximación, se propone un nuevo regularizador, llamado Variación Total Grupal, que es una extensión grupal del regularizador clásico de Variación Total y que por tanto induce constancia sobre grupos de características. Para aplicarlo, se desarrolla una aproximación para calcular su ProxOp. Además, se muestra que este regularizador puede utilizarse directamente para limpiar señales multidimensionales ruidosas (como imágenes a color) o para definir un nuevo modelo lineal, el Fused Lasso Grupal (GFL), que se entrena con FISTA. Se ilustra cómo este modelo, cuando se aplica a problemas de regresión, es capaz de proporcionar soluciones que identifican la estructura subyacente del problema. Como resultado adicional de esta tesis, se publica una implementación software del modelo GFL. Asimismo, se aplican los PMs al problema de Matriz de Correlación Próxima (NCM) bajo incertidumbre. El problema original consiste en encontrar la matriz de correlación más cercana a la empírica verdadera. Algunas variantes introducen pesos para ajustar la confianza que se da a cada entrada de la matriz; con un carácter más general, en esta tesis se explora el problema considerando incertidumbre en las observaciones, que se formaliza como un conjunto de intervalos en el que se encuentran las matrices medidas. Bajo este marco se definen dos variantes: una aproximación robusta llamada NCM Robusta (que minimiza el caso peor) y una exploratoria, NCM Exploratoria (que se centra en el caso mejor). Ambos problemas de optimización pueden resolverse con el PM de Douglas–Rachford y una partición adecuada de las funciones objetivo. La tesis concluye con una discusión global y referencias a trabajo futur

スパース時間周波数表現に関する研究

早大学位記番号:新9160博士（工学）早稲田大