Convex Identifcation of Stable Dynamical Systems

This thesis concerns the scalable application of convex optimization to data-driven modeling of dynamical systems, termed system identi cation in the control community. Two problems commonly arising in system identi cation are model instability (e.g. unreliability of long-term, open-loop predictions), and nonconvexity of quality-of- t criteria, such as simulation error (a.k.a. output error). To address these problems, this thesis presents convex parametrizations of stable dynamical systems, convex quality-of- t criteria, and e cient algorithms to optimize the latter over the former. In particular, this thesis makes extensive use of Lagrangian relaxation, a technique for generating convex approximations to nonconvex optimization problems. Recently, Lagrangian relaxation has been used to approximate simulation error and guarantee nonlinear model stability via semide nite programming (SDP), however, the resulting SDPs have large dimension, limiting their practical utility. The rst contribution of this thesis is a custom interior point algorithm that exploits structure in the problem to signi cantly reduce computational complexity. The new algorithm enables empirical comparisons to established methods including Nonlinear ARX, in which superior generalization to new data is demonstrated. Equipped with this algorithmic machinery, the second contribution of this thesis is the incorporation of model stability constraints into the maximum likelihood framework. Speci - cally, Lagrangian relaxation is combined with the expectation maximization (EM) algorithm to derive tight bounds on the likelihood function, that can be optimized over a convex parametrization of all stable linear dynamical systems. Two di erent formulations are presented, one of which gives higher delity bounds when disturbances (a.k.a. process noise) dominate measurement noise, and vice versa. Finally, identi cation of positive systems is considered. Such systems enjoy substantially simpler stability and performance analysis compared to the general linear time-invariant iv Abstract (LTI) case, and appear frequently in applications where physical constraints imply nonnegativity of the quantities of interest. Lagrangian relaxation is used to derive new convex parametrizations of stable positive systems and quality-of- t criteria, and substantial improvements in accuracy of the identi ed models, compared to existing approaches based on weighted equation error, are demonstrated. Furthermore, the convex parametrizations of stable systems based on linear Lyapunov functions are shown to be amenable to distributed optimization, which is useful for identi cation of large-scale networked dynamical systems

Passivity enforcement via chordal methods

Orientador: Prof. Dr. Gustavo Henrique da Costa OliveiraTese (doutorado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa : Curitiba, 27/08/2019Inclui referências: p. 164-175Resumo: Neste documento são propostos três algoritmos inéditos associados aos problemas subsequentes de aferição e imposição da passividade, a qual é uma propriedade qualitativa, geral e fundamental na modelagem matemática de transitórios eletromagnéticos de sistemas elétricos passivos, como transformadores. Esses algoritmos baseiam-se numa combinação de teoria dos grafos e otimização convexa. O primeiro deles consiste na aferição de subsistemas passivos contidos num sistema não passivo, intuitivamente busca-se partes passivas contidas num todo não passivo. Já na etapa de imposição de passividade, o segundo algoritmo é consequência natural do primeiro: retendo apenas os parâmetros associados às partes passivas e descartando os demais, parte-se de um sistema passivo parcialmente especificado para se determinar novos parâmetros em substituição àqueles descartados de modo que o sistema como um todo seja passivo. A possibilidade de determinação dos novos parâmetros depende de uma propriedade topológica de um grafo associado às matrizes de parâmetros do modelo, tal propriedade é denominada cordalidade. O terceiro algoritmo aborda novamente a questão de imposição da passividade e também faz uso da cordalidade, não mais como condição de existência de solução, mas sim como uma forma de explorar a esparsidade das matrizes de parâmetros. O problema de imposição da passividade encerra dois desafios no seu processo de solução, a saber: (i) compensação de parâmetros resultando na degradação do modelo bem como (ii) longos tempos de solução. Os algoritmos ora propostos são uma resposta a essas questões e os resultados obtidos demonstraram-se comparáveis àqueles já existentes na literatura especializada, em alguns casos apresentando melhorias, seja em termos de aproximação ou tempo computacionais. Os algoritmos foram testados a partir de dados de medição de um Transformador de Potencial Indutivo bem como de um Transformador de Potência. Palavras-chave: Macro-modelagem Passiva. Teoria de Sistemas. Álgebra Linear Aplicada. Análise de Transitórios. Transformadores.Abstract: Three novel algorithms are herein proposed to solve passivity assessment and enforcement problems. Passivity is a general, qualitative and fundamental property pertaining to the modeling associated with electromagnetic transients in passive power systems, such as transformers. These algorithms make combined use of Graph Theory and Convex Optimization. The first algorithm is concerned with passivity assesment. In particular, it searches for passive subsystems embedded into a larger nonpassive system and eventually specifies a partially specified passive system. Focusing on the subsequent step, algorithm two is a natural consequence of the preceeding one: retaining only the parameter set associated with passive subsystems as determined before, this partially specified passive system is used to further determine the remaining parameters so that the entire system be fully specified and passive. The existence condition for finding a fully specified system hinges on the fulfillment of a topological property of the graph associated the parameter matrices, namely chordality. The third algorithm also solves the passivity enforcement problem by making use of chordality, not as an existence condition, but rather by exploiting chordal sparsity patterns obtained with the parameter matrices. Solving passivity enforcement problems entails two persisting challenges, namely: (i) passivity compensations to parameters prompting increased model degradation as well as (ii) large computation times. The algorithms herein proposed tackle these issues and yield results comparable to those already in use, sometimes resulting in improved performance in terms of either approximation accuracy or runtime. These results herein reported entail data from actual measurements of an Inductive Voltage Transformer and a Power Transformer. Keywords: Passive Macromodeling. System Theory. Applied Linear Algebra. Transient Analysis. Transformers

Non-convex Optimization for Machine Learning

A vast majority of machine learning algorithms train their models and perform inference by solving optimization problems. In order to capture the learning and prediction problems accurately, structural constraints such as sparsity or low rank are frequently imposed or else the objective itself is designed to be a non-convex function. This is especially true of algorithms that operate in high-dimensional spaces or that train non-linear models such as tensor models and deep networks. The freedom to express the learning problem as a non-convex optimization problem gives immense modeling power to the algorithm designer, but often such problems are NP-hard to solve. A popular workaround to this has been to relax non-convex problems to convex ones and use traditional methods to solve the (convex) relaxed optimization problems. However this approach may be lossy and nevertheless presents significant challenges for large scale optimization. On the other hand, direct approaches to non-convex optimization have met with resounding success in several domains and remain the methods of choice for the practitioner, as they frequently outperform relaxation-based techniques - popular heuristics include projected gradient descent and alternating minimization. However, these are often poorly understood in terms of their convergence and other properties. This monograph presents a selection of recent advances that bridge a long-standing gap in our understanding of these heuristics. The monograph will lead the reader through several widely used non-convex optimization techniques, as well as applications thereof. The goal of this monograph is to both, introduce the rich literature in this area, as well as equip the reader with the tools and techniques needed to analyze these simple procedures for non-convex problems.Comment: The official publication is available from now publishers via http://dx.doi.org/10.1561/220000005

Sensor Signal and Information Processing II

In the current age of information explosion, newly invented technological sensors and software are now tightly integrated with our everyday lives. Many sensor processing algorithms have incorporated some forms of computational intelligence as part of their core framework in problem solving. These algorithms have the capacity to generalize and discover knowledge for themselves and learn new information whenever unseen data are captured. The primary aim of sensor processing is to develop techniques to interpret, understand, and act on information contained in the data. The interest of this book is in developing intelligent signal processing in order to pave the way for smart sensors. This involves mathematical advancement of nonlinear signal processing theory and its applications that extend far beyond traditional techniques. It bridges the boundary between theory and application, developing novel theoretically inspired methodologies targeting both longstanding and emergent signal processing applications. The topic ranges from phishing detection to integration of terrestrial laser scanning, and from fault diagnosis to bio-inspiring filtering. The book will appeal to established practitioners, along with researchers and students in the emerging field of smart sensors processing

Novel methods for biological network inference: an application to circadian Ca2+ signaling network

Biological processes involve complex biochemical interactions among a large number of species like cells, RNA, proteins and metabolites. Learning these interactions is essential to interfering artificially with biological processes in order to, for example, improve crop yield, develop new therapies, and predict new cell or organism behaviors to genetic or environmental perturbations. For a biological process, two pieces of information are of most interest. For a particular species, the first step is to learn which other species are regulating it. This reveals topology and causality. The second step involves learning the precise mechanisms of how this regulation occurs. This step reveals the dynamics of the system. Applying this process to all species leads to the complete dynamical network. Systems biology is making considerable efforts to learn biological networks at low experimental costs. The main goal of this thesis is to develop advanced methods to build models for biological networks, taking the circadian system of Arabidopsis thaliana as a case study. A variety of network inference approaches have been proposed in the literature to study dynamic biological networks. However, many successful methods either require prior knowledge of the system or focus more on topology. This thesis presents novel methods that identify both network topology and dynamics, and do not depend on prior knowledge. Hence, the proposed methods are applicable to general biological networks. These methods are initially developed for linear systems, and, at the cost of higher computational complexity, can also be applied to nonlinear systems. Overall, we propose four methods with increasing computational complexity: one-to-one, combined group and element sparse Bayesian learning (GESBL), the kernel method and reversible jump Markov chain Monte Carlo method (RJMCMC). All methods are tested with challenging dynamical network simulations (including feedback, random networks, different levels of noise and number of samples), and realistic models of circadian system of Arabidopsis thaliana. These simulations show that, while the one-to-one method scales to the whole genome, the kernel method and RJMCMC method are superior for smaller networks. They are robust to tuning variables and able to provide stable performance. The simulations also imply the advantage of GESBL and RJMCMC over the state-of-the-art method. We envision that the estimated models can benefit a wide range of research. For example, they can locate biological compounds responsible for human disease through mathematical analysis and help predict the effectiveness of new treatments

Guided Matching Pursuit and its Application to Sound Source Separation

In the last couple of decades there has been an increasing interest in the application of source separation technologies to musical signal processing. Given a signal that consists of a mixture of musical sources, source separation aims at extracting and/or isolating the signals that correspond to the original sources. A system capable of high quality source separation could be an invaluable tool for the sound engineer as well as the end user. Applications of source separation include, but are not limited to, remixing, up-mixing, spatial re-configuration, individual source modification such as filtering, pitch detection/correction and time stretching, music transcription, voice recognition and source-specific audio coding to name a few. Of particular interest is the problem of separating sources from a mixture comprising two channels (2.0 format) since this is still the most commonly used format in the music industry and most domestic listening environments. When the number of sources is greater than the number of mixtures (which is usually the case with stereophonic recordings) then the problem of source separation becomes under-determined and traditional source separation techniques, such as “Independent Component Analysis” (ICA) cannot be successfully applied. In such cases a family of techniques known as “Sparse Component Analysis” (SCA) are better suited. In short a mixture signal is decomposed into a new domain were the individual sources are sparsely represented which implies that their corresponding coefficients will have disjoint (or almost) disjoint supports. Taking advantage of this property along with the spatial information within the mixture and other prior information that could be available, it is possible to identify the sources in the new domain and separate them by going back to the time domain. It is a fact that sparse representations lead to higher quality separation. Regardless, the most commonly used front-end for a SCA system is the ubiquitous short-time Fourier transform (STFT) which although is a sparsifying transform it is not the best choice for this job. A better alternative is the matching pursuit (MP) decomposition. MP is an iterative algorithm that decomposes a signal into a set of elementary waveforms called atoms chosen from an over-complete dictionary in such a way so that they represent the inherent signal structures. A crucial part of MP is the creation of the dictionary which directly affects the results of the decomposition and subsequently the quality of source separation. Selecting an appropriate dictionary could prove a difficult task and an adaptive approach would be appropriate. This work proposes a new MP variant termed guided matching pursuit (GMP) which adds a new pre-processing step into the main sequence of the MP algorithm. The purpose of this step is to perform an analysis of the signal and extract important features, termed guide maps, that are used to create dynamic mini-dictionaries comprising atoms which are expected to correlate well with the underlying signal structures thus leading to focused and more efficient searches around particular supports of the signal. This algorithm is accompanied by a modular and highly flexible MATLAB implementation which is suited to the processing of long duration audio signals. Finally the new algorithm is applied to the source separation of two-channel linear instantaneous mixtures and preliminary testing demonstrates that the performance of GMP is on par with the performance of state of the art systems

Online Machine Learning for Graph Topology Identification from Multiple Time Series

High dimensional time series data are observed in many complex systems. In networked data, some of the time series are influenced by other time series. Identifying these relations encoded in a graph structure or topology among the time series is of paramount interest in certain applications since the identified structure can provide insights about the underlying system and can assist in inference tasks. In practice, the underlying topology is usually sparse, that is, not all the participating time series in influence each other. The goal of this dissertation pertains to study the problem of sparse topology identification under various settings. Topology identification from time series is a challenging task. The first major challenge in topology identification is that the assumption of static topology does not hold always in practice since most of the practical systems are evolving with time. For instance, in econometrics, social networks, etc., the relations among the time series can change over time. Identifying the topologies of such dynamic networks is a major challenge. The second major challenge is that in most practical scenarios, the data is not available at once - it is coming in a streaming fashion. Hence, batch approaches are either not applicable or they become computationally expensive since a batch algorithm is needed to be run when a new datum becomes available. The third challenge is that the multi-dimensional time series data can contain missing values due faulty sensors, privacy and security reasons, or due to saving energy. We address the aforementioned challenges in this dissertation by proposing online/-batch algorithms to solve the problem of time-varying topology identification. A model based on vector autoregressive (VAR) process is adopted initially. The parameters of the VAR model reveal the topology of the underlying network. First, two online algorithms are proposed for the case of streaming data. Next, using the same VAR model, two online algorithms under the framework of online optimization are presented to track the time-varying topologies. To evaluate the performance of propose online algorithms, we show that both the proposed algorithms incur a sublinear static regret. To characterize the performance theoretically in time-varying scenarios, a bound on the dynamic regret for one of the proposed algorithms (TIRSO) is derived. Next, using a structural equation model (SEM) for topology identification, an online algorithm for tracking time-varying topologies is proposed, and a bound on the dynamic regret is also derived for the proposed algorithm. Moreover, using a non-stationary VAR model, an algorithm for dynamic topology identification and breakpoint detection is also proposed, where the notion of local structural breakpoint is introduced to accommodate the concept of breakpoint where instead of the whole topology, only a few edges vary. Finally, the problem of tracking VAR-based time-varying topologies with missing data is investigated. Online algorithms are proposed where the joint signal and topology estimation is carried out. Dynamic regret analysis is also presented for the proposed algorithm. For all the previously mentioned works, simulation tests about the proposed algorithms are also presented and discussed in this dissertation. The numerical results of the proposed algorithms corroborate with the theoretical analysis presented in this dissertation.publishedVersio

l0 Sparse signal processing and model selection with applications

Sparse signal processing has far-reaching applications including compressed sensing, media compression/denoising/deblurring, microarray analysis and medical imaging. The main reason for its popularity is that many signals have a sparse representation given that the basis is suitably selected. However the difficulty lies in developing an efficient method of recovering such a representation. To this aim, two efficient sparse signal recovery algorithms are developed in the first part of this thesis. The first method is based on direct minimization of the l0 norm via cyclic descent, which is called the L0LS-CD (l0 penalized least squares via cyclic descent) algorithm. The other method minimizes smooth approximations of sparsity measures including those of the l0 norm via the majorization minimization (MM) technique, which is called the QC (quadratic concave) algorithm. The L0LS-CD algorithm is developed further by extending it to its multivariate (V-L0LS-CD (vector L0LS-CD)) and group (gL0LS-CD (group L0LS-CD)) regression variants. Computational speed-ups to the basic cyclic descent algorithm are discussed and a greedy version of L0LS-CD is developed. Stability of these algorithms is analyzed and the impact of the penalty parameter and proper initialization on the algorithm performance are highlighted. A suitable method for performance comparison of sparse approximating algorithms in the presence of noise is established. Simulations compare L0LS-CD and V-L0LS-CD with a range of alternatives on under-determined as well as over-determined systems. The QC algorithm is applicable to a class of penalties that are neither convex nor concave but have what we call the quadratic concave property. Convergence proofs of this algorithm are presented and it is compared with the Newton algorithm, concave convex (CC) procedure, as well as with the class of proximity algorithms. Simulations focus on the smooth approximations of the l0 norm and compare them with other l0 denoising algorithms. Next, two applications of sparse modeling are considered. In the first application the L0LS-CD algorithm is extended to recover a sparse transfer function in the presence of coloured noise. The second uses gL0LS-CD to recover the topology of a sparsely connected network of dynamic systems. Both applications use Laguerre basis functions for model expansion. The role of model selection in sparse signal processing is widely neglected in literature. The tuning/penalty parameter of a sparse approximating problem should be selected using a model selection criterion which minimizes a desired discrepancy measure. Compared to the commonly used model selection methods, the SURE (Stein's unbiased risk estimator) estimator stands out as one which does not suffer from the limitations of other methods. Most model selection criterion are developed based on signal or prediction mean squared error. The last section of this thesis develops an SURE criterion instead for parameter mean square error and applies this result to l1 penalized least squares problem with grouped variables. Simulations based on topology identification of a sparse network are presented to illustrate and compare with alternative model selection criteria