Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization

The problem of learning using connectionist networks, in which network connection strengths are modified systematically so that the response of the network increasingly approximates the desired response can be structured as an optimization problem. The widely used back propagation method of connectionist learning [19, 21, 18] is set in the context of nonlinear optimization. In this framework, the issues of stability, convergence and parallelism are considered. As a form of gradient descent with fixed step size, back propagation is known to be unstable, which is illustrated using Rosenbrock\u27s function. This is contrasted with stable methods which involve a line search in the gradient direction. The convergence criterion for connectionist problems involving binary functions is discussed relative to the behavior of gradient descent in the vicinity of local minima. A minimax criterion is compared with the least squares criterion. The contribution of the momentum term [19, 18] to more rapid convergence is interpreted relative to the geometry of the weight space. It is shown that in plateau regions of relatively constant gradient, the momentum term acts to increase the step size by a factor of 1/1-μ, where μ is the momentum term. In valley regions with steep sides, the momentum constant acts to focus the search direction toward the local minimum by averaging oscillations in the gradient

Stochastic quasi-Newton molecular simulations

Article / Letter to editorLeiden Institute of Chemistr

Stochastic quasi-Newton molecular simulations

Soft Matter Chemistr

A stochastic quasi Newton method for molecular simulations

In this thesis the Langevin equation with a space-dependent alternative mobility matrix has been considered. Simulations of a complex molecular system with many different length and time scales based on the fundamental equations of motion take a very long simulation time before capturing the functional and relevant motions. This problem is called critical slowing down. To avoid this problem multi-scale simulation methods are applied, which permits the use of different size time steps and thus enables acceleration of the relevant (slow) movements. The aim of this thesis is to develop a stochastic quasi Newton method, such that by incorporating multi-scaling the relevant motions are effectively taken with larger time step. Due to the integration of the slow motions with a larger time step, critical slowing down can be avoided. The proposed stochastic quasi Newton method enables automatic multi-scaling in the Langevin dynamics and contributes to efficient calculation of the noise term. The construction of the proposed method also enables the construction of a limited memory version for the mobility. This results in a method where less storage is needed. Together with the reduction of the computation time and the multi-scaling property, a powerful method for molecular simulations has been provided.UBL - phd migration 201

Learning Stable Koopman Models for Identification and Control of Dynamical Systems

Learning models of dynamical systems from data is a widely-studied problem in control theory and machine learning. One recent approach for modelling nonlinear systems considers the class of Koopman models, which embeds the nonlinear dynamics in a higher-dimensional linear subspace. Learning a Koopman embedding would allow for the analysis and control of nonlinear systems using tools from linear systems theory. Many recent methods have been proposed for data-driven learning of such Koopman embeddings, but most of these methods do not consider the stability of the Koopman model. Stability is an important and desirable property for models of dynamical systems. Unstable models tend to be non-robust to input perturbations and can produce unbounded outputs, which are both undesirable when the model is used for prediction and control. In addition, recent work has shown that stability guarantees may act as a regularizer for model fitting. As such, a natural direction would be to construct Koopman models with inherent stability guarantees. Two new classes of Koopman models are proposed that bridge the gap between Koopman-based methods and learning stable nonlinear models. The first model class is guaranteed to be stable, while the second is guaranteed to be stabilizable with an explicit stabilizing controller that renders the model stable in closed-loop. Furthermore, these models are unconstrained in their parameter sets, thereby enabling efficient optimization via gradient-based methods. Theoretical connections between the stability of Koopman models and forms of nonlinear stability such as contraction are established. To demonstrate the effect of the stability guarantees, the stable Koopman model is applied to a system identification problem, while the stabilizable model is applied to an imitation learning problem. Experimental results show empirically that the proposed models achieve better performance over prior methods without stability guarantees

First-order Convex Optimization Methods for Signal and Image Processing

In this thesis we investigate the use of first-order convex optimization methods applied to problems in signal and image processing. First we make a general introduction to convex optimization, first-order methods and their iteration com-plexity. Then we look at different techniques, which can be used with first-order methods such as smoothing, Lagrange multipliers and proximal gradient meth-ods. We continue by presenting different applications of convex optimization and notable convex formulations with an emphasis on inverse problems and sparse signal processing. We also describe the multiple-description problem. We finally present the contributions of the thesis. The remaining parts of the thesis consist of five research papers. The first paper addresses non-smooth first-order convex optimization and the trade-off between accuracy and smoothness of the approximating smooth function. The second and third papers concern discrete linear inverse problems and reliable numerical reconstruction software. The last two papers present a convex opti-mization formulation of the multiple-description problem and a method to solve it in the case of large-scale instances. i i

Hybrid solutions to instantaneous MIMO blind separation and decoding: narrowband, QAM and square cases

Future wireless communication systems are desired to support high data rates and high quality transmission when considering the growing multimedia applications. Increasing the channel throughput leads to the multiple input and multiple output and blind equalization techniques in recent years. Thereby blind MIMO equalization has attracted a great interest.Both system performance and computational complexities play important roles in real time communications. Reducing the computational load and providing accurate performances are the main challenges in present systems. In this thesis, a hybrid method which can provide an affordable complexity with good performance for Blind Equalization in large constellation MIMO systems is proposed first. Saving computational cost happens both in the signal sep- aration part and in signal detection part. First, based on Quadrature amplitude modulation signal characteristics, an efficient and simple nonlinear function for the Independent Compo- nent Analysis is introduced. Second, using the idea of the sphere decoding, we choose the soft information of channels in a sphere, and overcome the so- called curse of dimensionality of the Expectation Maximization (EM) algorithm and enhance the final results simultaneously. Mathematically, we demonstrate in the digital communication cases, the EM algorithm shows Newton -like convergence.Despite the widespread use of forward -error coding (FEC), most multiple input multiple output (MIMO) blind channel estimation techniques ignore its presence, and instead make the sim- plifying assumption that the transmitted symbols are uncoded. However, FEC induces code structure in the transmitted sequence that can be exploited to improve blind MIMO channel estimates. In final part of this work, we exploit the iterative channel estimation and decoding performance for blind MIMO equalization. Experiments show the improvements achievable by exploiting the existence of coding structures and that it can access the performance of a BCJR equalizer with perfect channel information in a reasonable SNR range. All results are confirmed experimentally for the example of blind equalization in block fading MIMO systems

Construction of a zero-coupon yield curve for the Nairobi Securities Exchange and its application in pricing derivatives

Thesis submitted in partial fulfillment of the requirements for the degree for PhD in Financial Mathematics at Strathmore UniversityYield curves are used to forecast interest rates for different products when their risk parameters are known, to calibrate no-arbitrage term structure models, and (mostly by investors) to detect whether there is arbitrage opportunity. By yield curve information, investors have opportunity of immunizing/hedging their investment portfolios against financial risks if they have to make an investment with some determined time of maturity. Private sector firms look at yields of different maturities and then choose their borrowing strategy. The differences in yields for long maturity and short maturities are an important indicator for central bank to use in monetary policy process. These differences may show the tightness of the government monetary policy and can be monitored to predict recession in coming years. A lot of research has been done in yield curve modeling and as we will see later in the thesis, most of the models developed had one major shortcoming: non differentiability at the interpolating knot points. The aim of this thesis is to construct a zero coupon yield curve for Nairobi Securities Exchange, and use the risk- free rates to price derivatives, with particular attention given to pricing coffee futures. This study looks into the three methods of constructing yield curves: by use of spline-based models, by interpolation and by using parametric models. We suggest an improvement in the interpolation methods used in the most celebrated spline-based model, monotonicity-preserving interpolation on r(t). We also use operator form of numerical differentiation to estimate the forward rates at the knot points, at which points the spot curve is non-differential. In derivative pricing, dynamical processes (Ito^ processes) are reviewed; and geometric Brownian motion is included, together with its properties and applications. Conventional techniques used in estimation of the drift and volatility parameters such as historical techniques are reviewed and discussed. We also use the Hough Transform, an artificial intelligence method, to detect market patterns and estimate the drift and volatility parameters simultaneously. We look at different ways of calculating derivative prices. For option pricing, we use different methods but apply Bellalahs models in calculation of the Coffee Futures prices because they incorporate an incomplete information parameter

Cancelamento de interferência em sistemas celulares distribuídos

Doutoramento em Engenharia ElectrotécnicaO tema principal desta tese é o problema de cancelamento de interferência para sistemas multi-utilizador, com antenas distribuídas. Como tal, ao iniciar, uma visão geral das principais propriedades de um sistema de antenas distribuídas é apresentada. Esta descrição inclui o estudo analítico do impacto da ligação, dos utilizadores do sistema, a mais antenas distribuídas. Durante essa análise é demonstrado que a propriedade mais importante do sistema para obtenção do ganho máximo, através da ligação de mais antenas de transmissão, é a simetria espacial e que os utilizadores nas fronteiras das células são os mais bene ciados. Tais resultados são comprovados através de simulação. O problema de cancelamento de interferência multi-utilizador é considerado tanto para o caso unidimensional (i.e. sem codi cação) como para o multidimensional (i.e. com codi cação). Para o caso unidimensional um algoritmo de pré-codi cação não-linear é proposto e avaliado, tendo como objectivo a minimização da taxa de erro de bit. Tanto o caso de portadora única como o de multipla-portadora são abordados, bem como o cenário de antenas colocadas e distribuidas. É demonstrado que o esquema proposto pode ser visto como uma extensão do bem conhecido esquema de zeros forçados, cuja desempenho é provado ser um limite inferior para o esquema generalizado. O algoritmo é avaliado, para diferentes cenários, através de simulação, a qual indica desempenho perto do óptimo, com baixa complexidade. Para o caso multi-dimensional um esquema para efectuar "dirty paper coding" binário, tendo como base códigos de dupla camada é proposto. No desenvolvimento deste esquema, a compressão com perdas de informação, é considerada como um subproblema. Resultados de simulação indicam transmissão dedigna proxima do limite de Shannon.This thesis focus on the interference cancellation problem for multiuser distributed antenna systems. As such it starts by giving an overview of the main properties of a distributed antenna system. This overview includes, an analytical investigation of the impact of the connection of additional distributed antennas, to the system users. That analysis shows that the most important system property to reach the maximum gain, with the connection of additional transmit antennas, is spatial symmetry and that the users at the cell borders are the most bene ted. The multiuser interference problem has been considered for both the one dimensional (i.e. without coding) and multidimensional (i.e. with coding) cases. In the unidimensional case, we propose and evaluate a nonlinear precoding algorithm for the minimization of the bit-error-rate, of a multiuser MIMO system. Both the single-carrier and multi-carrier cases are tackled as well as the co-located and distributed scenarios. It is demonstrated that the proposed scheme can be viewed as an extension of the well-known zero-forcing, whose performance is proven to be a lower bound for the generalized scheme. The algorithm was validated extensively through numerical simulations, which indicate a performance close to the optimal, with reduced complexity. For the multi-dimensional case, a binary dirty paper coding scheme, base on bilayer codes, is proposed. In the development of this scheme, we consider the lossy compression of a binary source as a sub-problem. Simulation results indicate reliable transmission close to the Shannon limit