Faster inference from state space models via GPU computing

Funding: C.F.-J. is funded via a doctoral scholarship from the University of St Andrews, School of Mathematics and Statistics.Inexpensive Graphics Processing Units (GPUs) offer the potential to greatly speed up computation by employing their massively parallel architecture to perform arithmetic operations more efficiently. Population dynamics models are important tools in ecology and conservation. Modern Bayesian approaches allow biologically realistic models to be constructed and fitted to multiple data sources in an integrated modelling framework based on a class of statistical models called state space models. However, model fitting is often slow, requiring hours to weeks of computation. We demonstrate the benefits of GPU computing using a model for the population dynamics of British grey seals, fitted with a particle Markov chain Monte Carlo algorithm. Speed-ups of two orders of magnitude were obtained for estimations of the log-likelihood, compared to a traditional ‘CPU-only’ implementation, allowing for an accurate method of inference to be used where this was previously too computationally expensive to be viable. GPU computing has enormous potential, but one barrier to further adoption is a steep learning curve, due to GPUs' unique hardware architecture. We provide a detailed description of hardware and software setup, and our case study provides a template for other similar applications. We also provide a detailed tutorial-style description of GPU hardware architectures, and examples of important GPU-specific programming practices.Publisher PDFPeer reviewe

Estimation of Hidden Markov Models and Their Applications in Finance

Movements of financial variables exhibit extreme fluctuations during periods of economic crisis and times of market uncertainty. They are also affected by institutional policies and intervention of regulatory authorities. These structural changes driving prices and other economic indicators can be captured reasonably by models featuring regime-switching capabilities. Hidden Markov models (HMM) modulating the model parameters to incorporate such regime-switching dynamics have been put forward in recent years, but many of them could still be further improved. In this research, we aim to address some of the inadequacies of previous regime-switching models in terms of their capacity to provide better forecasts and efficiency in estimating parameters. New models are developed, and their corresponding filtering results are obtained and tested on financial data sets. The contributions of this research work include the following: (i) Recursive filtering algorithms are constructed for a regime-switching financial model consistent with no-arbitrage pricing. An application to the filtering and forecasting of futures prices under a multivariate set-up is presented. (ii) The modelling of risk due to market and funding liquidity is considered by capturing the joint dynamics of three time series (Treasury-Eurodollar spread, VIX and S\&P 500 spread-derived metric), which mirror liquidity levels in the financial markets. HMM filters under a multi-regime mean- reverting model are established. (iii) Kalman filtering techniques and the change of reference probability-based filtering methods are integrated to obtain hybrid algorithms. A pairs trading investment strategy is supported by the combined power of both HMM and Kalman filters. It is shown that an investor is able to benefit from the proposed interplay of the two filtering methods. (iv) A zero-delay HMM is devised for the evolution of multivariate foreign exchange rate data under a high-frequency trading environment. Recursive filters for quantities that are functions of a Markov chain are derived, which in turn provide optimal parameter estimates. (v) An algorithm is designed for the efficient calculation of the joint probability function for the occupation time in a Markov-modulated model for asset returns under a general number of economic regimes. The algorithm is constructed with accessible implementation and practical considerations in mind

Sequential Modelling and Inference of High-frequency Limit Order Book with State-space Models and Monte Carlo Algorithms

The high-frequency limit order book (LOB) market has recently attracted increasing research attention from both the industry and the academia as a result of expanding algorithmic trading. However, the massive data throughput and the inherent complexity of high-frequency market dynamics also present challenges to some classic statistical modelling approaches. By adopting powerful state-space models from the field of signal processing as well as a number of Bayesian inference algorithms such as particle filtering, Markov chain Monte Carlo and variational inference algorithms, this thesis presents my extensive research into the high-frequency limit order book covering a wide scope of topics. Chapter 2 presents a novel construction of the non-homogeneous Poisson process to allow online intensity inference of limit order transactions arriving at a central exchange as point data. Chapter 3 extends a baseline jump diffusion model for market fair-price process to include three additional model features taken from real-world market intuitions. In Chapter 4, another price model is developed to account for both long-term and short-term diffusion behaviours of the price process. This is achieved by incorporating multiple jump-diffusion processes each exhibiting a unique characteristic. Chapter 5 observes the multi-regime nature of price diffusion processes as well as the non-Markovian switching behaviour between regimes. As such, a novel model is proposed which combines the continuous-time state-space model, the hidden semi-Markov switching model and the non-parametric Dirichlet process model. Additionally, building upon the general structure of the particle Markov chain Monte Carlo algorithm, I further propose an algorithm which achieves sequential state inference, regime identification and regime parameters learning requiring minimal prior assumptions. Chapter 6 focuses on the development of efficient parameter-learning algorithms for state-space models and presents three algorithms each demonstrating promising results in comparison to some well-established methods. The models and algorithms proposed in this thesis not only are practical tools for analysing high-frequency LOB markets, but can also be applied in various areas and disciplines beyond finance

Inference of Markovian-regime-switching models with application to South Australian electricity prices

Markovian-Regime-Switching (MRS) models are commonly used for modelling economic time series, including electricity prices. In this application it is common to include inde- pendent regimes as these can more accurately capture the dynamics of electricity prices compared to traditional MRS models. The advantage of independent regime MRS specifications is that they allow us to seperate dynamics between regimes. Despite their popularity, parameter inference for MRS models with independent regimes is underdeveloped. Until this thesis, there was no computationally feasible method to evaluate the likelihood of, or find maximum likelihood estimate for, MRS models with independent regimes. Moreover, there are no good discussions of Bayesian methods for such models applied to electricity prices. In this thesis we develop both maximum likelihood and Bayesian inference methodologies for MRS models with independent regimes, and use simulations to investigate their behaviours. We use our methods to investigate the South Australian wholesale electricity market, and find evidence of a significant jump in price volatility which coincides with the closure of South Australia's only coal generation facility, and therefore a significant change in market structure. Our work also suggests that Bayesian methods can be advantageous compared to maximum likelihood, since Bayesian methods can avoid issues with inferring parameters of shifted distributions, which are commonly used in this context.Thesis (MPhil) -- University of Adelaide, School of Mathematical Sciences, 201

Methods of likelihood based inference for constructing stochastic climate models

This thesis is about the construction of low dimensional diffusion models of climate variables. It assesses the predictive skill of models derived from a principled averaging procedure and a purely empirical approach. The averaging procedure starts from the equations for the original system then approximates the \weather" variables by a stochastic process. They are then averaged with respect to their invariant measure. This assumes that they equilibriate much faster than the climate variables. The empirical approach argues for a very general model form, then parameters are estimated using likelihood based inference for Stochastic Differential Equations. This is computationally demanding and relies upon Markov Chain Monte Carlo methods. A large part of this thesis is focused upon techniques to improve the efficiency of these algorithms. The empirical approach works well on simple one dimensional models but performs poorly on multivariate problems due to the rapid increase in unknown parameters. The averaging procedure is skillful in multivariate problems but is sensitive to lack of complete time scale separation in the system. In conclusion, the averaging procedure is better and can be improved by estimating parameters in a principled way based on the likelihood function and by including a latent noise process in the model

A Celebration of the Ties That Bind Us: Connections Between Actuarial Science and Mathematical Finance

The articles in this volume are contributed by scholars who are not only experts in areas of Actuarial Science (AS) and Mathematical Finance (MF), but also those who present diverse perspectives from both industry and academia. Topics from multiple areas, such as Stochastic Modeling, Credit Risk, Monte Carlo Simulation, and Pension Valuation, among others, that were maybe thought to be the domain of one type of risk manager, are shown time and again to have deep value to other areas of risk management as well. The articles in this collection, in my opinion, contribute techniques, ideas, and overviews of tools that folks in both AS and MF will find useful and interesting to implement in their work. It is also my hope that this collection will inspire future collaboration between those who seek an interdisciplinary approach to risk management

Signal separation of musical instruments: simulation-based methods for musical signal decomposition and transcription

This thesis presents techniques for the modelling of musical signals, with particular regard to monophonic and polyphonic pitch estimation. Musical signals are modelled as a set of notes, each comprising of a set of harmonically-related sinusoids. An hierarchical model is presented that is very general and applicable to any signal that can be decomposed as the sum of basis functions. Parameter estimation is posed within a Bayesian framework, allowing for the incorporation of prior information about model parameters. The resulting posterior distribution is of variable dimension and so reversible jump MCMC simulation techniques are employed for the parameter estimation task. The extension of the model to time-varying signals with high posterior correlations between model parameters is described. The parameters and hyperparameters of several frames of data are estimated jointly to achieve a more robust detection. A general model for the description of time-varying homogeneous and heterogeneous multiple component signals is developed, and then applied to the analysis of musical signals. The importance of high level musical and perceptual psychological knowledge in the formulation of the model is highlighted, and attention is drawn to the limitation of pure signal processing techniques for dealing with musical signals. Gestalt psychological grouping principles motivate the hierarchical signal model, and component identifiability is considered in terms of perceptual streaming where each component establishes its own context. A major emphasis of this thesis is the practical application of MCMC techniques, which are generally deemed to be too slow for many applications. Through the design of efficient transition kernels highly optimised for harmonic models, and by careful choice of assumptions and approximations, implementations approaching the order of realtime are viable.Engineering and Physical Sciences Research Counci

Manycore Algorithms for Genetic Linkage Analysis

Exact algorithms to perform linkage analysis scale exponentially with the size of the input. Beyond a critical point, the amount of work that needs to be done exceeds both available time and memory. In these circumstances, we are forced to either abbreviate the input in some manner or else use an approximation. Approximate methods, like Markov chain Monte Carlo (MCMC), though they make the problem tractable, can take an immense amount of time to converge. The problem of high convergence time is compounded by software which is single-threaded and, as computer processors are manufactured with increasing numbers of physical processing cores, are not designed to take advantage of the available processing power. In this thesis, we will describe our program SwiftLink that embodies our work adapting existing Gibbs samplers to modern computer processor architectures. The processor architectures we target are: multicore processors, that currently feature between 4–8 processor cores, and computer graphics cards (GPUs) that already feature hundreds of processor cores. We implemented parallel versions of the meiosis sampler, that mixes well with tightly linked markers but suffers from irreducibility issues, and the locus sampler which is guaranteed to be irreducible but mixes slowly with tightly linked markers. We evaluate SwiftLink’s performance on real-world datasets of large consanguineous families. We demonstrate that using four processor cores for a single analysis is 3–3.2x faster than the single-threaded implementation of SwiftLink. With respect to the existing MCMC-based programs: it achieves a 6.6–8.7x speedup compared to Morgan and a 66.4– 72.3x speedup compared to Simwalk. Utilising both a multicore processor and a GPU performs 7–7.9x faster than the single-threaded implementation, a 17.6–19x speedup compared to Morgan and a 145.5–192.3x speedup compared to Simwalk

Non-parametric Bayesian models for structured output prediction

Structured output prediction is a machine learning tasks in which an input object is not just assigned a single class, as in classification, but multiple, interdependent labels. This means that the presence or value of a given label affects the other labels, for instance in text labelling problems, where output labels are applied to each word, and their interdependencies must be modelled. Non-parametric Bayesian (NPB) techniques are probabilistic modelling techniques which have the interesting property of allowing model capacity to grow, in a controllable way, with data complexity, while maintaining the advantages of Bayesian modelling. In this thesis, we develop NPB algorithms to solve structured output problems. We first study a map-reduce implementation of a stochastic inference method designed for the infinite hidden Markov model, applied to a computational linguistics task, part-of-speech tagging. We show that mainstream map-reduce frameworks do not easily support highly iterative algorithms. The main contribution of this thesis consists in a conceptually novel discriminative model, GPstruct. It is motivated by labelling tasks, and combines attractive properties of conditional random fields (CRF), structured support vector machines, and Gaussian process (GP) classifiers. In probabilistic terms, GPstruct combines a CRF likelihood with a GP prior on factors; it can also be described as a Bayesian kernelized CRF. To train this model, we develop a Markov chain Monte Carlo algorithm based on elliptical slice sampling and investigate its properties. We then validate it on real data experiments, and explore two topologies: sequence output with text labelling tasks, and grid output with semantic segmentation of images. The latter case poses scalability issues, which are addressed using likelihood approximations and an ensemble method which allows distributed inference and prediction. The experimental validation demonstrates: (a) the model is flexible and its constituent parts are modular and easy to engineer; (b) predictive performance and, most crucially, the probabilistic calibration of predictions are better than or equal to that of competitor models, and (c) model hyperparameters can be learnt from data