Model Selection and Adaptive Markov chain Monte Carlo for Bayesian Cointegrated VAR model

This paper develops a matrix-variate adaptive Markov chain Monte Carlo (MCMC) methodology for Bayesian Cointegrated Vector Auto Regressions (CVAR). We replace the popular approach to sampling Bayesian CVAR models, involving griddy Gibbs, with an automated efficient alternative, based on the Adaptive Metropolis algorithm of Roberts and Rosenthal, (2009). Developing the adaptive MCMC framework for Bayesian CVAR models allows for efficient estimation of posterior parameters in significantly higher dimensional CVAR series than previously possible with existing griddy Gibbs samplers. For a n-dimensional CVAR series, the matrix-variate posterior is in dimension $3n^2 + n$, with significant correlation present between the blocks of matrix random variables. We also treat the rank of the CVAR model as a random variable and perform joint inference on the rank and model parameters. This is achieved with a Bayesian posterior distribution defined over both the rank and the CVAR model parameters, and inference is made via Bayes Factor analysis of rank. Practically the adaptive sampler also aids in the development of automated Bayesian cointegration models for algorithmic trading systems considering instruments made up of several assets, such as currency baskets. Previously the literature on financial applications of CVAR trading models typically only considers pairs trading (n=2) due to the computational cost of the griddy Gibbs. We are able to extend under our adaptive framework to $n >> 2$ and demonstrate an example with n = 10, resulting in a posterior distribution with parameters up to dimension 310. By also considering the rank as a random quantity we can ensure our resulting trading models are able to adjust to potentially time varying market conditions in a coherent statistical framework.Comment: to appear journal Bayesian Analysi

Nonparametric Involutive Markov Chain Monte Carlo: a MCMC algorithm for universal probabilistic programming

Probabilistic programming, the idea to write probabilistic models as computer programs, has proven to be a powerful tool for statistical analysis thanks to the computation power of built-in inference algorithms. Developing suitable inference algorithms that work for arbitrary programs in a Turing-complete probabilistic programming language (PPL) has become increasingly important. This thesis presents the Nonparametric Involutive Markov chain Monte Carlo (NP-iMCMC) framework for the construction of MCMC inference machines for nonparametric models that can be expressed in Turing-complete PPLs. Relying on the tree representable structure of probabilistic programs, the NP-iMCMC algorithm automates the trans-dimensional movement in the sampling process and only requires the specification of proposal distributions and mappings on fixed dimensional spaces which are provided by inferences like the popular Hamiltonian Monte Carlo (HMC). We gave a theoretical justification for the NP-iMCMC algorithm and put NP-iMCMC into action by introducing the Nonparametric HMC (NP-HMC) algorithm, a nonparametric variant of the HMC sampler. This NP-HMC sampler works out-of-the-box and can be applied to virtually all useful probabilistic models. We further improved NP-HMC by applying the techniques specified for NP-iMCMC to construct irreversible extensions that have shown significant performance improvements against other existing inference methods

Advances in approximate Bayesian computation and trans-dimensional sampling methodology

Bayesian statistical models continue to grow in complexity, driven in part by a few key factors: the massive computational resources now available to statisticians; the substantial gains made in sampling methodology and algorithms such as Markov chain Monte Carlo (MCMC), trans-dimensional MCMC (TDMCMC), sequential Monte Carlo (SMC), adaptive algorithms and stochastic approximation methods and approximate Bayesian computation (ABC); and development of more realistic models for real world phenomena as demonstrated in this thesis for financial models and telecommunications engineering. Sophisticated statistical models are increasingly proposed for practical solutions to real world problems in order to better capture salient features of increasingly more complex data. With sophistication comes a parallel requirement for more advanced and automated statistical computational methodologies. The key focus of this thesis revolves around innovation related to the following three significant Bayesian research questions. 1. How can one develop practically useful Bayesian models and corresponding computationally efficient sampling methodology, when the likelihood model is intractable? 2. How can one develop methodology in order to automate Markov chain Monte Carlo sampling approaches to efficiently explore the support of a posterior distribution, defined across multiple Bayesian statistical models? 3. How can these sophisticated Bayesian modelling frameworks and sampling methodologies be utilized to solve practically relevant and important problems in the research fields of financial risk modeling and telecommunications engineering ? This thesis is split into three bodies of work represented in three parts. Each part contains journal papers with novel statistical model and sampling methodological development. The coherent link between each part involves the novel sampling methodologies developed in Part I and utilized in Part II and Part III. Papers contained in each part make progress at addressing the core research questions posed. Part I of this thesis presents generally applicable key statistical sampling methodologies that will be utilized and extended in the subsequent two parts. In particular it presents novel developments in statistical methodology pertaining to likelihood-free or ABC and TDMCMC methodology. The TDMCMC methodology focuses on several aspects of automation in the between model proposal construction, including approximation of the optimal between model proposal kernel via a conditional path sampling density estimator. Then this methodology is explored for several novel Bayesian model selection applications including cointegrated vector autoregressions (CVAR) models and mixture models in which there is an unknown number of mixture components. The second area relates to development of ABC methodology with particular focus on SMC Samplers methodology in an ABC context via Partial Rejection Control (PRC). In addition to novel algorithmic development, key theoretical properties are also studied for the classes of algorithms developed. Then this methodology is developed for a highly challenging practically significant application relating to multivariate Bayesian $\alpha$-stable models. Then Part II focuses on novel statistical model development in the areas of financial risk and non-life insurance claims reserving. In each of the papers in this part the focus is on two aspects: foremost the development of novel statistical models to improve the modeling of risk and insurance; and then the associated problem of how to fit and sample from such statistical models efficiently. In particular novel statistical models are developed for Operational Risk (OpRisk) under a Loss Distributional Approach (LDA) and for claims reserving in Actuarial non-life insurance modelling. In each case the models developed include an additional level of complexity which adds flexibility to the model in order to better capture salient features observed in real data. The consequence of the additional complexity comes at the cost that standard fitting and sampling methodologies are generally not applicable, as a result one is required to develop and apply the methodology from Part I. Part III focuses on novel statistical model development in the area of statistical signal processing for wireless communications engineering. Statistical models will be developed or extended for two general classes of wireless communications problem: the first relates to detection of transmitted symbols and joint channel estimation in Multiple Input Multiple Output (MIMO) systems coupled with Orthogonal Frequency Division Multiplexing (OFDM); the second relates to co-operative wireless communications relay systems in which the key focus is on detection of transmitted symbols. Both these areas will require advanced sampling methodology developed in Part I to find solutions to these real world engineering problems

Monte Carlo methods based on novel classes of regeneration-enriched Markov processes

Enriching some underlying continuous-time Markov process with regenerations from a fixed regeneration distribution µ at a particular regeneration rate Ƙ results in a Markov process that has a target distribution π as its invariant distribution. Firstly, we introduce a method for adapting the regeneration distribution, which allows a significantly smaller regeneration rate to be used, which makes simulation feasible for a wider range of target distributions. The regeneration distribution is adapted on-the-fly, by adding point masses to it. Secondly, we show that a class of non- π -invariant jump processes, which are defined on an augmented statespace and have a jump chain transition kernel corresponding to a deterministic, invertible mapping, may be enriched with regenerations so that the resulting process is π -invariant. Since the underlying jump process does not need to be π invariant, its dynamics may be chosen to use gradient information to guide the process to areas of high probability mass, which makes the sampler a promising algorithm for multi-modal target distributions

Studies in probabilistic methods for scene analysis

In this thesis, probabilistic methods are applied to a number of problems in computer vision. The goal is to provide means for a vision based system that is able to analyze and recognize scenes and objects in camera images and to use that information for autonomous navigation and machine learning. New methods are developed for different functions that are needed in such a system, including segmentation of images, model-based recognition of objects, robot navigation and model complexity control. The approach is based on generative probability models, and Bayesian statistical inference is used to match these models with image data. Stochastic sampling methods are applied to obtain numerical results. The self-organizing map is a neural network algorithm that has many applications in computer vision. In this thesis, the algorithm is analyzed in a probabilistic framework. A probability density model is derived and new model selection techniques are proposed, which enable complexity control for the self-organizing map. The analysis of images is discussed from the point of view of segmentation and object recognition. Segmentation aims at dividing the image into parts of different appearance, while object recognition is meant to identify objects that fulfill given criteria. These are different goals, but they complement each other. When the recognition of all objects in an image is not possible, segmentation can provide an explanation to the rest of the image. For object recognition, different two and three dimensional object models are considered and Bayesian matching techniques are applied to them. Efficient techniques for image segmentation are proposed and results are presented.Tässä väitöskirjassa sovelletaan todennäköisyyslaskennan menetelmiä eräisiin tietokonenäköongelmiin. Työn tarkoituksena on tuottaa keinoja näköön perustuvaan järjestelmään, joka voi analysoida ja tunnistaa näkymiä ja kohteita kamerakuvista ja käyttää näin saatua informaatiota itsenäiseen navigointiin ja koneoppimiseen. Työssä kehitetään uusia menetelmiä järjestelmän tarvitsemiin toimintoihin kuten kuvien segmentointiin, mallipohjaiseen kohteiden tunnistukseen, robottinavigointiin ja mallien kompleksisuuden hallintaan. Työssä käytettävä lähestymistapa perustuu generatiivisiin todennäköisyysmalleihin, ja mallit sovitetaan kuvadataan bayesiläistä tilastollista päättelyä soveltaen. Numeeristen tulosten saamiseksi käytetään stokastisia poimintamenetelmiä. Itsejärjestyvä kartta on neuroverkkoalgoritmi, jolla on useita tietokonenäköalan sovelluksia. Tässä työssä algoritmia analysoidaan todennäköisyyspohjaisesti. Algoritmin tuottamalle mallille johdetaan todennäköisyysjakaumamalli ja sille esitetään uusia mallinvalintamenetelmiä, jotka mahdollistavat itsejärjestyvän kartan kompleksisuuden hallinnan. Kuvien analysointia käsitellään sekä segmentoinnin että kohteiden tunnistuksen näkökulmasta. Segmentoinnissa kuva jaetaan erilaisilta näyttäviin osiin. Kohteiden tunnistus perustuu niiden ennalta tunnettuihin ominaisuuksiin. Tavoitteet ovat siten varsin erilaisia, mutta ne täydentävät toisiaan. Silloin kun vain osa kuvassa olevista kohteista pystytään tunnistamaan, segmentoinnilla voidaan saada kuvan muille osille selitys. Väitöskirjassa esitetään laskennallisesti tehokkaita menetelmiä kuvien segmentointiin. Kohteiden tunnistusta kaksi- ja kolmiulotteisten mallien avulla tarkastellaan bayesiläisiä menetelmiä käyttäen.reviewe