Automatic Variational Inference in Stan

Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). The user only provides a Bayesian model and a dataset; nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We implement ADVI in Stan (code available now), a probabilistic programming framework. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan

Missing Spectrum-Data Recovery in Cognitive Radio Networks Using Piecewise Constant Nonnegative Matrix Factorization

In this paper, we propose a missing spectrum data recovery technique for cognitive radio (CR) networks using Nonnegative Matrix Factorization (NMF). It is shown that the spectrum measurements collected from secondary users (SUs) can be factorized as product of a channel gain matrix times an activation matrix. Then, an NMF method with piecewise constant activation coefficients is introduced to analyze the measurements and estimate the missing spectrum data. The proposed optimization problem is solved by a Majorization-Minimization technique. The numerical simulation verifies that the proposed technique is able to accurately estimate the missing spectrum data in the presence of noise and fading.Comment: 6 pages, 6 figures, Accepted for presentation in MILCOM'15 Conferenc

Automatic Differentiation Variational Inference

Probabilistic modeling is iterative. A scientist posits a simple model, fits it to her data, refines it according to her analysis, and repeats. However, fitting complex models to large data is a bottleneck in this process. Deriving algorithms for new models can be both mathematically and computationally challenging, which makes it difficult to efficiently cycle through the steps. To this end, we develop automatic differentiation variational inference (ADVI). Using our method, the scientist only provides a probabilistic model and a dataset, nothing else. ADVI automatically derives an efficient variational inference algorithm, freeing the scientist to refine and explore many models. ADVI supports a broad class of models-no conjugacy assumptions are required. We study ADVI across ten different models and apply it to a dataset with millions of observations. ADVI is integrated into Stan, a probabilistic programming system; it is available for immediate use

Blind source separation using statistical nonnegative matrix factorization

PhD ThesisBlind Source Separation (BSS) attempts to automatically extract and track a signal of interest in real world scenarios with other signals present. BSS addresses the problem of recovering the original signals from an observed mixture without relying on training knowledge. This research studied three novel approaches for solving the BSS problem based on the extensions of non-negative matrix factorization model and the sparsity regularization methods. 1) A framework of amalgamating pruning and Bayesian regularized cluster nonnegative tensor factorization with Itakura-Saito divergence for separating sources mixed in a stereo channel format: The sparse regularization term was adaptively tuned using a hierarchical Bayesian approach to yield the desired sparse decomposition. The modified Gaussian prior was formulated to express the correlation between different basis vectors. This algorithm automatically detected the optimal number of latent components of the individual source. 2) Factorization for single-channel BSS which decomposes an information-bearing matrix into complex of factor matrices that represent the spectral dictionary and temporal codes: A variational Bayesian approach was developed for computing the sparsity parameters for optimizing the matrix factorization. This approach combined the advantages of both complex matrix factorization (CMF) and variational -sparse analysis. BLIND SOURCE SEPARATION USING STATISTICAL NONNEGATIVE MATRIX FACTORIZATION ii 3) An imitated-stereo mixture model developed by weighting and time-shifting the original single-channel mixture where source signals can be modelled by the AR processes. The proposed mixing mixture is analogous to a stereo signal created by two microphones with one being real and another virtual. The imitated-stereo mixture employed the nonnegative tensor factorization for separating the observed mixture. The separability analysis of the imitated-stereo mixture was derived using Wiener masking. All algorithms were tested with real audio signals. Performance of source separation was assessed by measuring the distortion between original source and the estimated one according to the signal-to-distortion (SDR) ratio. The experimental results demonstrate that the proposed uninformed audio separation algorithms have surpassed among the conventional BSS methods; i.e. IS-cNTF, SNMF and CMF methods, with average SDR improvement in the ranges from 2.6dB to 6.4dB per source.Payap Universit

Automatic Localization of Epileptic Spikes in EEGs of Children with Infantile Spasms

Infantile Spasms (ISS) characterized by electroencephalogram (EEG) recordings exhibiting hypsarrythmia (HYPS) are a severe form of epilepsy. Many clinicians have been trying to improve ISS outcomes; however, quantification of discharges from hypsarrythmic EEG readings remains challenging. This thesis describes the development of a novel method that assists clinicians to successfully localize the epileptic discharges associated with ISS in HYPS. The approach includes: construct the time-frequency domain (TFD) of the EEG recording using matching pursuit TFD (MP-TFD), decompose the TFD matrix into two submatrices using nonnegative matrix factorizations (NMF), and employ the decomposed vectors to locate the spikes. The proposed method was employed to an EEG dataset of five ISS individuals, and identification of spikes was compared with those which were identified by the epileptologists and those obtained using clinical software (Persyst). Performance evaluations showed results based on classification techniques: thresholdings, and support vector machine (SVM). Using the thresholdings, average true positive (TP) and false negative (FN) percentages of 86% and 14% were achieved, which represented a significant improvement over the use of Persyst, which only achieved average TP and FN percentages of 4% and 96%, respectively. Using SVM, the percentage of area under curve (AUC) of receiver operating characteristic (ROC) was significantly improved up to 98.56%. In summary, the proposed novel algorithm based on MP-TFD and NMF was able to successfully detect the epileptic discharges from the dataset. The development of the proposed automated method can potentially assist clinicians to successfully localize the epileptic discharges associated with ISS in HYPS. The quantitative assessment of spike detection, as well as other features of HYPS, is expected to allow a more accurate assessment of the relevance of EEG to clinical outcomes, which is significant in therapy management of ISS