Bayesian nonparametric learning of complex dynamical phenomena

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 257-270).The complexity of many dynamical phenomena precludes the use of linear models for which exact analytic techniques are available. However, inference on standard nonlinear models quickly becomes intractable. In some cases, Markov switching processes, with switches between a set of simpler models, are employed to describe the observed dynamics. Such models typically rely on pre-specifying the number of Markov modes. In this thesis, we instead take a Bayesian nonparametric approach in defining a prior on the model parameters that allows for flexibility in the complexity of the learned model and for development of efficient inference algorithms. We start by considering dynamical phenomena that can be well-modeled as a hidden discrete Markov process, but in which there is uncertainty about the cardinality of the state space. The standard finite state hidden Markov model (HMM) has been widely applied in speech recognition, digital communications, and bioinformatics, amongst other fields. Through the use of the hierarchical Dirichlet process (HDP), one can examine an HMM with an unbounded number of possible states. We revisit this HDPHMM and develop a generalization of the model, the sticky HDP-HMM, that allows more robust learning of smoothly varying state dynamics through a learned bias towards self-transitions. We show that this sticky HDP-HMM not only better segments data according to the underlying state sequence, but also improves the predictive performance of the learned model. Additionally, the sticky HDP-HMM enables learning more complex, multimodal emission distributions.(cont.) We demonstrate the utility of the sticky HDP-HMM on the NIST speaker diarization database, segmenting audio files into speaker labels while simultaneously identifying the number of speakers present. Although the HDP-HMM and its sticky extension are very flexible time series models, they make a strong Markovian assumption that observations are conditionally independent given the discrete HMM state. This assumption is often insufficient for capturing the temporal dependencies of the observations in real data. To address this issue, we develop extensions of the sticky HDP-HMM for learning two classes of switching dynamical processes: the switching linear dynamical system (SLDS) and the switching vector autoregressive (SVAR) process. These conditionally linear dynamical models can describe a wide range of complex dynamical phenomena from the stochastic volatility of financial time series to the dance of honey bees, two examples we use to show the power and flexibility of our Bayesian nonparametric approach. For all of the presented models, we develop efficient Gibbs sampling algorithms employing a truncated approximation to the HDP that allows incorporation of dynamic programming techniques, greatly improving mixing rates. In many applications, one would like to discover and model dynamical behaviors which are shared among several related time series. By jointly modeling such sequences, we may more robustly estimate representative dynamic models, and also uncover interesting relationships among activities.(cont.) In the latter part of this thesis, we consider a Bayesian nonparametric approach to this problem by harnessing the beta process to allow each time series to have infinitely many potential behaviors, while encouraging sharing of behaviors amongst the time series. For this model, we develop an efficient and exact Markov chain Monte Carlo (MCMC) inference algorithm. In particular, we exploit the finite dynamical system induced by a fixed set of behaviors to efficiently compute acceptance probabilities, and reversible jump birth and death proposals to explore new behaviors. We present results on unsupervised segmentation of data from the CMU motion capture database.by Emily B. Fox.Ph.D

Bayesian Semiparametric Hidden Markov Tensor Partition Models for Longitudinal Data with Local Variable Selection

We present a flexible Bayesian semiparametric mixed model for longitudinal data analysis in the presence of potentially high-dimensional categorical covariates. Building on a novel hidden Markov tensor decomposition technique, our proposed method allows the fixed effects components to vary between dependent random partitions of the covariate space at different time points. The mechanism not only allows different sets of covariates to be included in the model at different time points but also allows the selected predictors' influences to vary flexibly over time. Smooth time-varying additive random effects are used to capture subject specific heterogeneity. We establish posterior convergence guarantees for both function estimation and variable selection. We design a Markov chain Monte Carlo algorithm for posterior computation. We evaluate the method's empirical performances through synthetic experiments and demonstrate its practical utility through real world applications

Bayesian nonparametrics for time series modeling

Mención Internacional en el título de doctorIn many real-world signal processing problems, an observed temporal sequence can be explained by several unobservable independent causes, and we are interested in recovering the canonical signals that lead to these observations. For example, we may want to separate the overlapping voices on a single recording, distinguish the individual players on a financial market, or recover the underlying brain signals from electroencephalography data. This problem, known as source separation, is in general highly underdetermined or ill-posed. Methods for source separation generally seek to narrow the set of possible solutions in a way that is unlikely to exclude the desired solution. However, most classical approaches for source separation assume a fixed and known number of latent sources. This may represent a limitation in contexts in which the number of independent causes is unknown and is not limited to a small range. In this Thesis, we address the signal separation problem from a probabilistic modeling perspective. We encode our independence assumptions in a probabilistic model and develop inference algorithms to unveil the underlying sequences that explain the observed signal. We adopt a Bayesian nonparametric (BNP) approach in order to let the inference procedure estimate the number of independent sequences that best explain the data. BNP models place a prior distribution over an infinite-dimensional parameter space, which makes them particularly useful in probabilistic models in which the number of hidden parameters is unknown a priori. Under this prior distribution, the posterior distribution of the hidden parameters given the data assigns higher probability mass to those configurations that best explain the observations. Hence, inference over the hidden variables is performed using standard Bayesian inference techniques, which avoids expensive model selection steps. We develop two novel BNP models for source separation in time series. First, we propose a non-binary infinite factorial hidden Markov model (IFHMM), in which the number of parallel chains of a factorial hidden Markov model (FHMM) is treated in a nonparametric fashion. This model constitutes an extension of the binary IFHMM, but the hidden states are not restricted to take binary values. Moreover, by placing a Poisson prior distribution over the cardinality of the hidden states, we develop the infinite factorial unbounded-state hidden Markov model (IFUHMM), and an inference algorithm that can infer both the number of chains and the number of states in the factorial model. Second, we introduce the infinite factorial finite state machine (IFFSM) model, in which the number of independent Markov chains is also potentially infinite, but each of them evolves according to a stochastic finite-memory finite state machine model. For the IFFSM, we apply an efficient inference algorithm, based on particle Markov chain Monte Carlo (MCMC) methods, that avoids the exponential runtime complexity of more standard MCMC algorithms such as forward-filtering backward-sampling. Although our models are applicable in a broad range of fields, we focus on two specific problems: power disaggregation and multiuser channel estimation and symbol detection. The power disaggregation problem consists in estimating the power draw of individual devices, given the aggregate whole-home power consumption signal. Blind multiuser channel estimation and symbol detection involves inferring the channel coefficients and the transmitted symbol in a multiuser digital communication system, such as a wireless communication network, with no need of training data. We assume that the number of electrical devices or the number of transmitters is not known in advance. Our experimental results show that the proposed methodology can provide accurate results, outperforming state-of-the-art approaches.En multitud de problemas reales de procesado de señal, se tiene acceso a una secuencia temporal que puede explicarse mediante varias causas latentes independientes, y el objetivo es la recuperación de las señales canónicas que dan lugar a dichas observaciones. Por ejemplo, podemos estar interesados en separar varias señales de voz solapadas en una misma grabación, distinguir los agentes que operan en un mismo mercado financiero, o recuperar las señales cerebrales a partir de los datos de un electroencefalograma. Este problema, conocido como separación de fuente, es en general sobredeterminado. Los métodos de separación de fuente normalmente tratan de reducir el conjunto de posibles soluciones de tal manera que sea poco probable excluir la solución deseada. Sin embargo, en la mayoría de métodos clásicos de separación de fuente, se asume que el número de fuentes latentes es conocido. Esto puede representar una limitación en aplicaciones en las que no se conoce el número de causas independientes y dicho número no está acotado en un pequeño intervalo. En esta Tesis, consideramos un enfoque probabilístico para el problema de separación de fuente, en el que las asunciones de independencia se pueden incluir en el modelo probabilístico, y desarrollamos algoritmos de inferencia que permiten recuperar las señales latentes que explican la secuencia observada. Nos basamos en la utilización de métodos bayesianos no paramétricos (BNP) para permitir al algoritmo estimar adicionalmente el número de secuencias que mejor expliquen los datos. Los modelos BNP nos permiten definir una distribución de probabilidad sobre un espacio de dimensionalidad infinita, lo cual los hace particularmente útiles para su aplicación en modelos probabilísticos en los que el número de parámetros ocultos es desconocido a priori. Bajo esta distribución de probabilidad, la distribución a posteriori sobre los parámetros ocultos del modelo, dados los datos, asignará una mayor densidad de probabilidad a las configuraciones que mejor expliquen las observaciones, evitando por tanto los métodos de selección de modelo, que son computacionalmente costosos. En esta Tesis, desarrollamos dos nuevos modelos BNP para la separación de fuente en secuencias temporales. En primer lugar, proponemos un modelo oculto de Markov factorial infinito (IFHMM) no binario, en el que tratamos de manera no paramétrica el número de cadenas paralelas de un modelo oculto de Markov factorial (FHMM). Este modelo constituye una extensión del IFHMM binario, pero se elimina la restricción de que los estados ocultos sean variables binarias. Además, imponiendo una distribución de Poisson sobre la cardinalidad de los estados ocultos, desarrollamos el modelo oculto de Markov factorial infinito con estados no acotados (IFUHMM), y un algoritmo de inferencia con la capacidad de inferir tanto el número de cadenas como el número de estados del modelo factorial. En segundo lugar, proponemos un modelo de máquina de estados factorial infinita (IFFSM), en el que el número de cadenas de Markov paralelas e independientes también es potencialmente infinito, pero cada una de ellas evoluciona según un modelo de máquina de estados estocástica con memoria finita. Para el IFFSM, aplicamos un eficiente algoritmo de inferencia, basado en métodos Markov chain Monte Carlo (MCMC) de partículas, que evita la complejidad exponencial en tiempo de ejecución de otros algoritmos MCMC más comunes, como el de filtrado hacia adelante y muestreo hacia atrás. A pesar de que nuestros modelos son aplicables en una amplia variedad de campos, nos centramos en dos problemas específicos: separación de energía, y estimación de canal y detección de símbolos en un sistema multi-usuario. El problema de separación de energía consiste en, dada la señal de potencia total consumida en una casa, estimar de manera individual el consumo de potencia de cada dispositivo. La estimación de canal y detección de símbolos consiste en inferir los coeficientes de canal y los símbolos transmitidos en un sistema de comunicaciones digital multiusuario, como una red de comunicaciones inalámbrica, sin necesidad de transmitir símbolos piloto. Asumimos que tanto el número de dispositivos eléctricos como el número de transmisores es en principio desconocido y no acotado. Los resultados experimentales demuestran que la metodología propuesta ofrece buenos resultados y presenta mejoras sobre otros métodos propuestos en la literatura.Beca FPU (referencia AP-2010-5333)Programa Oficial de Doctorado en Multimedia y ComunicacionesPresidente: Antonio Artés Rodríguez.- Secretario: Juan José Murillo Fuentes.- Vocal: Konstantina Pall

Bayesian partition models for local inference in longitudinal and survival data

This dissertation proposes novel Bayesian semiparametric and nonparametric methods for complex, large and potentially high-dimensional longitudinal and survival data. The first part, comprising the bulk of this thesis, develops sophisticated dynamic partition models for longitudinal data that allow common features to be shared across some time segments while differing across others. These ideas are then specifically adapted to develop novel drift-diffusion models for the analysis of behavioral data on category learning in auditory neuroscience. The second part of this work proposes a bivariate survival regression method, borrowing information across two outcomes via common features in parts of the induced marginal partitions. In terms of flexibility and interpretability, the methods presented here provide significant improvements over many previously available tools and techniques, leading to interesting, novel and meaningful inference in many diverse application areas.Statistic

Recommending Structured Objects: Paths and Sets

Recommender systems have been widely adopted in industry to help people find the most appropriate items to purchase or consume from the increasingly large collection of available resources (e.g., books, songs and movies). Conventional recommendation techniques follow the approach of ``ranking all possible options and pick the top'', which can work effectively for single item recommendation but fall short when the item in question has internal structures. For example, a travel trajectory with a sequence of points-of-interest or a music playlist with a set of songs. Such structured objects pose critical challenges to recommender systems due to the intractability of ranking all possible candidates. This thesis study the problem of recommending structured objects, in particular, the recommendation of path (a sequence of unique elements) and set (a collection of distinct elements). We study the problem of recommending travel trajectories in a city, which is a typical instance of path recommendation. We propose methods that combine learning to rank and route planning techniques for efficient trajectory recommendation. Another contribution of this thesis is to develop the structured recommendation approach for path recommendation by substantially modifying the loss function, the learning and inference procedures of structured support vector machines. A novel application of path decoding techniques helps us achieve efficient learning and recommendation. Additionally, we investigate the problem of recommending a set of songs to form a playlist as an example of the set recommendation problem. We propose to jointly learn user representations by employing the multi-task learning paradigm, and a key result of equivalence between bipartite ranking and binary classification enables efficient learning of our set recommendation method. Extensive evaluations on real world datasets demonstrate the effectiveness of our proposed approaches for path and set recommendation

Speech Recognition

Chapters in the first part of the book cover all the essential speech processing techniques for building robust, automatic speech recognition systems: the representation for speech signals and the methods for speech-features extraction, acoustic and language modeling, efficient algorithms for searching the hypothesis space, and multimodal approaches to speech recognition. The last part of the book is devoted to other speech processing applications that can use the information from automatic speech recognition for speaker identification and tracking, for prosody modeling in emotion-detection systems and in other speech processing applications that are able to operate in real-world environments, like mobile communication services and smart homes

Identifying and Modeling Code-Switched Language

Code-switching is the phenomenon by which bilingual speakers switch between multiple languages during written or spoken communication. The importance of developing language technologies that are able to process code-switched language is immense, given the large populations that routinely code-switch. Current NLP and Speech models break down when used on code-switched data, interrupting the language processing pipeline in back-end systems and forcing users to communicate in ways which for them are unnatural. There are four main challenges that arise in building code-switched models: lack of code-switched data on which to train generative language models; lack of multilingual language annotations on code-switched examples which are needed to train supervised models; little understanding of how to leverage monolingual and parallel resources to build better code-switched models; and finally, how to use these models to learn why and when code-switching happens across language pairs. In this thesis, I look into different aspects of these four challenges. The first part of this thesis focuses on how to obtain reliable corpora of code-switched language. We collected a large corpus of code-switched language from social media using a combination of sets of anchor words that exist in one language and sentence-level language taggers. The newly obtained corpus is superior to other corpora collected via different strategies when it comes to the amount and type of bilingualism in it. It also helps train better language tagging models. We also have proposed a new annotation scheme to obtain part-of-speech tags for code-switched English-Spanish language. The annotation scheme is composed of three different subtasks including automatic labeling, word-specific questions labeling and question-tree word labeling. The part-of-speech labels obtained for the Miami Bangor corpus of English-Spanish conversational speech show very high agreement and accuracy. The second section of this thesis focuses on the tasks of part-of-speech tagging and language modeling. For the first task, we proposed a state-of-the-art approach to part-of-speech tagging of code-switched English-Spanish data based on recurrent neural networks.Our models were tested on the Miami Bangor corpus on the task of POS tagging alone, for which we achieved 96.34% accuracy, and joint part-of-speech and language ID tagging,which achieved similar POS tagging accuracy (96.39%) and very high language ID accuracy (98.78%). For the task of language modeling, we first conducted an exhaustive analysis of the relationship between cognate words and code-switching. We then proposed a set of cognate-based features that helped improve language modeling performance by 12% relative points. Furthermore, we showed that these features can also be used across language pairs and still obtain performance improvements. Finally, we tackled the question of how to use monolingual resources for code-switching models by pre-training state-of-the-art cross-lingual language models on large monolingual corpora and fine-tuning them on the tasks of language modeling and word-level language tagging on code-switched data. We obtained state-of-the-art results on both tasks

Coping with uncertain dynamics in visual tracking : redundant state models and discrete search methods

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 133-142).A model of the world dynamics is a vital part of any tracking algorithm. The observed world can exhibit multiple complex dynamics at different spatio-temporal scales. Faithfully modeling all motion constraints in a computationally efficient manner may be too complicated or completely impossible. Resorting to use of approximate motion models complicates tracking by making it less robust to unmodeled noise and increasing running times. We propose two complimentary approaches to tracking with approximate dynamic models in a probabilistic setting. The Redundant State Multi-Chain Model formalism described in the first part of the thesis allows combining multiple weak motion models, each representing a particular aspect of overall dynamic, in a cooperative manner to improve state estimates. This is applicable, in particular, to hierarchical machine vision systems that combine trackers at several spatio-temporal scales. In the second part of the dissertation, we propose supplementing exploration of the continuous likelihood surface with the discrete search in a fixed set of points distributed through the state space. We demonstrate the utility of these approaches on a range of machine vision problems: adaptive background subtraction, structure from motion estimation, and articulated body tracking.by Leonid Taycher.Ph.D