Noise Benefits in Expectation-Maximization Algorithms

This dissertation shows that careful injection of noise into sample data can substantially speed up Expectation-Maximization algorithms. Expectation-Maximization algorithms are a class of iterative algorithms for extracting maximum likelihood estimates from corrupted or incomplete data. The convergence speed-up is an example of a noise benefit or "stochastic resonance" in statistical signal processing. The dissertation presents derivations of sufficient conditions for such noise-benefits and demonstrates the speed-up in some ubiquitous signal-processing algorithms. These algorithms include parameter estimation for mixture models, the $k$-means clustering algorithm, the Baum-Welch algorithm for training hidden Markov models, and backpropagation for training feedforward artificial neural networks. This dissertation also analyses the effects of data and model corruption on the more general Bayesian inference estimation framework. The main finding is a theorem guaranteeing that uniform approximators for Bayesian model functions produce uniform approximators for the posterior pdf via Bayes theorem. This result also applies to hierarchical and multidimensional Bayesian models.Comment: A Dissertation Presented to The Faculty of The USC Graduate School University of Southern California In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy (Electrical Engineering) August 2013. (252 pages, 45 figures), Online: http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll3/id/29434

DOI 10.1007/s10700-012-9130-0 Triply fuzzy function approximation for hierarchical Bayesian inference

Abstract We prove that three independent fuzzy systems can uniformly approximate Bayesian posterior probability density functions by approximating the prior and likelihood probability densities as well as the hyperprior probability densities that underly the priors. This triply fuzzy function approximation extends the recent theorem for uniformly approximating the posterior density by approximating just the prior and likelihood densities. This approximation allows users to state priors and hyper-priors in words or rules as well as to adapt them from sample data. A fuzzy system with just two rules can exactly represent common closed-form probability densities so long as they are bounded. The function approximators can also be neural networks or any other type of uniform function approximator. Iterative fuzzy Bayesian inference can lead to rule explosion. We prove that conjugacy in the if-part set functions for prior, hyperprior, and likelihood fuzzy approximators reduces rule explosion. We also prove that a type of semi-conjugacy of if-part set functions for those fuzzy approximators results in fewer parameters in the fuzzy posterior approximator