Learning probability distributions generated by finite-state machines

We review methods for inference of probability distributions generated by probabilistic automata and related models for sequence generation. We focus on methods that can be proved to learn in the inference in the limit and PAC formal models. The methods we review are state merging and state splitting methods for probabilistic deterministic automata and the recently developed spectral method for nondeterministic probabilistic automata. In both cases, we derive them from a high-level algorithm described in terms of the Hankel matrix of the distribution to be learned, given as an oracle, and then describe how to adapt that algorithm to account for the error introduced by a finite sample.Peer ReviewedPostprint (author's final draft

Conditional Random Field Autoencoders for Unsupervised Structured Prediction

We introduce a framework for unsupervised learning of structured predictors with overlapping, global features. Each input's latent representation is predicted conditional on the observable data using a feature-rich conditional random field. Then a reconstruction of the input is (re)generated, conditional on the latent structure, using models for which maximum likelihood estimation has a closed-form. Our autoencoder formulation enables efficient learning without making unrealistic independence assumptions or restricting the kinds of features that can be used. We illustrate insightful connections to traditional autoencoders, posterior regularization and multi-view learning. We show competitive results with instantiations of the model for two canonical NLP tasks: part-of-speech induction and bitext word alignment, and show that training our model can be substantially more efficient than comparable feature-rich baselines

Pseudo Random Coins Show More Heads Than Tails

Tossing a coin is the most elementary Monte Carlo experiment. In a computer the coin is replaced by a pseudo random number generator. It can be shown analytically and by exact enumerations that popular random number generators are not capable of imitating a fair coin: pseudo random coins show more heads than tails. This bias explains the empirically observed failure of some random number generators in random walk experiments. It can be traced down to the special role of the value zero in the algebra of finite fields.Comment: 10 pages, 12 figure