91 research outputs found
Learning Nonsingular Phylogenies and Hidden Markov Models
In this paper we study the problem of learning phylogenies and hidden Markov models. We call a Markov model nonsingular if all transition matrices have determinants bounded away from 0 (and 1). We highlight the role of the nonsingularity condition for the learning problem. Learning hidden Markov models without the nonsingularity condition is at least as hard as learning parity with noise, a well-known learning problem conjectured to be computationally hard. On the other hand, we give a polynomial-time algorithm for learning nonsingular phylogenies and hidden Markov models
Learning loopy graphical models with latent variables: Efficient methods and guarantees
The problem of structure estimation in graphical models with latent variables
is considered. We characterize conditions for tractable graph estimation and
develop efficient methods with provable guarantees. We consider models where
the underlying Markov graph is locally tree-like, and the model is in the
regime of correlation decay. For the special case of the Ising model, the
number of samples required for structural consistency of our method scales
as , where p is the
number of variables, is the minimum edge potential, is
the depth (i.e., distance from a hidden node to the nearest observed nodes),
and is a parameter which depends on the bounds on node and edge
potentials in the Ising model. Necessary conditions for structural consistency
under any algorithm are derived and our method nearly matches the lower bound
on sample requirements. Further, the proposed method is practical to implement
and provides flexibility to control the number of latent variables and the
cycle lengths in the output graph.Comment: Published in at http://dx.doi.org/10.1214/12-AOS1070 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Uniform Chernoff and Dvoretzky-Kiefer-Wolfowitz-type inequalities for Markov chains and related processes
We observe that the technique of Markov contraction can be used to establish
measure concentration for a broad class of non-contracting chains. In
particular, geometric ergodicity provides a simple and versatile framework.
This leads to a short, elementary proof of a general concentration inequality
for Markov and hidden Markov chains (HMM), which supercedes some of the known
results and easily extends to other processes such as Markov trees. As
applications, we give a Dvoretzky-Kiefer-Wolfowitz-type inequality and a
uniform Chernoff bound. All of our bounds are dimension-free and hold for
countably infinite state spaces
Learning Mixtures of Distributions over Large Discrete Domains
We discuss recent results giving algorithms for learning mixtures of unstructured distributions
Identifiability and Unmixing of Latent Parse Trees
This paper explores unsupervised learning of parsing models along two
directions. First, which models are identifiable from infinite data? We use a
general technique for numerically checking identifiability based on the rank of
a Jacobian matrix, and apply it to several standard constituency and dependency
parsing models. Second, for identifiable models, how do we estimate the
parameters efficiently? EM suffers from local optima, while recent work using
spectral methods cannot be directly applied since the topology of the parse
tree varies across sentences. We develop a strategy, unmixing, which deals with
this additional complexity for restricted classes of parsing models
Alignment-free phylogenetic reconstruction: Sample complexity via a branching process analysis
We present an efficient phylogenetic reconstruction algorithm allowing
insertions and deletions which provably achieves a sequence-length requirement
(or sample complexity) growing polynomially in the number of taxa. Our
algorithm is distance-based, that is, it relies on pairwise sequence
comparisons. More importantly, our approach largely bypasses the difficult
problem of multiple sequence alignment.Comment: Published in at http://dx.doi.org/10.1214/12-AAP852 the Annals of
Applied Probability (http://www.imstat.org/aap/) by the Institute of
Mathematical Statistics (http://www.imstat.org
A Spectral Algorithm for Learning Hidden Markov Models
Hidden Markov Models (HMMs) are one of the most fundamental and widely used
statistical tools for modeling discrete time series. In general, learning HMMs
from data is computationally hard (under cryptographic assumptions), and
practitioners typically resort to search heuristics which suffer from the usual
local optima issues. We prove that under a natural separation condition (bounds
on the smallest singular value of the HMM parameters), there is an efficient
and provably correct algorithm for learning HMMs. The sample complexity of the
algorithm does not explicitly depend on the number of distinct (discrete)
observations---it implicitly depends on this quantity through spectral
properties of the underlying HMM. This makes the algorithm particularly
applicable to settings with a large number of observations, such as those in
natural language processing where the space of observation is sometimes the
words in a language. The algorithm is also simple, employing only a singular
value decomposition and matrix multiplications.Comment: Published in JCSS Special Issue "Learning Theory 2009
Spectral Sequence Motif Discovery
Sequence discovery tools play a central role in several fields of
computational biology. In the framework of Transcription Factor binding
studies, motif finding algorithms of increasingly high performance are required
to process the big datasets produced by new high-throughput sequencing
technologies. Most existing algorithms are computationally demanding and often
cannot support the large size of new experimental data. We present a new motif
discovery algorithm that is built on a recent machine learning technique,
referred to as Method of Moments. Based on spectral decompositions, this method
is robust under model misspecification and is not prone to locally optimal
solutions. We obtain an algorithm that is extremely fast and designed for the
analysis of big sequencing data. In a few minutes, we can process datasets of
hundreds of thousand sequences and extract motif profiles that match those
computed by various state-of-the-art algorithms.Comment: 20 pages, 3 figures, 1 tabl
Geometric Learning of Hidden Markov Models via a Method of Moments Algorithm
We present a novel algorithm for learning the parameters of hidden Markov
models (HMMs) in a geometric setting where the observations take values in
Riemannian manifolds. In particular, we elevate a recent second-order method of
moments algorithm that incorporates non-consecutive correlations to a more
general setting where observations take place in a Riemannian symmetric space
of non-positive curvature and the observation likelihoods are Riemannian
Gaussians. The resulting algorithm decouples into a Riemannian Gaussian mixture
model estimation algorithm followed by a sequence of convex optimization
procedures. We demonstrate through examples that the learner can result in
significantly improved speed and numerical accuracy compared to existing
learners
- …