2,780 research outputs found
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
We propose a novel method to directly learn a stochastic transition operator
whose repeated application provides generated samples. Traditional undirected
graphical models approach this problem indirectly by learning a Markov chain
model whose stationary distribution obeys detailed balance with respect to a
parameterized energy function. The energy function is then modified so the
model and data distributions match, with no guarantee on the number of steps
required for the Markov chain to converge. Moreover, the detailed balance
condition is highly restrictive: energy based models corresponding to neural
networks must have symmetric weights, unlike biological neural circuits. In
contrast, we develop a method for directly learning arbitrarily parameterized
transition operators capable of expressing non-equilibrium stationary
distributions that violate detailed balance, thereby enabling us to learn more
biologically plausible asymmetric neural networks and more general non-energy
based dynamical systems. The proposed training objective, which we derive via
principled variational methods, encourages the transition operator to "walk
back" in multi-step trajectories that start at data-points, as quickly as
possible back to the original data points. We present a series of experimental
results illustrating the soundness of the proposed approach, Variational
Walkback (VW), on the MNIST, CIFAR-10, SVHN and CelebA datasets, demonstrating
superior samples compared to earlier attempts to learn a transition operator.
We also show that although each rapid training trajectory is limited to a
finite but variable number of steps, our transition operator continues to
generate good samples well past the length of such trajectories, thereby
demonstrating the match of its non-equilibrium stationary distribution to the
data distribution. Source Code: http://github.com/anirudh9119/walkback_nips17Comment: To appear at NIPS 201
In All Likelihood, Deep Belief Is Not Enough
Statistical models of natural stimuli provide an important tool for
researchers in the fields of machine learning and computational neuroscience. A
canonical way to quantitatively assess and compare the performance of
statistical models is given by the likelihood. One class of statistical models
which has recently gained increasing popularity and has been applied to a
variety of complex data are deep belief networks. Analyses of these models,
however, have been typically limited to qualitative analyses based on samples
due to the computationally intractable nature of the model likelihood.
Motivated by these circumstances, the present article provides a consistent
estimator for the likelihood that is both computationally tractable and simple
to apply in practice. Using this estimator, a deep belief network which has
been suggested for the modeling of natural image patches is quantitatively
investigated and compared to other models of natural image patches. Contrary to
earlier claims based on qualitative results, the results presented in this
article provide evidence that the model under investigation is not a
particularly good model for natural image
Nested sampling for Potts models
Nested sampling is a new Monte Carlo method by Skilling [1] intended for general Bayesian computation. Nested sampling provides a robust alternative to annealing-based methods for computing normalizing constants. It can also generate estimates of other quantities such as posterior expectations. The key technical requirement is an ability to draw samples uniformly from the prior subject to a constraint on the likelihood. We provide a demonstration with the Potts model, an undirected graphical model
- ā¦