Smoothed Gradients for Stochastic Variational Inference

Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.Comment: Appears in Neural Information Processing Systems, 201

Transport and Non-Equilibrium Dynamics in Optical Lattices. From Expanding Atomic Clouds to Negative Absolute Temperatures

Transport properties and nonequilibrium dynamics in strongly correlated materials are typically difficult to calculate. This holds true even for minimalistic model Hamiltonians of these systems, such as the fermionic Hubbard model. Ultracold atoms in optical lattices enable an alternative realization of the Hubbard model and have the advantage of being free of additional complications such as phonons, lattice defects or impurities. This way, cold atoms can be used as quantum simulators of strongly interacting materials. Being thermally isolated systems, however, we show that cold atoms in optical lattices can also behave very differently from solids and can show a plethora of novel dynamic effects. In this thesis, several out-of equilibrium processes involving interacting fermionic atoms in optical lattices are presented. We first analyze the expansion dynamics of an initially confined atomic cloud in the lowest band of an optical lattice. While non-interacting atoms expand ballistically, the cloud expands with a dramatically reduced velocity in the presence of interactions. Most prominently, the expansion velocity is independent of the attractive or repulsive character of the interactions, highlighting a novel dynamic symmetry of the Hubbard model. In a second project, we discuss the possibility of realizing negative absolute temperatures in optical lattices. Negative absolute temperatures characterize equilibrium states with an inverted occupation of energy levels. Here, we propose a dynamical process to realize equilibrated Fermions at negative temperatures and analyze the time scales of global relaxation to equilibrium, which are associated with a redistribution of energy and particles by slow diffusive processes. We show that energy conservation has a major impact on the dynamics of an interacting cloud in an optical lattice, which is exposed to an additional weak linear (gravitational) potential. Instead of ‘falling downwards‘, the cloud diffuses symmetrically upwards and downwards in the gravitational potential. Furthermore, we show analytically that the radius R grows with the time t according to R ∼ t^1/3, consistent with numerical simulations of the Boltzmann equation. Finally, we analyze the damping of Bloch oscillations by interactions. For a homogeneous system, we discuss the possibility of mapping the dynamics of the particle current to a classical damped harmonic oscillator equation, thereby giving an analytic explanation for the transition from weakly damped to over-damped Bloch oscillations. We show that the dynamics of a strongly Bloch oscillating and weakly interacting atomic cloud can be discribed in terms of a novel effective “stroboscopic” diffusion equation. In this approximation, the cloud’s radius R grows asymptotically in time t according to R ∼ t^1/5

Interacting Fermionic Atoms in Optical Lattices Diffuse Symmetrically Upwards and Downwards in a Gravitational Potential

We consider a cloud of fermionic atoms in an optical lattice described by a Hubbard model with an additional linear potential. While homogeneous interacting systems mainly show damped Bloch oscillations and heating, a finite cloud behaves differently: It expands symmetrically such that gains of potential energy at the top are compensated by losses at the bottom. Interactions stabilize the necessary heat currents by inducing gradients of the inverse temperature 1/T, with T<0 at the bottom of the cloud. An analytic solution of hydrodynamic equations shows that the width of the cloud increases with t^(1/3) for long times consistent with results from our Boltzmann simulations.Comment: 4 pages, 4 figures plus supplementary material (2 pages, 1 figure), published versio

Iterative Amortized Inference

Inference models are a key component in scaling variational inference to deep latent variable models, most notably as encoder networks in variational auto-encoders (VAEs). By replacing conventional optimization-based inference with a learned model, inference is amortized over data examples and therefore more computationally efficient. However, standard inference models are restricted to direct mappings from data to approximate posterior estimates. The failure of these models to reach fully optimized approximate posterior estimates results in an amortization gap. We aim toward closing this gap by proposing iterative inference models, which learn to perform inference optimization through repeatedly encoding gradients. Our approach generalizes standard inference models in VAEs and provides insight into several empirical findings, including top-down inference techniques. We demonstrate the inference optimization capabilities of iterative inference models and show that they outperform standard inference models on several benchmark data sets of images and text.Comment: International Conference on Machine Learning (ICML) 201

Equilibration rates and negative absolute temperatures for ultracold atoms in optical lattices

As highly tunable interacting systems, cold atoms in optical lattices are ideal to realize and observe negative absolute temperatures, T < 0. We show theoretically that by reversing the confining potential, stable superfluid condensates at finite momentum and T < 0 can be created with low entropy production for attractive bosons. They may serve as `smoking gun' signatures of equilibrated T < 0. For fermions, we analyze the time scales needed to equilibrate to T < 0. For moderate interactions, the equilibration time is proportional to the square of the radius of the cloud and grows with increasing interaction strengths as atoms and energy are transported by diffusive processes.Comment: published version, minor change