Variational Dropout and the Local Reparameterization Trick

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments

Auto-Encoding Variational Bayes

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions is two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results

Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets

Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can often be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations of the latent variables. The choice of parameterization greatly influences the efficiency of gradient-based posterior inference; we show that they are often complementary to eachother, we clarify when each parameterization is preferred and show how inference can be made robust. In the non-centered form, a simple Monte Carlo estimator of the marginal likelihood can be used for learning the parameters. Theoretical results are supported by experiments

An Introduction to Variational Autoencoders

Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to variational autoencoders and some important extensions

The 2+1 Kepler Problem and Its Quantization

We study a system of two pointlike particles coupled to three dimensional Einstein gravity. The reduced phase space can be considered as a deformed version of the phase space of two special-relativistic point particles in the centre of mass frame. When the system is quantized, we find some possibly general effects of quantum gravity, such as a minimal distances and a foaminess of the spacetime at the order of the Planck length. We also obtain a quantization of geometry, which restricts the possible asymptotic geometries of the universe.Comment: 59 pages, LaTeX2e, 9 eps figure

Broad Absorption Line Variability in Radio-Loud Quasars

We investigate C IV broad absorption line (BAL) variability within a sample of 46 radio-loud quasars (RLQs), selected from SDSS/FIRST data to include both core-dominated (39) and lobe-dominated (7) objects. The sample consists primarily of high-ionization BAL quasars, and a substantial fraction have large BAL velocities or equivalent widths; their radio luminosities and radio-loudness values span ~2.5 orders of magnitude. We have obtained 34 new Hobby-Eberly Telescope (HET) spectra of 28 BAL RLQs to compare to earlier SDSS data, and we also incorporate archival coverage (primarily dual-epoch SDSS) for a total set of 78 pairs of equivalent width measurements for 46 BAL RLQs, probing rest-frame timescales of ~80-6000 d (median 500 d). In general, only modest changes in the depths of segments of absorption troughs are observed, akin to those seen in prior studies of BAL RQQs. Also similar to previous findings for RQQs, the RLQs studied here are more likely to display BAL variability on longer rest-frame timescales. However, typical values of |Delta_EW| and |Delta_EW|/ are about 40+/-20% lower for BAL RLQs when compared with those of a timescale-matched sample of BAL RQQs. Optical continuum variability is of similar amplitude in BAL RLQs and BAL RQQs; for both RLQs and RQQs, continuum variability tends to be stronger on longer timescales. BAL variability in RLQs does not obviously depend upon their radio luminosities or radio-loudness values, but we do find tentative evidence for greater fractional BAL variability within lobe-dominated RLQs. Enhanced BAL variability within more edge-on (lobe-dominated) RLQs supports some geometrical dependence to the outflow structure.Comment: 27 pages, 16 figures, 6 tables, accepted to MNRAS, full Appendix A at http://www.macalester.edu/~bmille13/balrlqs.htm