Variance Reduced Stochastic Gradient Descent with Neighbors

Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a computational bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.Comment: Appears in: Advances in Neural Information Processing Systems 28 (NIPS 2015). 13 page

Barrier Frank-Wolfe for Marginal Inference

We introduce a globally-convergent algorithm for optimizing the tree-reweighted (TRW) variational objective over the marginal polytope. The algorithm is based on the conditional gradient method (Frank-Wolfe) and moves pseudomarginals within the marginal polytope through repeated maximum a posteriori (MAP) calls. This modular structure enables us to leverage black-box MAP solvers (both exact and approximate) for variational inference, and obtains more accurate results than tree-reweighted algorithms that optimize over the local consistency relaxation. Theoretically, we bound the sub-optimality for the proposed algorithm despite the TRW objective having unbounded gradients at the boundary of the marginal polytope. Empirically, we demonstrate the increased quality of results found by tightening the relaxation over the marginal polytope as well as the spanning tree polytope on synthetic and real-world instances.Comment: 25 pages, 12 figures, To appear in Neural Information Processing Systems (NIPS) 2015, Corrected reference and cleaned up bibliograph

Information, information processing and gravity

I discuss fundamental limits placed on information and information processing by gravity. Such limits arise because both information and its processing require energy, while gravitational collapse (formation of a horizon or black hole) restricts the amount of energy allowed in a finite region. Specifically, I use a criterion for gravitational collapse called the hoop conjecture. Once the hoop conjecture is assumed a number of results can be obtained directly: the existence of a fundamental uncertainty in spatial distance of order the Planck length, bounds on information (entropy) in a finite region, and a bound on the rate of information processing in a finite region. In the final section I discuss some cosmological issues related to the total amount of information in the universe, and note that almost all detailed aspects of the late universe are determined by the randomness of quantum outcomes. This paper is based on a talk presented at a 2007 Bellairs Research Institute (McGill University) workshop on black holes and quantum information.Comment: 7 pages, 5 figures, revte

Hybrid quantum information processing

The development of quantum information processing has traditionally followed two separate and not immediately connected lines of study. The main line has focused on the implementation of quantum bit (qubit) based protocols whereas the other line has been devoted to implementations based on high-dimensional Gaussian states (such as coherent and squeezed states). The separation has been driven by the experimental difficulty in interconnecting the standard technologies of the two lines. However, in recent years, there has been a significant experimental progress in refining and connecting the technologies of the two fields which has resulted in the development and experimental realization of numerous new hybrid protocols. In this Review, we summarize these recent efforts on hybridizing the two types of schemes based on discrete and continuous variables.Comment: 13 pages, 6 figure

Metabolically Efficient Information Processing

Energy efficient information transmission may be relevant to biological sensory signal processing as well as to low power electronic devices. We explore its consequences in two different regimes. In an ``immediate'' regime, we argue that the information rate should be maximized subject to a power constraint, while in an ``exploratory'' regime, the transmission rate per power cost should be maximized. In the absence of noise, discrete inputs are optimally encoded into Boltzmann distributed output symbols. In the exploratory regime, the partition function of this distribution is numerically equal to 1. The structure of the optimal code is strongly affected by noise in the transmission channel. The Arimoto-Blahut algorithm, generalized for cost constraints, can be used to derive and interpret the distribution of symbols for optimal energy efficient coding in the presence of noise. We outline the possibilities and problems in extending our results to information coding and transmission in neurobiological systems.Comment: LaTeX, 15 pages, 4 separate Postscript figure

Physics as Information Processing

I review some recent advances in foundational research at Pavia QUIT group. The general idea is that there is only Quantum Theory without quantization rules, and the whole Physics---including space-time and relativity--is emergent from the quantum-information processing. And since Quantum Theory itself is axiomatized solely on informational principles, the whole Physics must be reformulated in information-theoretical terms: this is the "It from Bit of J. A. Wheeler. The review is divided into four parts: a) the informational axiomatization of Quantum Theory; b) how space-time and relativistic covariance emerge from quantum computation; c) what is the information-theoretical meaning of inertial mass and of $\hbar$, and how the quantum field emerges; d) an observational consequence of the new quantum field theory: a mass-dependent refraction index of vacuum. I will conclude with the research lines that will follow in the immediate future.Comment: Work presented at the conference "Advances in Quantum Theory" held on 14-17 June 2010 at the Linnaeus University, Vaxjo, Swede

Efficient optical quantum information processing

Quantum information offers the promise of being able to perform certain communication and computation tasks that cannot be done with conventional information technology (IT). Optical Quantum Information Processing (QIP) holds particular appeal, since it offers the prospect of communicating and computing with the same type of qubit. Linear optical techniques have been shown to be scalable, but the corresponding quantum computing circuits need many auxiliary resources. Here we present an alternative approach to optical QIP, based on the use of weak cross-Kerr nonlinearities and homodyne measurements. We show how this approach provides the fundamental building blocks for highly efficient non-absorbing single photon number resolving detectors, two qubit parity detectors, Bell state measurements and finally near deterministic control-not (CNOT) gates. These are essential QIP devicesComment: Accepted to the Journal of optics B special issue on optical quantum computation; References update