Emergence of Invariance and Disentanglement in Deep Representations

Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then decompose the cross-entropy loss used during training and highlight the presence of an inherent overfitting term. We propose regularizing the loss by bounding such a term in two equivalent ways: One with a Kullbach-Leibler term, which relates to a PAC-Bayes perspective; the other using the information in the weights as a measure of complexity of a learned model, yielding a novel Information Bottleneck for the weights. Finally, we show that invariance and independence of the components of the representation learned by the network are bounded above and below by the information in the weights, and therefore are implicitly optimized during training. The theory enables us to quantify and predict sharp phase transitions between underfitting and overfitting of random labels when using our regularized loss, which we verify in experiments, and sheds light on the relation between the geometry of the loss function, invariance properties of the learned representation, and generalization error.Comment: Deep learning, neural network, representation, flat minima, information bottleneck, overfitting, generalization, sufficiency, minimality, sensitivity, information complexity, stochastic gradient descent, regularization, total correlation, PAC-Baye

Generalized Holographic Principle, Gauge Invariance and the Emergence of Gravity a la Wilczek

We show that a generalized version of the holographic principle can be derived from the Hamiltonian description of information flow within a quantum system that maintains a separable state. We then show that this generalized holographic principle entails a general principle of gauge invariance. When this is realized in an ambient Lorentzian space-time, gauge invariance under the Poincare group is immediately achieved. We apply this pathway to retrieve the action of gravity. The latter is cast a la Wilczek through a similar formulation derived by MacDowell and Mansouri, which involves the representation theory of the Lie groups SO(3,2) and SO(4,1).Comment: 26 pages, 1 figur

Continued Fraction Representation of Temporal Multi Scaling in Turbulence

It was shown recently that the anomalous scaling of simultaneous correlation functions in turbulence is intimately related to the breaking of temporal scale invariance, which is equivalent to the appearance of infinitely many times scales in the time dependence of time-correlation functions. In this paper we derive a continued fraction representation of turbulent time correlation functions which is exact and in which the multiplicity of time scales is explicit. We demonstrate that this form yields precisely the same scaling laws for time derivatives and time integrals as the "multi-fractal" representation that was used before. Truncating the continued fraction representation yields the "best" estimates of time correlation functions if the given information is limited to the scaling exponents of the simultaneous correlation functions up to a certain, finite order. It is worth noting that the derivation of a continued fraction representation obtained here for an operator which is not Hermitian or anti-Hermitian may be of independent interest.Comment: 7 pages, no figur

Localization in the Rindler Wedge

One of the striking features of QED is that charged particles create a coherent cloud of photons. The resultant coherent state vectors of photons generate a non-trivial representation of the localized algebra of observables that do not support a representation of the Lorentz group: Lorentz symmetry is spontaneously broken. We show in particular that Lorentz boost generators diverge in this representation, a result shown also in [1] (See also [2]). Localization of observables, for example in the Rindler wedge, uses Poincar\'e invariance in an essential way [3]. Hence in the presence of charged fields, the photon observables cannot be localized in the Rindler wedge. These observations may have a bearing on the black hole information loss paradox, as the physics in the exterior of the black hole has points of resemblance to that in the Rindler wedge.Comment: 11 page

Structure of Probabilistic Information and Quantum Laws

In quantum experiments the acquisition and representation of basic experimental information is governed by the multinomial probability distribution. There exist unique random variables, whose standard deviation becomes asymptotically invariant of physical conditions. Representing all information by means of such random variables gives the quantum mechanical probability amplitude and a real alternative. For predictions, the linear evolution law (Schrodinger or Dirac equation) turns out to be the only way to extend the invariance property of the standard deviation to the predicted quantities. This indicates that quantum theory originates in the structure of gaining pure, probabilistic information, without any mechanical underpinning.Comment: RevTex, 6 pages incl. 2 figures. Contribution to conference "Foundations of Probability and Physics", Vaxjo, Sweden, 27 Nov. - 1 Dec. 200

Effective QED Actions: Representations, Gauge Invariance, Anomalies and Mass Expansions

We analyze and give explicit representations for the effective abelian vector gauge field actions generated by charged fermions with particular attention to the thermal regime in odd dimensions, where spectral asymmetry can be present. We show, through $\zeta-$function regularization, that both small and large gauge invariances are preserved at any temperature and for any number of fermions at the usual price of anomalies: helicity/parity invariance will be lost in even/odd dimensions, and in the latter even at zero mass. Gauge invariance dictates a very general ``Fourier'' representation of the action in terms of the holonomies that carry the novel, large gauge invariant, information. We show that large (unlike small) transformations and hence their Ward identities, are not perturbative order-preserving, and clarify the role of (properly redefined) Chern-Simons terms in this context. From a powerful representation of the action in terms of massless heat kernels, we are able to obtain rigorous gauge invariant expansions, for both small and large fermion masses, of its separate parity even and odd parts in arbitrary dimension. The representation also displays both the nonperturbative origin of a finite renormalization ambiguity, and its physical resolution by requiring decoupling at infinite mass. Finally, we illustrate these general results by explicit computation of the effective action for some physical examples of field configurations in the three dimensional case, where our conclusions on finite temperature effects may have physical relevance. Nonabelian results will be presented separately.Comment: 36 pages, RevTeX, no figure