Orthogonally Decoupled Variational Gaussian Processes

Gaussian processes (GPs) provide a powerful non-parametric framework for reasoning over functions. Despite appealing theory, its superlinear computational and memory complexities have presented a long-standing challenge. State-of-the-art sparse variational inference methods trade modeling accuracy against complexity. However, the complexities of these methods still scale superlinearly in the number of basis functions, implying that that sparse GP methods are able to learn from large datasets only when a small model is used. Recently, a decoupled approach was proposed that removes the unnecessary coupling between the complexities of modeling the mean and the covariance functions of a GP. It achieves a linear complexity in the number of mean parameters, so an expressive posterior mean function can be modeled. While promising, this approach suffers from optimization difficulties due to ill-conditioning and non-convexity. In this work, we propose an alternative decoupled parametrization. It adopts an orthogonal basis in the mean function to model the residues that cannot be learned by the standard coupled approach. Therefore, our method extends, rather than replaces, the coupled approach to achieve strictly better performance. This construction admits a straightforward natural gradient update rule, so the structure of the information manifold that is lost during decoupling can be leveraged to speed up learning. Empirically, our algorithm demonstrates significantly faster convergence in multiple experiments.Comment: Appearing NIPS 201

Quenched Computation of the Complexity of the Sherrington-Kirkpatrick Model

The quenched computation of the complexity in the Sherrington-Kirkpatrick model is presented. A modified Full Replica Symmetry Breaking Ansatz is introduced in order to study the complexity dependence on the free energy. Such an Ansatz corresponds to require Becchi-Rouet-Stora-Tyutin supersymmetry. The complexity computed this way is the Legendre transform of the free energy averaged over the quenched disorder. The stability analysis shows that this complexity is inconsistent at any free energy level but the equilibirum one. The further problem of building a physically well defined solution not invariant under supersymmetry and predicting an extensive number of metastable states is also discussed.Comment: 19 pages, 13 figures. Some formulas added corrected, changes in discussion and conclusion, one figure adde

Complexity of the Sherrington-Kirkpatrick Model in the Annealed Approximation

A careful critical analysis of the complexity, at the annealed level, of the Sherrington-Kirkpatrick model has been performed. The complexity functional is proved to be always invariant under the Becchi-Rouet-Stora-Tyutin supersymmetry, disregarding the formulation used to define it. We consider two different saddle points of such functional, one satisfying the supersymmetry [A. Cavagna {\it et al.}, J. Phys. A {\bf 36} (2003) 1175] and the other one breaking it [A.J. Bray and M.A. Moore, J. Phys. C {\bf 13} (1980) L469]. We review the previews studies on the subject, linking different perspectives and pointing out some inadequacies and even inconsistencies in both solutions.Comment: 20 pages, 4 figure