SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method.Comment: Advances In Neural Information Processing Systems, Nov 2014, Montreal, Canad

Rethinking LDA: moment matching for discrete ICA

We consider moment matching techniques for estimation in Latent Dirichlet Allocation (LDA). By drawing explicit links between LDA and discrete versions of independent component analysis (ICA), we first derive a new set of cumulant-based tensors, with an improved sample complexity. Moreover, we reuse standard ICA techniques such as joint diagonalization of tensors to improve over existing methods based on the tensor power method. In an extensive set of experiments on both synthetic and real datasets, we show that our new combination of tensors and orthogonal joint diagonalization techniques outperforms existing moment matching methods.Comment: 30 pages; added plate diagrams and clarifications, changed style, corrected typos, updated figures. in Proceedings of the 29-th Conference on Neural Information Processing Systems (NIPS), 201

On the Equivalence between Herding and Conditional Gradient Algorithms

We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for the task of approximating integrals in a reproducing kernel Hilbert space. We study the behavior of the different variants through numerical simulations. The experiments indicate that while we can improve over herding on the task of approximating integrals, the original herding algorithm tends to approach more often the maximum entropy distribution, shedding more light on the learning bias behind herding

Sequential Kernel Herding: Frank-Wolfe Optimization for Particle Filtering

Recently, the Frank-Wolfe optimization algorithm was suggested as a procedure to obtain adaptive quadrature rules for integrals of functions in a reproducing kernel Hilbert space (RKHS) with a potentially faster rate of convergence than Monte Carlo integration (and "kernel herding" was shown to be a special case of this procedure). In this paper, we propose to replace the random sampling step in a particle filter by Frank-Wolfe optimization. By optimizing the position of the particles, we can obtain better accuracy than random or quasi-Monte Carlo sampling. In applications where the evaluation of the emission probabilities is expensive (such as in robot localization), the additional computational cost to generate the particles through optimization can be justified. Experiments on standard synthetic examples as well as on a robot localization task indicate indeed an improvement of accuracy over random and quasi-Monte Carlo sampling.Comment: in 18th International Conference on Artificial Intelligence and Statistics (AISTATS), May 2015, San Diego, United States. 38, JMLR Workshop and Conference Proceeding

A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.Comment: 8 pages, 6 figures. Changes with previous version: Added reference to concurrently submitted work arXiv:1212.1824v1; clarifications added; typos corrected; title changed to 'subgradient method' as 'subgradient descent' is misnome

PAC-Bayesian Theory Meets Bayesian Inference

We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by an i.i.d distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.Comment: Published at NIPS 2015 (http://papers.nips.cc/paper/6569-pac-bayesian-theory-meets-bayesian-inference

Ground States in the Spin Boson Model

We prove that the Hamiltonian of the model describing a spin which is linearly coupled to a field of relativistic and massless bosons, also known as the spin-boson model, admits a ground state for small values of the coupling constant lambda. We show that the ground state energy is an analytic function of lambda and that the corresponding ground state can also be chosen to be an analytic function of lambda. No infrared regularization is imposed. Our proof is based on a modified version of the BFS operator theoretic renormalization analysis. Moreover, using a positivity argument we prove that the ground state of the spin-boson model is unique. We show that the expansion coefficients of the ground state and the ground state energy can be calculated using regular analytic perturbation theory

Forced to go virtual. Working-from-home arrangements and their effect on team communication during COVID-19 lockdown

Working-from-home arrangements have become increasingly important for firms’ work organization. In this context, the COVID-19 pandemic has led to teams that previously did not work virtually being forced to interact and communicate virtually. In this study, we analyze changes in intra-team communication of four teams in a German medium-sized enterprise. Quantitative network analyses of email communication and qualitative analyses of interviews before and during the COVID-19 lockdown in spring 2020 show that flat hierarchies and self-managing processes helped team members to mitigate negative effects due to spatial and temporal dispersion in forced working-from-home arrangements. Moreover, analysis of the teams’ communication networks shows that forced remote work can trigger faultlines to become salient but that team cohesion, identification with the team, and individuals taking on broker roles prevent negative effects of faultlines on team performance. In discussing these findings, our study contributes to the research on coordination and communication in virtual teams by analyzing contextual, organizational, team-related as well as individual factors that explain how and why teams differ in successfully implementing working-from-home arrangements

Creative industries and the IPR dilemma between appropriation and creation: some insights from the videogame and music industries

La propriété intellectuelle (PI) joue un rôle stratégique dans les industries créatives où la créativité est un processus collectif impliquant des acteurs aux intérêts contradictoires, conduisant à un “dilemne de la PI”. Les firmes veulent s’approprier le travail créatif et lutter contre l’imitation; les communautés créatives souhaitent un régime de PI souple pour recombiner les créations passées et générer des nouveautés; les individus sont entre ces deux extrêmes. Des arrangements spécifiques sont alors développés (comme des pratiques d’open source ou de creative commons) pour concilier appropriation et création. Les industries de la musique et des jeux vidéo illustrent ces phénomènes.Intellectual property rights (IPR) play a strategic role in creative industries. Defined as a collective process, creativity involves actors with contradictory IPR needs. This leads to an “IPR dilemna”. Firms are looking into appropriating creative work and prevent imitation; whereas creative communities need a weak IPR to combine past work and generate novelty. It becomes problematic for individuals to find themselves between these two. As a result, actors are developing specific IPR arrangements (e.g. open source and creative commons practices) to preserve the balance between appropriation and openness allowing creation. Two creative industries are used as illustrations: music and video-games.Los derechos propiedad intelectual (DPI) juegan un rol estratégico en las industrias creativas definidas por un proceso colectivo que involucra diferentes actores cuyos intereses en los DPI son contradictorios. Mientras las firmas buscan apropiarse su trabajo creativo y prevenir la imitación, las comunidades creativas necesitan DPI débiles para poder combinar trabajos pasados y generar novedades. Por lo tanto actores encuentran dificultades para identificarse con una de estas categorías. En consecuencia, estos desarrollan acuerdos específicos de DPI para preservar un equilibrio entre apropiación y apertura que les permita crear. Dos industrias creativas ilustran un ejemplo: la música y los video juegos