Bio-inspired, task-free continual learning through activity regularization

The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the application of CL methods in the real world where tasks are not always well defined. Here, we take inspiration from neuroscience, where sparse, non-overlapping neuronal representations have been suggested to prevent catastrophic forgetting. As in the brain, we argue that these sparse representations should be chosen on the basis of feed forward (stimulus-specific) as well as top-down (context-specific) information. To implement such selective sparsity, we use a bio-plausible form of hierarchical credit assignment known as Deep Feedback Control (DFC) and combine it with a winner-take-all sparsity mechanism. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries. Overall, we showcase the idea of adopting computational principles from the brain to derive new, task-free learning algorithms for CL

Generalising the staircase models

Systems of integral equations are proposed which generalise those previously encountered in connection with the so-called staircase models. Under the assumption that these equations describe the finite-size effects of relativistic field theories via the Thermodynamic Bethe Ansatz, analytical and numerical evidence is given for the existence of a variety of new roaming renormalisation group trajectories. For each positive integer $k$ and $s=0,\dots, k-1$, there is a one-parameter family of trajectories, passing close by the coset conformal field theories $G^{(k)}\times G^{(nk+s)}/G^{((n+1)k+s)}$ before finally flowing to a massive theory for $s=0$, or to another coset model for $s \neq 0$.Comment: 19 pages (and two figures), preprint CERN-TH.6739/92 NI92009 DFUB-92-2

Nature of the deconfining phase transition in the 2+1-dimensional SU(N) Georgi-Glashow model

The nature of the deconfining phase transition in the 2+1-dimensional SU(N) Georgi-Glashow model is investigated. Within the dimensional-reduction hypothesis, the properties of the transition are described by a two-dimensional vectorial Coulomb gas models of electric and magnetic charges. The resulting critical properties are governed by a generalized SU(N) sine-Gordon model with self-dual symmetry. We show that this model displays a massless flow to an infrared fixed point which corresponds to the Z$\_N$ parafermions conformal field theory. This result, in turn, supports the conjecture of Kogan, Tekin, and Kovner that the deconfining transition in the 2+1-dimensional SU(N) Georgi-Glashow model belongs to the Z$\_N$ universality class.Comment: 12 page

All order I.R. finite expansion for short distance behavior of massless theories perturbed by a relevant operator

We consider here renormalizable theories without relevant couplings and present an I.R. consistent technique to study corrections to short distance behavior (Wilson O.P.E. coefficients) due to a relevant perturbation. Our method is the result of a complete reformulation of recent works on the field, and is characterized by a more orthodox treatment of U.V. divergences that allows for simpler formulae and consequently an explicit all order (regularization invariant) I.R. finitess proof. Underlying hypotheses are discussed in detail and found to be satisfied in conformal theories that constitute a natural field of application of this approach.Comment: 27 page

Moduli space coordinates and excited state g-functions

We consider the space of boundary conditions of Virasoro minimal models formed from the composition of a collection of flows generated by \phi_{1,3}. These have recently been shown to fall naturally into a sequence, each term having a coordinate on it in terms of a boundary parameter, but no global parameter has been proposed. Here we investigate the idea that the overlaps of particular bulk states with the boundary states give natural coordinates on the moduli space of boundary conditions. We find formulae for these overlaps using the known thermodynamic Bethe Ansatz descriptions of the ground and first excited state on the cylinder and show that they give a global coordinate on the space of boundary conditions, showing it is smooth and compact as expected.Comment: 10 pages, 4 figure

Bio-inspired, task-free continual learning through activity regularization

The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the application of CL methods in the real world where tasks are not always well defined. Here, we take inspiration from neuroscience, where sparse, non-overlapping neuronal representations have been suggested to prevent catastrophic forgetting. As in the brain, we argue that these sparse representations should be chosen on the basis of feed forward (stimulus-specific) as well as top-down (context-specific) information. To implement such selective sparsity, we use a bio-plausible form of hierarchical credit assignment known as Deep Feedback Control (DFC) and combine it with a winner-take-all sparsity mechanism. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries. Overall, we showcase the idea of adopting computational principles from the brain to derive new, task-free learning algorithms for CL.ISSN:0340-1200ISSN:1432-077