Dreaming neural networks: forgetting spurious memories and reinforcing pure ones

The standard Hopfield model for associative neural networks accounts for biological Hebbian learning and acts as the harmonic oscillator for pattern recognition, however its maximal storage capacity is $\alpha \sim 0.14$, far from the theoretical bound for symmetric networks, i.e. $\alpha =1$. Inspired by sleeping and dreaming mechanisms in mammal brains, we propose an extension of this model displaying the standard on-line (awake) learning mechanism (that allows the storage of external information in terms of patterns) and an off-line (sleep) unlearning$\&$consolidating mechanism (that allows spurious-pattern removal and pure-pattern reinforcement): this obtained daily prescription is able to saturate the theoretical bound $\alpha=1$, remaining also extremely robust against thermal noise. Both neural and synaptic features are analyzed both analytically and numerically. In particular, beyond obtaining a phase diagram for neural dynamics, we focus on synaptic plasticity and we give explicit prescriptions on the temporal evolution of the synaptic matrix. We analytically prove that our algorithm makes the Hebbian kernel converge with high probability to the projection matrix built over the pure stored patterns. Furthermore, we obtain a sharp and explicit estimate for the "sleep rate" in order to ensure such a convergence. Finally, we run extensive numerical simulations (mainly Monte Carlo sampling) to check the approximations underlying the analytical investigations (e.g., we developed the whole theory at the so called replica-symmetric level, as standard in the Amit-Gutfreund-Sompolinsky reference framework) and possible finite-size effects, finding overall full agreement with the theory.Comment: 31 pages, 12 figure

Booleovská faktorová analýza atraktorovou neuronovou sítí

Import 23/08/2017Methods for the discovery of hidden structures of high-dimensional binary data rank among the most important challenges facing the community of machine learning researchers at present. There are many approaches in the literature that try to solve this hitherto rather ill-defined task. The Boolean factor analysis (BFA) studied in this work represents a hidden structure of binary data as Boolean superposition of binary factors complied with the BFA generative model of signals, and the criterion of optimality of BFA solution is given. In these terms, the BFA is a well-defined task completely analogous to linear factor analysis. The main contributions of the dissertation thesis are as follows: Firstly, an efficient BFA method, based on the original attractor neural network with increasing activity (ANNIA), which is subsequently improved through a combination with the expectation-maximization method(EM),so LANNIA method has been developed. Secondly, the characteristics of the ANNIA that are important for LANNIA and ANNIA methods functioning were analyzed. Then the functioning of both methods was validated on artificially generated data sets. Next, the method was applied to real-world data from different areas of science to demonstrate their contribution to this type of analysis. Finally, the BFA method was compared with related methods, including applicability analysis.Jednou z nejdůležitějších výzev současnosti, která stojí před komunitou badatelů z oblasti strojového učení je výzkum metod pro analýzu vysoce-dimenzionálních binárních dat s cílem odhalení jejich skryté struktury. V literatuře můžeme nalézt mnoho přístupů, které se snaží tuto doposud poněkud vágně definovanou úlohu řešit. Booleovská Faktorová Analýza (BFA), jež je předmětem této práce, předpokládá, že skrytou strukturu binárních dat lze reprezentovat jako booleovskou superpozici binárních faktorů tak, aby co nejlépe odpovídala generativnímu modelu signálů BFA a danému kritériu optimálnosti. Za těchto podmínek je BFA dob��e definovaná úloha zcela analogická lineární faktorové analýze. Hlavní přínosy disertační práce, jsou následující: Za prvé byl vyvinut efektivní způsob BFA založený na původní atraktorové neuronové síti s rostoucí aktivitou (ANNIA), která byla následně zlepšena kombinací s metodou expectation–maximization (EM)a tak vytvo5ena metoda LANNIA. Dále byly provedeny analýzy charakteristik ANNIA, které jsou důležité pro fungování obou metod. Funkčnost obou metod byla také ověřena na uměle vytvořených souborech dat pokrývajících celou škálu parametrů generativního modelu. Dále je v práci ukázáno použití metod na reálných datech z různých oblastí vědy s cílem prokázat jejich přínos pro tento typ analýzy. A konečně bylo provedeno i srovnání metod BFA se podobnými metodami včetně analýzy jejich použitelnosti.460 - Katedra informatikyvyhově

Neural Distributed Autoassociative Memories: A Survey

Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension. The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons). Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints. Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors.Comment: 31 page

A study of the design and analysis of feed-forward neural networks

This thesis shows that a design and analysis system for feed forward neural networks is desirable, and that the currently available techniques do not work. Methods have been presented that solve the problem of analysis, showing that analysis is possible and desirable for classification networks. The biggest limitation is the size of the network and that the analysis tools are only applicable to properly designed classification systems. A method of reducing the size of classification networks is presented along with a design methodology for non classification systems