Transferring Learning from External to Internal Weights in Echo-State Networks with Sparse Connectivity

Modifying weights within a recurrent network to improve performance on a task has proven to be difficult. Echo-state networks in which modification is restricted to the weights of connections onto network outputs provide an easier alternative, but at the expense of modifying the typically sparse architecture of the network by including feedback from the output back into the network. We derive methods for using the values of the output weights from a trained echo-state network to set recurrent weights within the network. The result of this “transfer of learning” is a recurrent network that performs the task without requiring the output feedback present in the original network. We also discuss a hybrid version in which online learning is applied to both output and recurrent weights. Both approaches provide efficient ways of training recurrent networks to perform complex tasks. Through an analysis of the conditions required to make transfer of learning work, we define the concept of a “self-sensing” network state, and we compare and contrast this with compressed sensing

A hierarchy of recurrent networks for speech recognition

Generative models for sequential data based on directed graphs of Restricted Boltzmann Machines (RBMs) are able to accurately model high dimensional sequences as recently shown. In these models, temporal dependencies in the input are discovered by either buffering previous visible variables or by recurrent connections of the hidden variables. Here we propose a modification of these models, the Temporal Reservoir Machine (TRM). It utilizes a recurrent artificial neural network (ANN) for integrating information from the input over time. This information is then fed into a RBM at each time step. To avoid difficulties of recurrent network learning, the ANN remains untrained and hence can be thought of as a random feature extractor. Using the architecture of multi-layer RBMs (Deep Belief Networks), the TRMs can be used as a building block for complex hierarchical models. This approach unifies RBM-based approaches for sequential data modeling and the Echo State Network, a powerful approach for black-box system identification. The TRM is tested on a spoken digits task under noisy conditions, and competitive performances compared to previous models are observed

Echo State Condition at the Critical Point

Recurrent networks with transfer functions that fulfill the Lipschitz continuity with K=1 may be echo state networks if certain limitations on the recurrent connectivity are applied. It has been shown that it is sufficient if the largest singular value of the recurrent connectivity is smaller than 1. The main achievement of this paper is a proof under which conditions the network is an echo state network even if the largest singular value is one. It turns out that in this critical case the exact shape of the transfer function plays a decisive role in determining whether the network still fulfills the echo state condition. In addition, several examples with one neuron networks are outlined to illustrate effects of critical connectivity. Moreover, within the manuscript a mathematical definition for a critical echo state network is suggested

Enhanced surface interaction of water confined in hierarchical porous polymers induced by hydrogen bonding

Hierarchical porous polymer systems are increasingly applied to catalysis, bioengineering, or separation technology because of the versatility provided by the connection of mesopores with percolating macroporous structures. Nuclear magnetic resonance (NMR) is a suitable technique for the study of such systems as it can detect signals stemming from the confined liquid and translate this information into pore size, molecular mobility, and liquid−surface interactions. We focus on the properties of water confined in macroporous polymers of ethylene glycol dimethacrylate and 2-hydroxyethyl methacrylate [poly- (EGDMA-co-HEMA)] with different amounts of cross-linkers, in which a substantial variation of hydroxyl groups is achieved. As soft polymer scaffolds may swell upon saturation with determined liquids, the use of NMR is particularly important as it measures the system in its operational state. This study combines different NMR techniques to obtain information on surface interactions of water with hydrophilic polymer chains. A transition from a surface-induced relaxation in which relaxivity depends on the pore size to a regime where the organic pore surface strongly restricts water diffusion is observed. Surface affinities are defined through the molecular residence times near the network surface.Fil: Silletta, Emilia Victoria. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto de Física Enrique Gaviola. Universidad Nacional de Córdoba. Instituto de Física Enrique Gaviola; ArgentinaFil: Velasco, Manuel Isaac. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto de Física Enrique Gaviola. Universidad Nacional de Córdoba. Instituto de Física Enrique Gaviola; ArgentinaFil: Gomez, Cesar Gerardo. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Strumia, Miriam Cristina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Stapf, Siegfried. Technische Universität Ilmenau; AlemaniaFil: Mattea, Carlos. Technische Universität Ilmenau; AlemaniaFil: Monti, Gustavo Alberto. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto de Física Enrique Gaviola. Universidad Nacional de Córdoba. Instituto de Física Enrique Gaviola; ArgentinaFil: Acosta, Rodolfo Hector. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto de Física Enrique Gaviola. Universidad Nacional de Córdoba. Instituto de Física Enrique Gaviola; Argentin