Learning Motor Primitives with Reinforcement Learning

One of the major challenges in action generation for robotics and in the understanding of human motor control is to learn the "building blocks of move- ment generation," or more precisely, motor primitives. Recently, Ijspeert et al. [1, 2] suggested a novel framework how to use nonlinear dynamical systems as motor primitives. While a lot of progress has been made in teaching these mo- tor primitives using supervised or imitation learning, the self-improvement by interaction of the system with the environment remains a challenging problem. In this poster, we evaluate different reinforcement learning approaches can be used in order to improve the performance of motor primitives. For pursuing this goal, we highlight the difficulties with current reinforcement learning methods, and line out how these lead to a novel algorithm which is based on natural policy gradients [3]. We compare this algorithm to previous reinforcement learning algorithms in the context of dynamic motor primitive learning, and show that it outperforms these by at least an order of magnitude. We demonstrate the efficiency of the resulting reinforcement learning method for creating complex behaviors for automous robotics. The studied behaviors will include both discrete, finite tasks such as baseball swings, as well as complex rhythmic patterns as they occur in biped locomotion

Motion deficits in dyslexia are restricted to high external noise displays

Studies of motion perception in dyslexia have usually used random dot kinetograms with high external noise. Is the reported motion deficit in dyslexia due to deficiencies in motion perception per se, or due to deficiencies in excluding noise in the displays? In this study, we compared the motion perception thresholds of both dyslexic and nondyslexic children, and dyslexic and non-dyslexic adults using first-order coherent motion displays that varied in noise level and signal salience. Both dyslexic children and adults had higher motion thresholds than non-dyslexic children and adults when the task involved first-order motion processing in high noise. Dyslexics performed as well as non-dyslexics, however, when the signal was clearly separated from the noise or noise was reduced. Thus dyslexics appear to have normal motion perception, but have difficulty processing motion in high external noise. The ability to exclude noise or ignore distractors while focusing on the what is relevant may play a role the creation of appropriately flexible yet solid phonological and orthographic categories, a fundamental process in learning to read

Fast and Biologically Plausible Computation with Perturbed Gaussian Markov Processes

In recent years a wide range of statistical models have been applied to vision related problems and have enjoyed much success. In a generative probabilistic model, the probability distributions of the observed images together with hidden variables describing the images are formulated (in a top-down fashion), and the visual perception and learning can be understood as an inference (bottom-up) operation that computes the posterior probabilities over the hidden variables, based on which model selection and parameter tuning, for example, can be carried out. A 'good' model requires a realistic probabilistic formulation that closely matches the statistics of the input data, and requires that the computation resulting from such formulation is tractable, and hopefully also biologically plausible. Those two requirements are not trivial. In factor analysis, for example, the observed image is expressed as a linear superposition of many basis functions. While the generation or synthesis of the image is immediate, the inference operation would typically require iterations if non-Gaussian prior is assumed or if direct matrix inversion is not allowed. If, on the other hand, the image is simply projected onto a set of filters, e.g., Gabor functions, then the probabilistic formulation is confounded, that is, it's not immediately clear how confident it would be to interpret a certain filter's response as the detection of a feature,e.g., an edge. In this talk, we present a generative probabilistic model that consists of a mixture of perturbed 2D Gaussian Markov processes. (Because of this mixing, the resulting model is non-Gaussian.) In each Gaussian Markov process, the adjacent hidden nodes on a 2D grid is coupled by some bond energy that resembles the energy prescribed in the "plate" model. This bond energy, however, can be subject to perturbation. Specifically, the 'bond' can be 'broken' or weakened. This is a manipulation on the inverse of the covariance matrix of the Gaussian process, instead of a constant amount of addition/subtraction to the covariance matrix as in the case of adding/removing a basis in the factor analysis. We show that the inference of the posterior probability of such perturbation amounts to the following computation: the input image is projected onto several receptive fields, and their outputs then go through a quadratic nonlinearity, subtract a threshold (controlled by the prior) and subsequently undergo a sigmoid function. Low-level features such as edges and bars of different scale and orientation can be obtained by suitable perturbations. Therefore the output of those feature detectors correspond to the data-likelihood given those components in our mixture model. We demonstrate how different features interact with each other: specifically, lateral inhibition and colinear facilitation. Also, we show that a contour can 'gate', or modify the extent of other feature detectors in its vicinity. Note that those phenomena fall directly from our probabilistic formulation; there are no heuristics involved. When we move beyond individual feature detectors and try to infer the posterior probability of contours, we will encounter the computation involving matrix inversion. We then show that there exists a family of effective preconditioners for different configurations of contours. In fact, those preconditioners are so good that the matrix inversion can be obtained in a single step! The posterior mean and covariance of the hidden nodes can therefore be easily obtained (in negligible time on a PC). In contrast, algorithms such as anisotropic diffusion or Graduated Non- Convexity would typically need many iterations of lateral propagation of information. In summary, apart from adapting a few parameters (e.g., noise level), the inference of our model can be carried out in predominantly feedforward, fan-in/fan-out type of computation, and seems biologically plausible

What Can We Learn in Perceptual Learning?

Please see attached pdf fil

Use of correlation to assess the signal content of patterns of spike activity reflecting the population response of relays of the central auditorimotor system of cats

Please see attached PD

Factors contributing to Post-Traumatic Dentate Hyperexcitability: A Network Model Incorporating Topographic Connectivity Patterns

Head injury is a major risk factor in the etiology of temporal lobe epilepsy (TLE). Studies using a rodent model of concussive head trauma have identified specific patterns of cell loss and synaptic reorganization in the dentate gyrus after brain injury, which are similar to the changes in human TLE. However, the contribution of each of these cellular and synaptic alterations to increased excitability in the dentate neuronal circuits is not known. In order to independently examine the factors critical to post-traumatic dentate bhyperexcitability, we developed a reduced network model of the dentate gyrus with 500 granule cells, 15 mossy cells 6 basket cells and 6 hilar interneurons. The topographic networks were constructed with connectivity patterns constrained by the spatial distribution of the axonal arbors of the cell types. Sprouting was simulated by addition of mossy fiber to granule cell connections with the maximum sprouting (100%) estimated from the distribution of sprouted axons in a rodent model of spontaneous recurrent seizures (Buckmaster and Dudek 1999). Simulations were performed using NEURON (Hines 1993). Our results show that perforant path stimulation evoked greater granule cell firing in the dentate excitatory network with as low as 10% sprouting compared to the control topographic network. Additionally, the topographic network was more hyperexcitable than the non-topographic network with the same degree of sprouting. Mossy cell loss decreased the spread of hyperexcitability in the network 10% sprouting. With increasing sprouting, even the complete loss of mossy cells was unable to prevent the spread of hyperexcitability. Simulations of both purely excitatory network and the full network showed that mossy fiber sprouting was sufficient to elicit hyperexcitable perforant path evoked responses in all cell types examined. Mossy cell loss was neither necessary nor sufficient to cause granule cell hyperexcitability in the dentate network with inhibition. The network simulations show that mossy fiber sprouting can contribute to increased excitability in the dentate gyrus even in the absence of cell loss or changes in the intrinsic properties of the cells. The data from the topographically constrained simulations indicate that the lamellar topology of the sprouted mossy fibers is important for the spread of granule cell excitability. The results suggest that the moderate sprouting observed after concussive head trauma is likely to be a major factor in post-traumatic dentate hyperexcitability. Acknowledgment: Supported by the NIH (NS35915) to I.S

Characterizing and modeling temporal dynamics of perceptual decision making

We combined the external noise method (1) with the cue-to-respond speed accuracy trade-off (SAT) paradigm (2) to characterize the temporal dynamics of perceptual decision making. Observers were required to identify the orientation of one of eight briefly presented peripheral Gabor targets (+/- 12 deg) in both zero and high noise. An arrow, occurring in the center of the display cued the observer to the target location 234 ms before the onset of a brief target display; an auditory beep, occurring at one of eight delays (SOA=25 to 800 ms) after the target onset, cued the observers to respond. Five Gabor contrasts, spanning a wide range of performance levels, were tested in each external noise condition. Increasing accuracy of discrimination (d') was measured over processing times from 210 to 940 ms (as a function of SOA to the cue) in each external noise and Gabor contrast condition. All ten SAT functions were well fit by exponential functions with identical time constant and intercept but different asymptotic levels. This suggests that, despite enormous variation in the external noise and contrast energy in the stimulus, and in the ultimate accuracy of performance, information accumulated with the same rate and starting time across all the external noise and contrast conditions. In addition, we conducted a standard response time version of the experiment both before and halfway through the SAT procedure. Data from the response time version of the experiment were all consistent with the speed-accuracy trade-off data, but primarily differed in response accuracy. A simple elaboration of the perceptual template model (3) with a dynamic decision process in which information accumulates with the same rate but with step sizes proportional to the signal to noise ratio in the perceptual representation of the visual input fully accounts for the results. (1) Pelli, Dissertation; (2) Dosher, Cognitive Psychology'76; (3) Lu & Dosher, JOSA'99

Neural Correlate of Object-Based Selection in Area V4

Single unit studies of attention in monkeys have identified competitive circuits in extrastriate cortex that could mediate selection of one stimulus over another. While these studies show that attention operates by resolving competition, they used stimuli atseparate locations, confounding selection of objects with selection of spatial locations. To resolve this, we recorded responses of V4 neurons to two spatially superimposed transparent surfaces, one of which was delayed in onset. The surfaces were defined by patterns of dots that rotated rigidly around a common center. One set of dots was of the neuron's preferred color and the other was of an isoluminant non-preferred color. Human psychophysics using the same type of stimuli found that the delayed onset of one surface exogenously cues attention to that surface and suppresses processing of the other surface for several hundred milliseconds. Consistent with this, neurons in area V4 were preferentially driven by the delayed surface. Using superimposed surfaces ruled out spatial selection. But is this selection object-based? If it is, the selection should survive moving the superimposed surfaces through space. When the appearance of one of the two surfaces was delayed outside the neuron's receptive field and both surfaces then moved into the RF, the pair response was still preferentially driven by the delayed surface. Neurophysiological and functional imaging studies have shown that endogenously directing attention to the color or motion of a stimulus preferentially processes it throughout the visual field. We tested for feature-based selection by using placing two surfaces within the RF and two outside of the RF. When the delayed surface appeared within the RF, the results were similar to the first experiment, i.e. the delayed surface was preferentially processed. If this effect were the result of global color-based selection, thenthe same effect should be seen when the delayed surface appeared outside the RF. This effect was not seen, hence the selection was not of the color of the surface but of the surface itself. These results show that competitive circuits in V4 are not limited to mediating competition between spatial locations, but also select objects. These circuits are a likely neural substrate for object-based attention

Recognizing Persons with One-Shot Learning

There have been several attempts to solve the problem of Human Recognition i.e. the ability to identify individual persons in novel situations. Using facial features (e.g. Wiskott et al, Facial Recognition using Elastic Bunch Graph Matching, 1997) for this purpose has proved to be quite successful. However when a person is at a appreciable distance, then the facial resolution is insufficient for reliable recognition. Therefore, some systems use additional information such as: Walking Patterns (Collins et al, Silhouette-based Human Identification from Body Shape and Gait, 2002) or distinguish color and shape features using Support Vector Machine classifiers (Nakajima et al, Full-body Person Recognition System,2003). We present here a simple system, which is able to recognize and track people from video sequences in real time. The implemented system learns the representation of the person using just a single video sequence (one-shot), with enough detail to permit later recognition and enough generality to deal with variation. To achieve this we divide the image of a person into regions: head, torso and legs, using a minimal model of the human body (corresponding to a virtually naive spectator). It learns the color and texture features for each region and stores them in a database of people. Thereafter, for recognition, it computes a similarity function between the input 'instance' and each person in the database. The person that generates the maximum similarity is chosen as the recognized person (this similarity value often exceeds the other people in the database by over three orders of magnitude). Furthermore, since there is no specific parameter tuning required for either learning or recognition, the system illustrates superior ability for automatic visual surveillance. This system could be used in conjunction with a face recognition system to reduce the search space for faces, by narrowing down the number of possibilities based on person recognition

Estimation of linear and nonlinear spatial receptive fields from natural Images

Please see attached pd