Hardening against adversarial examples with the smooth gradient method

Commonly used methods in deep learning do not utilise transformations of the residual gradient available at the inputs to update the representation in the dataset. It has been shown that this residual gradient, which can be interpreted as the first-order gradient of the input sensitivity at a particular point, may be used to improve generalisation in feed-forward neural networks, including fully connected and convolutional layers. We explore how these input gradients are related to input perturbations used to generate adversarial examples and how the networks that are trained with this technique are more robust to attacks generated with the fast gradient sign method

Difficulty Classification of Mountainbike Downhill Trails utilizing Deep Neural Networks

The difficulty of mountainbike downhill trails is a subjective perception. However, sports-associations and mountainbike park operators attempt to group trails into different levels of difficulty with scales like the Singletrail-Skala (S0-S5) or colored scales (blue, red, black, ...) as proposed by The International Mountain Bicycling Association. Inconsistencies in difficulty grading occur due to the various scales, different people grading the trails, differences in topography, and more. We propose an end-to-end deep learning approach to classify trails into three difficulties easy, medium, and hard by using sensor data. With mbientlab Meta Motion r0.2 sensor units, we record accelerometer- and gyroscope data of one rider on multiple trail segments. A 2D convolutional neural network is trained with a stacked and concatenated representation of the aforementioned data as its input. We run experiments with five different sample- and five different kernel sizes and achieve a maximum Sparse Categorical Accuracy of 0.9097. To the best of our knowledge, this is the first work targeting computational difficulty classification of mountainbike downhill trails.Comment: 11 pages, 5 figure

A compact statistical model of the song syntax in Bengalese finch

Songs of many songbird species consist of variable sequences of a finite number of syllables. A common approach for characterizing the syntax of these complex syllable sequences is to use transition probabilities between the syllables. This is equivalent to the Markov model, in which each syllable is associated with one state, and the transition probabilities between the states do not depend on the state transition history. Here we analyze the song syntax in a Bengalese finch. We show that the Markov model fails to capture the statistical properties of the syllable sequences. Instead, a state transition model that accurately describes the statistics of the syllable sequences includes adaptation of the self-transition probabilities when states are repeatedly revisited, and allows associations of more than one state to the same syllable. Such a model does not increase the model complexity significantly. Mathematically, the model is a partially observable Markov model with adaptation (POMMA). The success of the POMMA supports the branching chain network hypothesis of how syntax is controlled within the premotor song nucleus HVC, and suggests that adaptation and many-to-one mapping from neural substrates to syllables are important features of the neural control of complex song syntax

Support for a synaptic chain model of neuronal sequence generation

In songbirds, the remarkable temporal precision of song is generated by a sparse sequence of bursts in the premotor nucleus HVC. To distinguish between two possible classes of models of neural sequence generation, we carried out intracellular recordings of HVC neurons in singing zebra finches (Taeniopygia guttata). We found that the subthreshold membrane potential is characterized by a large, rapid depolarization 5–10 ms before burst onset, consistent with a synaptically connected chain of neurons in HVC. We found no evidence for the slow membrane potential modulation predicted by models in which burst timing is controlled by subthreshold dynamics. Furthermore, bursts ride on an underlying depolarization of ~10-ms duration, probably the result of a regenerative calcium spike within HVC neurons that could facilitate the propagation of activity through a chain network with high temporal precision. Our results provide insight into the fundamental mechanisms by which neural circuits can generate complex sequential behaviours.National Institutes of Health (U.S.) (Grant MH067105)National Institutes of Health (U.S.) (Grant DC009280)National Science Foundation (U.S.) (IOS-0827731)Alfred P. Sloan Foundation (Research Fellowship

Encoding temporal regularities and information copying in hippocampal circuits

Discriminating, extracting and encoding temporal regularities is a critical requirement in the brain, relevant to sensory-motor processing and learning. However, the cellular mechanisms responsible remain enigmatic; for example, whether such abilities require specific, elaborately organized neural networks or arise from more fundamental, inherent properties of neurons. Here, using multi-electrode array technology, and focusing on interval learning, we demonstrate that sparse reconstituted rat hippocampal neural circuits are intrinsically capable of encoding and storing sub-second-order time intervals for over an hour timescale, represented in changes in the spatial-temporal architecture of firing relationships among populations of neurons. This learning is accompanied by increases in mutual information and transfer entropy, formal measures related to information storage and flow. Moreover, temporal relationships derived from previously trained circuits can act as templates for copying intervals into untrained networks, suggesting the possibility of circuit-to-circuit information transfer. Our findings illustrate that dynamic encoding and stable copying of temporal relationships are fundamental properties of simple in vitro networks, with general significance for understanding elemental principles of information processing, storage and replication

Characterization of Synaptically Connected Nuclei in a Potential Sensorimotor Feedback Pathway in the Zebra Finch Song System

Birdsong is a learned behavior that is controlled by a group of identified nuclei, known collectively as the song system. The cortical nucleus HVC (used as a proper name) is a focal point of many investigations as it is necessary for song production, song learning, and receives selective auditory information. HVC receives input from several sources including the cortical area MMAN (medial magnocellular nucleus of the nidopallium). The MMAN to HVC connection is particularly interesting as it provides potential sensorimotor feedback to HVC. To begin to understand the role of this connection, we investigated the physiological relation between MMAN and HVC activity with simultaneous multiunit extracellular recordings from these two nuclei in urethane anesthetized zebra finches. As previously reported, we found similar timing in spontaneous bursts of activity in MMAN and HVC. Like HVC, MMAN responds to auditory playback of the bird's own song (BOS), but had little response to reversed BOS or conspecific song. Stimulation of MMAN resulted in evoked activity in HVC, indicating functional excitation from MMAN to HVC. However, inactivation of MMAN resulted in no consistent change in auditory responses in HVC. Taken together, these results indicate that MMAN provides functional excitatory input to HVC but does not provide significant auditory input to HVC in anesthetized animals. We hypothesize that MMAN may play a role in motor reinforcement or coordination, or may provide modulatory input to the song system about the internal state of the animal as it receives input from the hypothalamus

Social Status Affects the Degree of Sex Difference in the Songbird Brain

It is thought that neural sex differences are functionally related to sex differences in the behaviour of vertebrates. A prominent example is the song control system of songbirds. Inter-specific comparisons have led to the hypothesis that sex differences in song nuclei size correlate with sex differences in song behaviour. However, only few species with similar song behaviour in both sexes have been investigated and not all data fit the hypothesis. We investigated the proposed structure – function relationship in a cooperatively breeding and duetting songbird, the white-browed sparrow weaver (Plocepasser mahali). This species lives in groups of 2–10 individuals, with a dominant breeding pair and male and female subordinates. While all male and female group members sing duet and chorus song, a male, once it has reached the dominant position in the group, sings an additional type of song that comprises a distinct and large syllable repertoire. Here we show for both types of male – female comparisons a male-biased sex difference in neuroanatomy of areas of the song production pathway (HVC and RA) that does not correlate with the observed polymorphism in song behaviour. In contrast, in situ hybridisation of mRNA of selected genes expressed in the song nucleus HVC reveals a gene expression pattern that is either similar between sexes in female – subordinate male comparisons or female-biased in female – dominant male comparisons. Thus, the polymorphic gene expression pattern would fit the sex- and status-related song behaviour. However, this implies that once a male has become dominant it produces the duetting song with a different neural phenotype than subordinate males

Encoding of Temporal Information by Timing, Rate, and Place in Cat Auditory Cortex

A central goal in auditory neuroscience is to understand the neural coding of species-specific communication and human speech sounds. Low-rate repetitive sounds are elemental features of communication sounds, and core auditory cortical regions have been implicated in processing these information-bearing elements. Repetitive sounds could be encoded by at least three neural response properties: 1) the event-locked spike-timing precision, 2) the mean firing rate, and 3) the interspike interval (ISI). To determine how well these response aspects capture information about the repetition rate stimulus, we measured local group responses of cortical neurons in cat anterior auditory field (AAF) to click trains and calculated their mutual information based on these different codes. ISIs of the multiunit responses carried substantially higher information about low repetition rates than either spike-timing precision or firing rate. Combining firing rate and ISI codes was synergistic and captured modestly more repetition information. Spatial distribution analyses showed distinct local clustering properties for each encoding scheme for repetition information indicative of a place code. Diversity in local processing emphasis and distribution of different repetition rate codes across AAF may give rise to concurrent feed-forward processing streams that contribute differently to higher-order sound analysis

Structure of Spontaneous UP and DOWN Transitions Self-Organizing in a Cortical Network Model

Synaptic plasticity is considered to play a crucial role in the experience-dependent self-organization of local cortical networks. In the absence of sensory stimuli, cerebral cortex exhibits spontaneous membrane potential transitions between an UP and a DOWN state. To reveal how cortical networks develop spontaneous activity, or conversely, how spontaneous activity structures cortical networks, we analyze the self-organization of a recurrent network model of excitatory and inhibitory neurons, which is realistic enough to replicate UP–DOWN states, with spike-timing-dependent plasticity (STDP). The individual neurons in the self-organized network exhibit a variety of temporal patterns in the two-state transitions. In addition, the model develops a feed-forward network-like structure that produces a diverse repertoire of precise sequences of the UP state. Our model shows that the self-organized activity well resembles the spontaneous activity of cortical networks if STDP is accompanied by the pruning of weak synapses. These results suggest that the two-state membrane potential transitions play an active role in structuring local cortical circuits