Dynamic Slicing for Deep Neural Networks

Program slicing has been widely applied in a variety of software engineering tasks. However, existing program slicing techniques only deal with traditional programs that are constructed with instructions and variables, rather than neural networks that are composed of neurons and synapses. In this paper, we propose NNSlicer, the first approach for slicing deep neural networks based on data flow analysis. Our method understands the reaction of each neuron to an input based on the difference between its behavior activated by the input and the average behavior over the whole dataset. Then we quantify the neuron contributions to the slicing criterion by recursively backtracking from the output neurons, and calculate the slice as the neurons and the synapses with larger contributions. We demonstrate the usefulness and effectiveness of NNSlicer with three applications, including adversarial input detection, model pruning, and selective model protection. In all applications, NNSlicer significantly outperforms other baselines that do not rely on data flow analysis.Comment: 11 pages, ESEC/FSE '2

BlockDrop: Dynamic Inference Paths in Residual Networks

Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal number of blocks while preserving recognition accuracy. We conduct extensive experiments on CIFAR and ImageNet. The results provide strong quantitative and qualitative evidence that these learned policies not only accelerate inference but also encode meaningful visual information. Built upon a ResNet-101 model, our method achieves a speedup of 20\% on average, going as high as 36\% for some images, while maintaining the same 76.4\% top-1 accuracy on ImageNet.Comment: CVPR 201

Developmental and interspecies comparison of morphology and plasticity in neuronal circuits involved in olfactory information processing

The anterior piriform cortex (aPCx) is a three layered paleocortex receiving afferent inputs from the olfactory bulb as well as local and long-range associational inputs. Neurons in layer 2 are segregated into layer 2A and layer 2B according to their position, morphology and implementation in the sensory and associative circuits. The dendritic architecture of these neurons is determined during postnatal development and plays an important role for the functionality and circuit integration of the two cell types. Here, confocal imaging, electrophysiology, morphometry and Ca2+ imaging, were combined in order to study the development of the dendritic arborizations for both subtypes of layer 2 neurons. Three different growth phases were identified: branch complexity determination, branch elongation and pruning, occurring at different time windows during development. Layer 2A and layer 2B neurons showed morphological differences between their apical and basal dendrites from the very first postnatal days; as well as phase-specific differences during development associated to differences in circuit implementation. During the first postnatal week, early spontaneous network activity in layer 2 of the aPCx displayed differences between layer 2A and layer 2B neurons in their functional connectivity, reflected in the morphological dissimilarities between their basal dendritic trees during the period of branch complexity determination. Additionally, strong differences in growth phase three were observed. Pruning was exclusive for layer 2B neurons and selective for apical dendrites receiving layer 1A sensory inputs. These differences between layer 2A and layer 2B cells in their morphological and functional development exhibit the close association between circuit specificity and neuronal development. Finally, synaptic plasticity in the mossy fiber (MF) pathway of the hippocampus in shrews was investigated and compared to mice. Although hippocampal structure in shrews is preserved, short and long-term plasticity at the MF synapsis was lower compared to mice, suggesting different involvement of these synapses in the behavioral outcome of different species.Der Cortex piriformis anterior (aPCx auf Englisch) ist ein dreischichtiger Paläokortex, der sensorische afferente Eingänge aus dem Riechkolben sowie intracerebrale assoziative Eingänge empfängt. Die Neuronen in Schicht 2 werden nach ihrer Position, Morphologie und Einbindung in die sensorischen und rekurrenten Netzwerke in die Schichten 2A und 2B unterteilt. Die dendritische Architektur dieser Neurone wird während der postnatalen Entwicklung festgelegt und spielt eine wichtige Rolle für die Funktionalität und Netzwerkintegration der beiden Zelltypen. Hier wurden konfokales Imaging, Elektrophysiologie, Morphometrie und Kalzium-Imaging kombiniert, um die Entwicklung der Dendritenbäume für beide Subtypen von Schicht-2-Neuronen zu untersuchen. Es wurden drei verschiedene Wachstumsphasen identifiziert: Bestimmung der Komplexität der Verzweigung, Verlängerung der Verzweigung und strukturelle Vereinfachung, die in verschiedenen Zeitfenstern während der Entwicklung auftreten. Neurone der Schicht 2A und der Schicht 2B zeigten bereits in den ersten postnatalen Tagen morphologische Unterschiede zwischen ihren apikalen und basalen Dendriten sowie phasenspezifische Unterschiede während der Entwicklung, die mit Unterschieden in der Netzwerkimplementierung verbunden sind. Während der ersten postnatalen Woche zeigte die frühe spontane Netzwerkaktivität in Schicht 2 des aPCx Unterschiede in der funktionellen Konnektivität zwischen Neuronen der Schicht 2A und Schicht 2B, die sich in den morphologischen Unterschieden zwischen ihren basalen Dendritenbäumen während der Bestimmung der Verzweigungskomplexität widerspiegelten. Außerdem wurden starke Unterschiede in der dritten Wachstumsphase beobachtet. Die strukturelle Vereinfachung fand ausschließlich bei Neuronen der Schicht 2B statt und war selektiv für apikale Dendriten, die sensorische Inputs der Schicht 1A erhielten. Diese Unterschiede zwischen Zellen der Schicht 2A und der Schicht 2B in ihrer morphologischen und funktionellen Entwicklung zeigen den engen Zusammenhang zwischen Netzwerkspezifität und neuronaler Entwicklung. Schließlich wurde die synaptische Plastizität des Moosfaser (MF)-Trakts des Hippocampus bei Spitzmäusen untersucht und mit der von Mäusen verglichen. Obwohl die Struktur des Hippocampus bei Spitzmäusen erhalten ist, war die Kurz- und Langzeitplastizität an den MF-Synapsen im Vergleich zu Mäusen geringer, was auf eine unterschiedliche Beteiligung dieser Synapsen an spezifisch adaptierte Verhaltensweisen der beiden Spezies hindeutet

Circuit Dynamics of Adult Visual Cortex

Learning and other forms of plasticity result from changes in transmission at existing synapses or the construction or elimination of synapses. Synapses occur at the juxtaposition of boutons, with their postsynaptic partners, dendrites and cell bodies. It has been assumed that connections in the primary visual cortex (V1) become static after the critical period. Recent studies show, however, that dendritic spines appear and disappear during adulthood in the normal brain. Our first objective was then to determine whether axonal branches and boutons also undergo morphological changes. To do so, we performed longitudinal studies of virally labeled neurons and their processes. An adeno-associated virus bearing the gene for enhanced green fluorescent protein (AAV.EGFP) provided long-term labeling of axons and their boutons in adult Macaque V1. To image the neurons in vivo, a custom-designed two-photon microscope and viewing chamber provided repeated imaging of selected locations. We examined the same EGFP-labeled axonal arbors at several time points over periods of weeks in the adult normal cortex. We found that axons are dynamic entities, in which a subset of boutons appeared and disappeared overtime, and that though axonal length and branching was largely stable, a small subset of terminals underwent elongation, retraction or appeared de novo. These results suggest an ongoing process of synaptogenesis and synapse elimination in adult V1. To further investigate structural plasticity in the adult V1, we studied the cortical reorganization that accompanies retinal lesions. Removal of visual input cause axonal sprouting of long-range horizontal connections from pyramidal cells in layer 2/3. Our in vivo approach allowed us to determine the dynamics of the process of sprouting. Immediately following retinal lesions, there was a remarkable rise in axonal density. In the following weeks, the massive increase in axon collaterals was accompanied by a comparable rate of axonal elimination. Also, boutons increased their rate of appearance and elimination beyond the rates seen in normal cortex. These data indicate that the initial sprouting of axons followed by the subsequent refinement, may account for the dynamics of receptive field changes observed during the course of topographic reorganization of visual cortex