32 research outputs found
Efficient ConvNets for Analog Arrays
Analog arrays are a promising upcoming hardware technology with the potential
to drastically speed up deep learning. Their main advantage is that they
compute matrix-vector products in constant time, irrespective of the size of
the matrix. However, early convolution layers in ConvNets map very unfavorably
onto analog arrays, because kernel matrices are typically small and the
constant time operation needs to be sequentially iterated a large number of
times, reducing the speed up advantage for ConvNets. Here, we propose to
replicate the kernel matrix of a convolution layer on distinct analog arrays,
and randomly divide parts of the compute among them, so that multiple kernel
matrices are trained in parallel. With this modification, analog arrays execute
ConvNets with an acceleration factor that is proportional to the number of
kernel matrices used per layer (here tested 16-128). Despite having more free
parameters, we show analytically and in numerical experiments that this
convolution architecture is self-regularizing and implicitly learns similar
filters across arrays. We also report superior performance on a number of
datasets and increased robustness to adversarial attacks. Our investigation
suggests to revise the notion that mixed analog-digital hardware is not
suitable for ConvNets
Internal Representation of Task Rules by Recurrent Dynamics: The Importance of the Diversity of Neural Responses
Neural activity of behaving animals, especially in the prefrontal cortex, is highly heterogeneous, with selective responses to diverse aspects of the executed task. We propose a general model of recurrent neural networks that perform complex rule-based tasks, and we show that the diversity of neuronal responses plays a fundamental role when the behavioral responses are context-dependent. Specifically, we found that when the inner mental states encoding the task rules are represented by stable patterns of neural activity (attractors of the neural dynamics), the neurons must be selective for combinations of sensory stimuli and inner mental states. Such mixed selectivity is easily obtained by neurons that connect with random synaptic strengths both to the recurrent network and to neurons encoding sensory inputs. The number of randomly connected neurons needed to solve a task is on average only three times as large as the number of neurons needed in a network designed ad hoc. Moreover, the number of needed neurons grows only linearly with the number of task-relevant events and mental states, provided that each neuron responds to a large proportion of events (dense/distributed coding). A biologically realistic implementation of the model captures several aspects of the activity recorded from monkeys performing context-dependent tasks. Our findings explain the importance of the diversity of neural responses and provide us with simple and general principles for designing attractor neural networks that perform complex computation
Model-Assisted Labeling via Explainability for Visual Inspection of Civil Infrastructures
Labeling images for visual segmentation is a time-consuming task which can be
costly, particularly in application domains where labels have to be provided by
specialized expert annotators, such as civil engineering. In this paper, we
propose to use attribution methods to harness the valuable interactions between
expert annotators and the data to be annotated in the case of defect
segmentation for visual inspection of civil infrastructures. Concretely, a
classifier is trained to detect defects and coupled with an attribution-based
method and adversarial climbing to generate and refine segmentation masks
corresponding to the classification outputs. These are used within an assisted
labeling framework where the annotators can interact with them as proposal
segmentation masks by deciding to accept, reject or modify them, and
interactions are logged as weak labels to further refine the classifier.
Applied on a real-world dataset resulting from the automated visual inspection
of bridges, our proposed method is able to save more than 50\% of annotators'
time when compared to manual annotation of defects
Hebbian Learning in a Random Network Captures Selectivity Properties of the Prefrontal Cortex
Complex cognitive behaviors, such as context-switching and rule-following, are thought to be supported by the prefrontal cortex (PFC). Neural activity in the PFC must thus be specialized to specific tasks while retaining flexibility. Nonlinear “mixed” selectivity is an important neurophysiological trait for enabling complex and context-dependent behaviors. Here we investigate (1) the extent to which the PFC exhibits computationally relevant properties, such as mixed selectivity, and (2) how such properties could arise via circuit mechanisms. We show that PFC cells recorded from male and female rhesus macaques during a complex task show a moderate level of specialization and structure that is not replicated by a model wherein cells receive random feedforward inputs. While random connectivity can be effective at generating mixed selectivity, the data show significantly more mixed selectivity than predicted by a model with otherwise matched parameters. A simple Hebbian learning rule applied to the random connectivity, however, increases mixed selectivity and enables the model to match the data more accurately. To explain how learning achieves this, we provide analysis along with a clear geometric interpretation of the impact of learning on selectivity. After learning, the model also matches the data on measures of noise, response density, clustering, and the distribution of selectivities. Of two styles of Hebbian learning tested, the simpler and more biologically plausible option better matches the data. These modeling results provide clues about how neural properties important for cognition can arise in a circuit and make clear experimental predictions regarding how various measures of selectivity would evolve during animal training