63,259 research outputs found
Large Scale Image Segmentation with Structured Loss based Deep Learning for Connectome Reconstruction
We present a method combining affinity prediction with region agglomeration,
which improves significantly upon the state of the art of neuron segmentation
from electron microscopy (EM) in accuracy and scalability. Our method consists
of a 3D U-NET, trained to predict affinities between voxels, followed by
iterative region agglomeration. We train using a structured loss based on
MALIS, encouraging topologically correct segmentations obtained from affinity
thresholding. Our extension consists of two parts: First, we present a
quasi-linear method to compute the loss gradient, improving over the original
quadratic algorithm. Second, we compute the gradient in two separate passes to
avoid spurious gradient contributions in early training stages. Our predictions
are accurate enough that simple learning-free percentile-based agglomeration
outperforms more involved methods used earlier on inferior predictions. We
present results on three diverse EM datasets, achieving relative improvements
over previous results of 27%, 15%, and 250%. Our findings suggest that a single
method can be applied to both nearly isotropic block-face EM data and
anisotropic serial sectioned EM data. The runtime of our method scales linearly
with the size of the volume and achieves a throughput of about 2.6 seconds per
megavoxel, qualifying our method for the processing of very large datasets
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
Feature learning, i.e. extracting meaningful representations of data, is
quintessential to the practical success of neural networks trained with
gradient descent, yet it is notoriously difficult to explain how and why it
occurs. Recent theoretical studies have shown that shallow neural networks
optimized on a single task with gradient-based methods can learn meaningful
features, extending our understanding beyond the neural tangent kernel or
random feature regime in which negligible feature learning occurs. But in
practice, neural networks are increasingly often trained on {\em many} tasks
simultaneously with differing loss functions, and these prior analyses do not
generalize to such settings. In the multi-task learning setting, a variety of
studies have shown effective feature learning by simple linear models. However,
multi-task learning via {\em nonlinear} models, arguably the most common
learning paradigm in practice, remains largely mysterious. In this work, we
present the first results proving feature learning occurs in a multi-task
setting with a nonlinear model. We show that when the tasks are binary
classification problems with labels depending on only directions within the
ambient -dimensional input space, executing a simple gradient-based
multitask learning algorithm on a two-layer ReLU neural network learns the
ground-truth directions. In particular, any downstream task on the
ground-truth coordinates can be solved by learning a linear classifier with
sample and neuron complexity independent of the ambient dimension , while a
random feature model requires exponential complexity in for such a
guarantee
SuperSpike: Supervised learning in multi-layer spiking neural networks
A vast majority of computation in the brain is performed by spiking neural
networks. Despite the ubiquity of such spiking, we currently lack an
understanding of how biological spiking neural circuits learn and compute
in-vivo, as well as how we can instantiate such capabilities in artificial
spiking circuits in-silico. Here we revisit the problem of supervised learning
in temporally coding multi-layer spiking neural networks. First, by using a
surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based
three factor learning rule capable of training multi-layer networks of
deterministic integrate-and-fire neurons to perform nonlinear computations on
spatiotemporal spike patterns. Second, inspired by recent results on feedback
alignment, we compare the performance of our learning rule under different
credit assignment strategies for propagating output errors to hidden units.
Specifically, we test uniform, symmetric and random feedback, finding that
simpler tasks can be solved with any type of feedback, while more complex tasks
require symmetric feedback. In summary, our results open the door to obtaining
a better scientific understanding of learning and computation in spiking neural
networks by advancing our ability to train them to solve nonlinear problems
involving transformations between different spatiotemporal spike-time patterns
- …