2,746 research outputs found
An investigation into the performance and representation of a stochastic evolutionary neural tree
Copyright Springer.The Stochastic Competitive Evolutionary Neural Tree (SCENT) is a new unsupervised neural net that dynamically evolves a representational structure in response to its training data. Uniquely SCENT requires no initial parameter setting as it autonomously creates appropriate parameterisation at runtime. Pruning and convergence are stochastically controlled using locally calculated heuristics. A thorough investigation into the performance of SCENT is presented. The network is compared to other dynamic tree based models and to a high quality flat clusterer over a variety of data sets and runs
Efficient Ultrasound Image Analysis Models with Sonographer Gaze Assisted Distillation.
Recent automated medical image analysis methods have attained state-of-the-art performance but have relied on memory and compute-intensive deep learning models. Reducing model size without significant loss in performance metrics is crucial for time and memory-efficient automated image-based decision-making. Traditional deep learning based image analysis only uses expert knowledge in the form of manual annotations. Recently, there has been interest in introducing other forms of expert knowledge into deep learning architecture design. This is the approach considered in the paper where we propose to combine ultrasound video with point-of-gaze tracked for expert sonographers as they scan to train memory-efficient ultrasound image analysis models. Specifically we develop teacher-student knowledge transfer models for the exemplar task of frame classification for the fetal abdomen, head, and femur. The best performing memory-efficient models attain performance within 5% of conventional models that are 1000Ă— larger in size
MobileNetV2: Inverted Residuals and Linear Bottlenecks
In this paper we describe a new mobile architecture, MobileNetV2, that
improves the state of the art performance of mobile models on multiple tasks
and benchmarks as well as across a spectrum of different model sizes. We also
describe efficient ways of applying these mobile models to object detection in
a novel framework we call SSDLite. Additionally, we demonstrate how to build
mobile semantic segmentation models through a reduced form of DeepLabv3 which
we call Mobile DeepLabv3.
The MobileNetV2 architecture is based on an inverted residual structure where
the input and output of the residual block are thin bottleneck layers opposite
to traditional residual models which use expanded representations in the input
an MobileNetV2 uses lightweight depthwise convolutions to filter features in
the intermediate expansion layer. Additionally, we find that it is important to
remove non-linearities in the narrow layers in order to maintain
representational power. We demonstrate that this improves performance and
provide an intuition that led to this design. Finally, our approach allows
decoupling of the input/output domains from the expressiveness of the
transformation, which provides a convenient framework for further analysis. We
measure our performance on Imagenet classification, COCO object detection, VOC
image segmentation. We evaluate the trade-offs between accuracy, and number of
operations measured by multiply-adds (MAdd), as well as the number of
parameter
Sparsely Aggregated Convolutional Networks
We explore a key architectural aspect of deep convolutional neural networks:
the pattern of internal skip connections used to aggregate outputs of earlier
layers for consumption by deeper layers. Such aggregation is critical to
facilitate training of very deep networks in an end-to-end manner. This is a
primary reason for the widespread adoption of residual networks, which
aggregate outputs via cumulative summation. While subsequent works investigate
alternative aggregation operations (e.g. concatenation), we focus on an
orthogonal question: which outputs to aggregate at a particular point in the
network. We propose a new internal connection structure which aggregates only a
sparse set of previous outputs at any given depth. Our experiments demonstrate
this simple design change offers superior performance with fewer parameters and
lower computational requirements. Moreover, we show that sparse aggregation
allows networks to scale more robustly to 1000+ layers, thereby opening future
avenues for training long-running visual processes.Comment: Accepted to ECCV 201
Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
Neural network pruning is a fruitful area of research with surging interest
in high sparsity regimes. Benchmarking in this domain heavily relies on
faithful representation of the sparsity of subnetworks, which has been
traditionally computed as the fraction of removed connections (direct
sparsity). This definition, however, fails to recognize unpruned parameters
that detached from input or output layers of underlying subnetworks,
potentially underestimating actual effective sparsity: the fraction of
inactivated connections. While this effect might be negligible for moderately
pruned networks (up to 10-100 compression rates), we find that it plays an
increasing role for thinner subnetworks, greatly distorting comparison between
different pruning algorithms. For example, we show that effective compression
of a randomly pruned LeNet-300-100 can be orders of magnitude larger than its
direct counterpart, while no discrepancy is ever observed when using SynFlow
for pruning [Tanaka et al., 2020]. In this work, we adopt the lens of effective
sparsity to reevaluate several recent pruning algorithms on common benchmark
architectures (e.g., LeNet-300-100, VGG-19, ResNet-18) and discover that their
absolute and relative performance changes dramatically in this new and more
appropriate framework. To aim for effective, rather than direct, sparsity, we
develop a low-cost extension to most pruning algorithms. Further, equipped with
effective sparsity as a reference frame, we partially reconfirm that random
pruning with appropriate sparsity allocation across layers performs as well or
better than more sophisticated algorithms for pruning at initialization [Su et
al., 2020]. In response to this observation, using a simple analogy of pressure
distribution in coupled cylinders from physics, we design novel layerwise
sparsity quotas that outperform all existing baselines in the context of random
pruning
- …