11 research outputs found
End-To-End Data-Dependent Routing in Multi-Path Neural Networks
Neural networks are known to give better performance with increased depth due
to their ability to learn more abstract features. Although the deepening of
networks has been well established, there is still room for efficient feature
extraction within a layer which would reduce the need for mere parameter
increment. The conventional widening of networks by having more filters in each
layer introduces a quadratic increment of parameters. Having multiple parallel
convolutional/dense operations in each layer solves this problem, but without
any context-dependent allocation of resources among these operations: the
parallel computations tend to learn similar features making the widening
process less effective. Therefore, we propose the use of multi-path neural
networks with data-dependent resource allocation among parallel computations
within layers, which also lets an input to be routed end-to-end through these
parallel paths. To do this, we first introduce a cross-prediction based
algorithm between parallel tensors of subsequent layers. Second, we further
reduce the routing overhead by introducing feature-dependent cross-connections
between parallel tensors of successive layers. Our multi-path networks show
superior performance to existing widening and adaptive feature extraction, and
even ensembles, and deeper networks at similar complexity in the image
recognition task
Neural Mixture Models with Expectation-Maximization for End-to-end Deep Clustering
Any clustering algorithm must synchronously learn to model the clusters and
allocate data to those clusters in the absence of labels. Mixture model-based
methods model clusters with pre-defined statistical distributions and allocate
data to those clusters based on the cluster likelihoods. They iteratively
refine those distribution parameters and member assignments following the
Expectation-Maximization (EM) algorithm. However, the cluster representability
of such hand-designed distributions that employ a limited amount of parameters
is not adequate for most real-world clustering tasks. In this paper, we realize
mixture model-based clustering with a neural network where the final layer
neurons, with the aid of an additional transformation, approximate cluster
distribution outputs. The network parameters pose as the parameters of those
distributions. The result is an elegant, much-generalized representation of
clusters than a restricted mixture of hand-designed distributions. We train the
network end-to-end via batch-wise EM iterations where the forward pass acts as
the E-step and the backward pass acts as the M-step. In image clustering, the
mixture-based EM objective can be used as the clustering objective along with
existing representation learning methods. In particular, we show that when
mixture-EM optimization is fused with consistency optimization, it improves the
sole consistency optimization performance in clustering. Our trained networks
outperform single-stage deep clustering methods that still depend on k-means,
with unsupervised classification accuracy of 63.8% in STL10, 58% in CIFAR10,
25.9% in CIFAR100, and 98.9% in MNIST
A Stochastic and Competitive Hebbian Learning Mechanism through Spike-Timing Dependent Plasticity to Lift Hard Weight Constraints on Hebbian Synapses
Spike-Timing-Dependent Plasticity and Short-Term Plasticity Jointly Control the Excitation of Hebbian Plasticity without Weight Constraints in Neural Networks
Hebbian plasticity precisely describes how synapses increase their synaptic strengths according to the correlated activities between two neurons; however, it fails to explain how these activities dilute the strength of the same synapses. Recent literature has proposed spike-timing-dependent plasticity and short-term plasticity on multiple dynamic stochastic synapses that can control synaptic excitation and remove many user-defined constraints. Under this hypothesis, a network model was implemented giving more computational power to receptors, and the behavior at a synapse was defined by the collective dynamic activities of stochastic receptors. An experiment was conducted to analyze can spike-timing-dependent plasticity interplay with short-term plasticity to balance the excitation of the Hebbian neurons without weight constraints? If so what underline mechanisms help neurons to maintain such excitation in computational environment? According to our results both plasticity mechanisms work together to balance the excitation of the neural network as our neurons stabilized its weights for Poisson inputs with mean firing rates from 10 Hz to 40 Hz. The behavior generated by the two neurons was similar to the behavior discussed under synaptic redistribution, so that synaptic weights were stabilized while there was a continuous increase of presynaptic probability of release and higher turnover rate of postsynaptic receptors
Transformers in Single Object Tracking: An Experimental Survey
Single-object tracking is a well-known and challenging research topic in computer vision. Over the last two decades, numerous researchers have proposed various algorithms to solve this problem and achieved promising results. Recently, Transformer-based tracking approaches have ushered in a new era in single-object tracking by introducing new perspectives and achieving superior tracking robustness. In this paper, we conduct an in-depth literature analysis of Transformer tracking approaches by categorizing them into CNN-Transformer based trackers, Two-stream Two-stage fully-Transformer based trackers, and One-stream One-stage fully-Transformer based trackers. In addition, we conduct experimental evaluations to assess their tracking robustness and computational efficiency using publicly available benchmark datasets. Furthermore, we measure their performances on different tracking scenarios to identify their strengths and weaknesses in particular situations. Our survey provides insights into the underlying principles of Transformer tracking approaches, the challenges they encounter, and the future directions they may take
Transformers in Single Object Tracking: An Experimental Survey
Single-object tracking is a well-known and challenging research topic in
computer vision. Over the last two decades, numerous researchers have proposed
various algorithms to solve this problem and achieved promising results.
Recently, Transformer-based tracking approaches have ushered in a new era in
single-object tracking by introducing new perspectives and achieving superior
tracking robustness. In this paper, we conduct an in-depth literature analysis
of Transformer tracking approaches by categorizing them into CNN-Transformer
based trackers, Two-stream Two-stage fully-Transformer based trackers, and
One-stream One-stage fully-Transformer based trackers. In addition, we conduct
experimental evaluations to assess their tracking robustness and computational
efficiency using publicly available benchmark datasets. Furthermore, we measure
their performances on different tracking scenarios to identify their strengths
and weaknesses in particular situations. Our survey provides insights into the
underlying principles of Transformer tracking approaches, the challenges they
encounter, and the future directions they may take.Comment: 36 pages, 22 figures, review paper, submitted to IEEE Access, updated
with CVPR-2023 paper