1,591 research outputs found
Deep Learning for Single Image Super-Resolution: A Brief Review
Single image super-resolution (SISR) is a notoriously challenging ill-posed
problem, which aims to obtain a high-resolution (HR) output from one of its
low-resolution (LR) versions. To solve the SISR problem, recently powerful deep
learning algorithms have been employed and achieved the state-of-the-art
performance. In this survey, we review representative deep learning-based SISR
methods, and group them into two categories according to their major
contributions to two essential aspects of SISR: the exploration of efficient
neural network architectures for SISR, and the development of effective
optimization objectives for deep SISR learning. For each category, a baseline
is firstly established and several critical limitations of the baseline are
summarized. Then representative works on overcoming these limitations are
presented based on their original contents as well as our critical
understandings and analyses, and relevant comparisons are conducted from a
variety of perspectives. Finally we conclude this review with some vital
current challenges and future trends in SISR leveraging deep learning
algorithms.Comment: Accepted by IEEE Transactions on Multimedia (TMM
Predicting speech from a cortical hierarchy of event-based timescales
How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse
Triplet Attention Transformer for Spatiotemporal Predictive Learning
Spatiotemporal predictive learning offers a self-supervised learning paradigm
that enables models to learn both spatial and temporal patterns by predicting
future sequences based on historical sequences. Mainstream methods are
dominated by recurrent units, yet they are limited by their lack of
parallelization and often underperform in real-world scenarios. To improve
prediction quality while maintaining computational efficiency, we propose an
innovative triplet attention transformer designed to capture both inter-frame
dynamics and intra-frame static features. Specifically, the model incorporates
the Triplet Attention Module (TAM), which replaces traditional recurrent units
by exploring self-attention mechanisms in temporal, spatial, and channel
dimensions. In this configuration: (i) temporal tokens contain abstract
representations of inter-frame, facilitating the capture of inherent temporal
dependencies; (ii) spatial and channel attention combine to refine the
intra-frame representation by performing fine-grained interactions across
spatial and channel dimensions. Alternating temporal, spatial, and
channel-level attention allows our approach to learn more complex short- and
long-range spatiotemporal dependencies. Extensive experiments demonstrate
performance surpassing existing recurrent-based and recurrent-free methods,
achieving state-of-the-art under multi-scenario examination including moving
object trajectory prediction, traffic flow prediction, driving scene
prediction, and human motion capture.Comment: Accepted to WACV 202
- …