330 research outputs found
Partially Linear Estimation with Application to Sparse Signal Recovery From Measurement Pairs
We address the problem of estimating a random vector X from two sets of
measurements Y and Z, such that the estimator is linear in Y. We show that the
partially linear minimum mean squared error (PLMMSE) estimator does not require
knowing the joint distribution of X and Y in full, but rather only its
second-order moments. This renders it of potential interest in various
applications. We further show that the PLMMSE method is minimax-optimal among
all estimators that solely depend on the second-order statistics of X and Y. We
demonstrate our approach in the context of recovering a signal, which is sparse
in a unitary dictionary, from noisy observations of it and of a filtered
version of it. We show that in this setting PLMMSE estimation has a clear
computational advantage, while its performance is comparable to
state-of-the-art algorithms. We apply our approach both in static and dynamic
estimation applications. In the former category, we treat the problem of image
enhancement from blurred/noisy image pairs, where we show that PLMMSE
estimation performs only slightly worse than state-of-the art algorithms, while
running an order of magnitude faster. In the dynamic setting, we provide a
recursive implementation of the estimator and demonstrate its utility in the
context of tracking maneuvering targets from position and acceleration
measurements.Comment: 13 pages, 5 figure
Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges
Machine Learning algorithms have had a profound impact on the field of
computer science over the past few decades. These algorithms performance is
greatly influenced by the representations that are derived from the data in the
learning process. The representations learned in a successful learning process
should be concise, discrete, meaningful, and able to be applied across a
variety of tasks. A recent effort has been directed toward developing Deep
Learning models, which have proven to be particularly effective at capturing
high-dimensional, non-linear, and multi-modal characteristics. In this work, we
discuss the principles and developments that have been made in the process of
learning representations, and converting them into desirable applications. In
addition, for each framework or model, the key issues and open challenges, as
well as the advantages, are examined
The Neural Tangent Link Between CNN Denoisers and Non-Local Filters
Convolutional Neural Networks (CNNs) are now a well-established tool for
solving computational imaging problems. Modern CNN-based algorithms obtain
state-of-the-art performance in diverse image restoration problems.
Furthermore, it has been recently shown that, despite being highly
overparameterized, networks trained with a single corrupted image can still
perform as well as fully trained networks. We introduce a formal link between
such networks through their neural tangent kernel (NTK), and well-known
non-local filtering techniques, such as non-local means or BM3D. The filtering
function associated with a given network architecture can be obtained in closed
form without need to train the network, being fully characterized by the random
initialization of the network weights. While the NTK theory accurately predicts
the filter associated with networks trained using standard gradient descent,
our analysis shows that it falls short to explain the behaviour of networks
trained using the popular Adam optimizer. The latter achieves a larger change
of weights in hidden layers, adapting the non-local filtering function during
training. We evaluate our findings via extensive image denoising experiments
Distantly-Supervised Named Entity Recognition with Uncertainty-aware Teacher Learning and Student-student Collaborative Learning
Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates
the burden of annotation, but meanwhile suffers from the label noise. Recent
works attempt to adopt the teacher-student framework to gradually refine the
training labels and improve the overall robustness. However, we argue that
these teacher-student methods achieve limited performance because poor network
calibration produces incorrectly pseudo-labeled samples, leading to error
propagation. Therefore, we attempt to mitigate this issue by proposing: (1)
Uncertainty-aware Teacher Learning that leverages the prediction uncertainty to
guide the selection of pseudo-labels, avoiding the number of incorrect
pseudo-labels in the self-training stage. (2) Student-student Collaborative
Learning that allows the transfer of reliable labels between two student
networks instead of completely relying on all pseudo-labels from its teacher.
Meanwhile, this approach allows a full exploration of mislabeled samples rather
than simply filtering unreliable pseudo-labeled samples. Extensive experimental
results on five DS-NER datasets demonstrate that our method is superior to
state-of-the-art teacher-student methods
Representation Learning: A Review and New Perspectives
The success of machine learning algorithms generally depends on data
representation, and we hypothesize that this is because different
representations can entangle and hide more or less the different explanatory
factors of variation behind the data. Although specific domain knowledge can be
used to help design representations, learning with generic priors can also be
used, and the quest for AI is motivating the design of more powerful
representation-learning algorithms implementing such priors. This paper reviews
recent work in the area of unsupervised feature learning and deep learning,
covering advances in probabilistic models, auto-encoders, manifold learning,
and deep networks. This motivates longer-term unanswered questions about the
appropriate objectives for learning good representations, for computing
representations (i.e., inference), and the geometrical connections between
representation learning, density estimation and manifold learning
- …