190 research outputs found
On Degrees of Freedom of Projection Estimators with Applications to Multivariate Nonparametric Regression
In this paper, we consider the nonparametric regression problem with
multivariate predictors. We provide a characterization of the degrees of
freedom and divergence for estimators of the unknown regression function, which
are obtained as outputs of linearly constrained quadratic optimization
procedures, namely, minimizers of the least squares criterion with linear
constraints and/or quadratic penalties. As special cases of our results, we
derive explicit expressions for the degrees of freedom in many nonparametric
regression problems, e.g., bounded isotonic regression, multivariate
(penalized) convex regression, and additive total variation regularization. Our
theory also yields, as special cases, known results on the degrees of freedom
of many well-studied estimators in the statistics literature, such as ridge
regression, Lasso and generalized Lasso. Our results can be readily used to
choose the tuning parameter(s) involved in the estimation procedure by
minimizing the Stein's unbiased risk estimate. As a by-product of our analysis
we derive an interesting connection between bounded isotonic regression and
isotonic regression on a general partially ordered set, which is of independent
interest.Comment: 72 pages, 7 figures, Journal of the American Statistical Association
(Theory and Methods), 201
Sequential Action-Induced Invariant Representation for Reinforcement Learning
How to accurately learn task-relevant state representations from
high-dimensional observations with visual distractions is a realistic and
challenging problem in visual reinforcement learning. Recently, unsupervised
representation learning methods based on bisimulation metrics, contrast,
prediction, and reconstruction have shown the ability for task-relevant
information extraction. However, due to the lack of appropriate mechanisms for
the extraction of task information in the prediction, contrast, and
reconstruction-related approaches and the limitations of bisimulation-related
methods in domains with sparse rewards, it is still difficult for these methods
to be effectively extended to environments with distractions. To alleviate
these problems, in the paper, the action sequences, which contain
task-intensive signals, are incorporated into representation learning.
Specifically, we propose a Sequential Action--induced invariant Representation
(SAR) method, in which the encoder is optimized by an auxiliary learner to only
preserve the components that follow the control signals of sequential actions,
so the agent can be induced to learn the robust representation against
distractions. We conduct extensive experiments on the DeepMind Control suite
tasks with distractions while achieving the best performance over strong
baselines. We also demonstrate the effectiveness of our method at disregarding
task-irrelevant information by deploying SAR to real-world CARLA-based
autonomous driving with natural distractions. Finally, we provide the analysis
results of generalization drawn from the generalization decay and t-SNE
visualization. Code and demo videos are available at
https://github.com/DMU-XMU/SAR.git
Smoothing Proximal Gradient Method for General Structured Sparse Learning
We study the problem of learning high dimensional regression models
regularized by a structured-sparsity-inducing penalty that encodes prior
structural information on either input or output sides. We consider two widely
adopted types of such penalties as our motivating examples: 1) overlapping
group lasso penalty, based on the l1/l2 mixed-norm penalty, and 2) graph-guided
fusion penalty. For both types of penalties, due to their non-separability,
developing an efficient optimization method has remained a challenging problem.
In this paper, we propose a general optimization approach, called smoothing
proximal gradient method, which can solve the structured sparse regression
problems with a smooth convex loss and a wide spectrum of
structured-sparsity-inducing penalties. Our approach is based on a general
smoothing technique of Nesterov. It achieves a convergence rate faster than the
standard first-order method, subgradient method, and is much more scalable than
the most widely used interior-point method. Numerical results are reported to
demonstrate the efficiency and scalability of the proposed method.Comment: arXiv admin note: substantial text overlap with arXiv:1005.471
Towards Open-Ended Visual Recognition with Large Language Model
Localizing and recognizing objects in the open-ended physical world poses a
long-standing challenge within the domain of machine perception. Recent methods
have endeavored to address the issue by employing a class-agnostic mask (or
box) proposal model, complemented by an open-vocabulary classifier (e.g., CLIP)
using pre-extracted text embeddings. However, it is worth noting that these
open-vocabulary recognition models still exhibit limitations in practical
applications. On one hand, they rely on the provision of class names during
testing, where the recognition performance heavily depends on this predefined
set of semantic classes by users. On the other hand, when training with
multiple datasets, human intervention is required to alleviate the label
definition conflict between them. In this paper, we introduce the OmniScient
Model (OSM), a novel Large Language Model (LLM) based mask classifier, as a
straightforward and effective solution to the aforementioned challenges.
Specifically, OSM predicts class labels in a generative manner, thus removing
the supply of class names during both training and testing. It also enables
cross-dataset training without any human interference, exhibiting robust
generalization capabilities due to the world knowledge acquired from the LLM.
By combining OSM with an off-the-shelf mask proposal model, we present
promising results on various benchmarks, and demonstrate its effectiveness in
handling novel concepts. Code/model are available at
https://github.com/bytedance/OmniScient-Model
- …