23,080 research outputs found
A Generic Path Algorithm for Regularized Statistical Estimation
Regularization is widely used in statistics and machine learning to prevent
overfitting and gear solution towards prior information. In general, a
regularized estimation problem minimizes the sum of a loss function and a
penalty term. The penalty term is usually weighted by a tuning parameter and
encourages certain constraints on the parameters to be estimated. Particular
choices of constraints lead to the popular lasso, fused-lasso, and other
generalized penalized regression methods. Although there has been a lot
of research in this area, developing efficient optimization methods for many
nonseparable penalties remains a challenge. In this article we propose an exact
path solver based on ordinary differential equations (EPSODE) that works for
any convex loss function and can deal with generalized penalties as well
as more complicated regularization such as inequality constraints encountered
in shape-restricted regressions and nonparametric density estimation. In the
path following process, the solution path hits, exits, and slides along the
various constraints and vividly illustrates the tradeoffs between goodness of
fit and model parsimony. In practice, the EPSODE can be coupled with AIC, BIC,
or cross-validation to select an optimal tuning parameter. Our
applications to generalized regularized generalized linear models,
shape-restricted regressions, Gaussian graphical models, and nonparametric
density estimation showcase the potential of the EPSODE algorithm.Comment: 28 pages, 5 figure
Gaussian processes with linear operator inequality constraints
This paper presents an approach for constrained Gaussian Process (GP)
regression where we assume that a set of linear transformations of the process
are bounded. It is motivated by machine learning applications for
high-consequence engineering systems, where this kind of information is often
made available from phenomenological knowledge. We consider a GP over
functions on taking values in
, where the process is still Gaussian when
is a linear operator. Our goal is to model under the
constraint that realizations of are confined to a convex set of
functions. In particular, we require that , given
two functions and where pointwise. This formulation provides a
consistent way of encoding multiple linear constraints, such as
shape-constraints based on e.g. boundedness, monotonicity or convexity. We
adopt the approach of using a sufficiently dense set of virtual observation
locations where the constraint is required to hold, and derive the exact
posterior for a conjugate likelihood. The results needed for stable numerical
implementation are derived, together with an efficient sampling scheme for
estimating the posterior process.Comment: Published in JMLR: http://jmlr.org/papers/volume20/19-065/19-065.pd
Estimation in high dimensions: a geometric perspective
This tutorial provides an exposition of a flexible geometric framework for
high dimensional estimation problems with constraints. The tutorial develops
geometric intuition about high dimensional sets, justifies it with some results
of asymptotic convex geometry, and demonstrates connections between geometric
results and estimation problems. The theory is illustrated with applications to
sparse recovery, matrix completion, quantization, linear and logistic
regression and generalized linear models.Comment: 56 pages, 9 figures. Multiple minor change
Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems
Learning-based control algorithms require data collection with abundant
supervision for training. Safe exploration algorithms ensure the safety of this
data collection process even when only partial knowledge is available. We
present a new approach for optimal motion planning with safe exploration that
integrates chance-constrained stochastic optimal control with dynamics learning
and feedback control. We derive an iterative convex optimization algorithm that
solves an \underline{Info}rmation-cost \underline{S}tochastic
\underline{N}onlinear \underline{O}ptimal \underline{C}ontrol problem
(Info-SNOC). The optimization objective encodes both optimal performance and
exploration for learning, and the safety is incorporated as distributionally
robust chance constraints. The dynamics are predicted from a robust regression
model that is learned from data. The Info-SNOC algorithm is used to compute a
sub-optimal pool of safe motion plans that aid in exploration for learning
unknown residual dynamics under safety constraints. A stable feedback
controller is used to execute the motion plan and collect data for model
learning. We prove the safety of rollout from our exploration method and
reduction in uncertainty over epochs, thereby guaranteeing the consistency of
our learning method. We validate the effectiveness of Info-SNOC by designing
and implementing a pool of safe trajectories for a planar robot. We demonstrate
that our approach has higher success rate in ensuring safety when compared to a
deterministic trajectory optimization approach.Comment: Submitted to RA-L 2020, review-
- …