7,086 research outputs found
Uncertainty Quantification in Lasso-Type Regularization Problems
Regularization techniques, which sit at the interface of statistical modeling and machine learning, are often used in the engineering or other applied sciences to tackle high dimensional regression (type) problems. While a number of regularization methods are commonly used, the 'Least Absolute Shrinkage and Selection Operator' or simply LASSO is popular because of its efficient variable selection property. This property of the LASSO helps to deal with problems where the number of predictors is larger than the total number of observations, as it shrinks the coefficients of non-important parameters to zero. In this chapter, both frequentist and Bayesian approaches for the LASSO are discussed, with particular attention to the problem of uncertainty quantification of regression parameters. For the frequentist approach, we discuss a refit technique as well as the classical bootstrap method, and for the Bayesian method, we make use of the equivalent LASSO formulation using a Laplace prior on the model parameters
Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery
Sparse model identification enables nonlinear dynamical system discovery from
data. However, the control of false discoveries for sparse model identification
is challenging, especially in the low-data and high-noise limit. In this paper,
we perform a theoretical study on ensemble sparse model discovery, which shows
empirical success in terms of accuracy and robustness to noise. In particular,
we analyse the bootstrapping-based sequential thresholding least-squares
estimator. We show that this bootstrapping-based ensembling technique can
perform a provably correct variable selection procedure with an exponential
convergence rate of the error rate. In addition, we show that the ensemble
sparse model discovery method can perform computationally efficient uncertainty
estimation, compared to expensive Bayesian uncertainty quantification methods
via MCMC. We demonstrate the convergence properties and connection to
uncertainty quantification in various numerical studies on synthetic sparse
linear regression and sparse model discovery. The experiments on sparse linear
regression support that the bootstrapping-based sequential thresholding
least-squares method has better performance for sparse variable selection
compared to LASSO, thresholding least-squares, and bootstrapping-based LASSO.
In the sparse model discovery experiment, we show that the bootstrapping-based
sequential thresholding least-squares method can provide valid uncertainty
quantification, converging to a delta measure centered around the true value
with increased sample sizes. Finally, we highlight the improved robustness to
hyperparameter selection under shifting noise and sparsity levels of the
bootstrapping-based sequential thresholding least-squares method compared to
other sparse regression methods.Comment: 32 pages, 7 figure
Uncertainty quantification in lasso-type regularization problems
Regularization techniques, which sit at the interface of statistical modeling and machine learning, are often used in the engineering or other applied sciences to tackle high dimensional regression (type) problems. While a number of regularization methods are commonly used, the 'Least Absolute Shrinkage and Selection Operator' or simply LASSO is popular because of its efficient variable selection property. This property of the LASSO helps to deal with problems where the number of predictors is larger than the total number of observations, as it shrinks the coefficients of non-important parameters to zero. In this chapter, both frequentist and Bayesian approaches for the LASSO are discussed, with particular attention to the problem of uncertainty quantification of regression parameters. For the frequentist approach, we discuss a refit technique as well as the classical bootstrap method, and for the Bayesian method, we make use of the equivalent LASSO formulation using a Laplace prior on the model parameters
Comparison of Variable Selection Methods
Use of classic variable selection methods in public health research is quite common. Many criteria, and various strategies for applying them, now exist including forward selection, backward elimination, stepwise selection, best-subset selection and so on, but all suffer from similar drawbacks. Chief among them is a failure to account for the uncertainty contained in the model selection process. Ignoring model uncertainty can cause several serious problems. Variance estimates are generally underestimated, p-values are generally inflated, prediction ability is overestimated, and results are not reproducible in another dataset. Modern variable selection methods have become increasingly popular, especially in applications of high-dimensional or sparse data. Some of these methods were developed to address the short-comings of classic variable selection methods, such as backward elimination and stepwise selection methods. However, it remains unclear how modern variable selection methods behave in a classical, meaning non-high-dimensional, setting. A simulation study investigates the estimation, predictive performance and variable selection capabilities of three representative modern variable selection methods: Bayesian model averaging (BMA), stochastic search variable selection (SSVS), and the adaptive lasso. These three methods are considered in the setting of linear regression with a single variable of interest which is always included in the model. A second simulation study compares BMA to classical variable selection methods, including backward elimination, two-stage method, and change-in-effect method in the setting of logistic regression. Additionally, the data generated in both simulation studies closely mimic a real study and reflect a realistic correlation structure between potential covariates. Sample sizes ranging from 150 to 20000 are investigated. BMA is demonstrated in an example building a predictive model using data from the China Health and Nutrition Survey.Doctor of Public Healt
Essays on Robust Model Selection and Model Averaging for Linear Models
Model selection is central to all applied statistical work.
Selecting the variables for use in a regression model is one
important example of model selection. This thesis is a collection
of essays on robust model selection procedures and model
averaging for linear regression models.
In the first essay, we propose robust Akaike information criteria
(AIC) for MM-estimation and an adjusted robust scale based AIC
for M and MM-estimation. Our proposed model selection criteria
can maintain their robust properties in the presence of a high
proportion of outliers and the outliers in the covariates. We
compare our proposed criteria with other robust model selection
criteria discussed in previous literature. Our simulation studies
demonstrate a significant outperformance of robust AIC based on
MM-estimation in the presence of outliers in the covariates. The
real data example also shows a better performance of robust AIC
based on MM-estimation.
The second essay focuses on robust versions of the ``Least
Absolute Shrinkage and Selection Operator" (lasso). The adaptive
lasso is a method for performing simultaneous parameter
estimation and variable selection. The adaptive weights used in
its penalty term mean that the adaptive lasso achieves the oracle
property. In this essay, we propose an extension of the adaptive
lasso named the Tukey-lasso. By using Tukey's biweight criterion,
instead of squared loss, the Tukey-lasso is resistant to outliers
in both the response and covariates. Importantly, we demonstrate
that the Tukey-lasso also enjoys the oracle property. A fast
accelerated proximal gradient (APG) algorithm is proposed and
implemented for computing the Tukey-lasso. Our extensive
simulations show that the Tukey-lasso, implemented with the APG
algorithm, achieves very reliable results, including for
high-dimensional data where p>n. In the presence of outliers, the
Tukey-lasso is shown to offer substantial improvements in
performance compared to the adaptive lasso and other robust
implementations of the lasso. Real data examples further
demonstrate the utility of the Tukey-lasso.
In many statistical analyses, a single model is used for
statistical inference, ignoring the process that leads to the
model being selected. To account for this model uncertainty, many
model averaging procedures have been proposed. In the last essay,
we propose an extension of a bootstrap model averaging approach,
called bootstrap lasso averaging (BLA). BLA utilizes the lasso
for model selection. This is in contrast to other forms of
bootstrap model averaging that use AIC or Bayesian information
criteria (BIC). The use of the lasso improves the computation
speed and allows BLA to be applied even when the number of
variables p is larger than the sample size n. Extensive
simulations confirm that BLA has outstanding finite sample
performance, in terms of both variable and prediction accuracies,
compared with traditional model selection and model averaging
methods. Several real data examples further demonstrate an
improved out-of-sample predictive performance of BLA
- …