151 research outputs found
Improved conformalized quantile regression
Conformalized quantile regression is a procedure that inherits the advantages
of conformal prediction and quantile regression. That is, we use quantile
regression to estimate the true conditional quantile and then apply a conformal
step on a calibration set to ensure marginal coverage. In this way, we get
adaptive prediction intervals that account for heteroscedasticity. However, the
aforementioned conformal step lacks adaptiveness as described in (Romano et
al., 2019). To overcome this limitation, instead of applying a single conformal
step after estimating conditional quantiles with quantile regression, we
propose to cluster the explanatory variables weighted by their permutation
importance with an optimized k-means and apply k conformal steps. To show that
this improved version outperforms the classic version of conformalized quantile
regression and is more adaptive to heteroscedasticity, we extensively compare
the prediction intervals of both in open datasets.Comment: 11 pages, 10 figure
Probabilistic Load Forecasting with Deep Conformalized Quantile Regression
The establishment of smart grids and the introduction of distributed generation posed new challenges in energy analytics that can be tackled with machine learning algorithms. The latter, are able to handle a combination of weather and consumption data, grid measurements, and their historical records to compute inference and make predictions. An accurate energy load forecasting is essential to assure reliable grid operation and power provision at peak times when power consumption is high. However, most of the existing load forecasting algorithms provide only point estimates or probabilistic forecasting methods that construct prediction intervals without coverage guarantee. Nevertheless, information about uncertainty and prediction intervals is very useful to grid operators to evaluate the reliability of operations in the power network and to enable a risk-based
strategy for configuring the grid over a conservative one.
There are two popular statistical methods used to generate prediction intervals in regression tasks: Quantile regression is a non-parametric probabilistic forecasting technique producing prediction intervals adaptive to local variability within the data by estimating quantile functions directly from the data. However, the actual coverage of the prediction intervals obtained via quantile regression is not guaranteed to satisfy the designed
coverage level for finite samples. Conformal prediction is an on-top probabilistic forecasting framework producing symmetric prediction intervals, most often with a fixed length, guaranteed to marginally satisfy the designed coverage level for finite samples.
This thesis proposes a probabilistic load forecasting method for constructing marginally valid prediction intervals adaptive to local variability and suitable for data characterized by temporal dependencies. The method is applied in conjunction with recurrent neural networks, deep learning architectures for sequential data, which are mostly used to compute point forecasts rather than probabilistic forecasts. Specifically, the use of an ensemble of pinball-loss guided deep neural networks performing quantile regression is used together with conformal prediction to address the individual shortcomings of both techniques
Efficiency of conformalized ridge regression
Conformal prediction is a method of producing prediction sets that can be
applied on top of a wide range of prediction algorithms. The method has a
guaranteed coverage probability under the standard IID assumption regardless of
whether the assumptions (often considerably more restrictive) of the underlying
algorithm are satisfied. However, for the method to be really useful it is
desirable that in the case where the assumptions of the underlying algorithm
are satisfied, the conformal predictor loses little in efficiency as compared
with the underlying algorithm (whereas being a conformal predictor, it has the
stronger guarantee of validity). In this paper we explore the degree to which
this additional requirement of efficiency is satisfied in the case of Bayesian
ridge regression; we find that asymptotically conformal prediction sets differ
little from ridge regression prediction intervals when the standard Bayesian
assumptions are satisfied.Comment: 22 pages, 1 figur
Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting
This article presents a novel probabilistic forecasting method called ensemble conformalized quantile regression (EnCQR). EnCQR constructs distribution-free and approximately marginally valid prediction intervals (PIs), which are suitable for nonstationary and heteroscedastic time series data. EnCQR can be applied on top of a generic forecasting model, including deep learning architectures. EnCQR exploits a bootstrap ensemble estimator, which enables the use of conformal predictors for time series by removing the requirement of data exchangeability. The ensemble learners are implemented as generic machine learning algorithms performing quantile regression (QR), which allow the length of the PIs to adapt to local variability in the data. In the experiments, we predict time series characterized by a different amount of heteroscedasticity. The results demonstrate that EnCQR outperforms models based only on QR or conformal prediction (CP), and it provides sharper, more informative, and valid PIs
Conformal Off-Policy Evaluation in Markov Decision Processes
Reinforcement Learning aims at identifying and evaluating efficient control
policies from data. In many real-world applications, the learner is not allowed
to experiment and cannot gather data in an online manner (this is the case when
experimenting is expensive, risky or unethical). For such applications, the
reward of a given policy (the target policy) must be estimated using historical
data gathered under a different policy (the behavior policy). Most methods for
this learning task, referred to as Off-Policy Evaluation (OPE), do not come
with accuracy and certainty guarantees. We present a novel OPE method based on
Conformal Prediction that outputs an interval containing the true reward of the
target policy with a prescribed level of certainty. The main challenge in OPE
stems from the distribution shift due to the discrepancies between the target
and the behavior policies. We propose and empirically evaluate different ways
to deal with this shift. Some of these methods yield conformalized intervals
with reduced length compared to existing approaches, while maintaining the same
certainty level
Deep neural networks for the quantile estimation of regional renewable energy production
Wind and solar energy forecasting have become crucial for the inclusion of renewable energy in electrical power systems. Although most works have focused on point prediction, it is currently becoming important to also estimate the forecast uncertainty. With regard to forecasting methods, deep neural networks have shown good performance in many fields. However, the use of these networks for comparative studies of probabilistic forecasts of renewable energies, especially for regional forecasts, has not yet received much attention. The aim of this article is to study the performance of deep networks for estimating multiple conditional quantiles on regional renewable electricity production and compare them with widely used quantile regression methods such as the linear, support vector quantile regression, gradient boosting quantile regression, natural gradient boosting and quantile regression forest methods. A grid of numerical weather prediction variables covers the region of interest. These variables act as the predictors of the regional model. In addition to quantiles, prediction intervals are also constructed, and the models are evaluated using different metrics. These prediction intervals are further improved through an adapted conformalized quantile regression methodology. Overall, the results show that deep networks are the best performing method for both solar and wind energy regions, producing narrow prediction intervals with good coverage
Sparse Quantile Regression
We consider both -penalized and -constrained quantile
regression estimators. For the -penalized estimator, we derive an
exponential inequality on the tail probability of excess quantile prediction
risk and apply it to obtain non-asymptotic upper bounds on the mean-square
parameter and regression function estimation errors. We also derive analogous
results for the -constrained estimator. The resulting rates of
convergence are nearly minimax-optimal and the same as those for
-penalized estimators. Further, we characterize expected Hamming loss
for the -penalized estimator. We implement the proposed procedure via
mixed integer linear programming and also a more scalable first-order
approximation algorithm. We illustrate the finite-sample performance of our
approach in Monte Carlo experiments and its usefulness in a real data
application concerning conformal prediction of infant birth weights (with
and up to ). In sum, our -based method
produces a much sparser estimator than the -penalized approach
without compromising precision.Comment: 45 pages, 3 figures, 2 table
Conformalized Multimodal Uncertainty Regression and Reasoning
This paper introduces a lightweight uncertainty estimator capable of
predicting multimodal (disjoint) uncertainty bounds by integrating conformal
prediction with a deep-learning regressor. We specifically discuss its
application for visual odometry (VO), where environmental features such as
flying domain symmetries and sensor measurements under ambiguities and
occlusion can result in multimodal uncertainties. Our simulation results show
that uncertainty estimates in our framework adapt sample-wise against
challenging operating conditions such as pronounced noise, limited training
data, and limited parametric size of the prediction model. We also develop a
reasoning framework that leverages these robust uncertainty estimates and
incorporates optical flow-based reasoning to improve prediction prediction
accuracy. Thus, by appropriately accounting for predictive uncertainties of
data-driven learning and closing their estimation loop via rule-based
reasoning, our methodology consistently surpasses conventional deep learning
approaches on all these challenging scenarios--pronounced noise, limited
training data, and limited model size-reducing the prediction error by 2-3x
- …