Robust importance-weighted cross-validation under sample selection bias

Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights

Robust importance-weighted cross-validation under sample selection bias

Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights

Multifidelity Monte Carlo estimation for large-scale uncertainty propagation

One important task of uncertainty quantification is propagating input uncertainties through a system of interest to quantify the uncertainties’ effects on the system outputs; however, numerical methods for uncertainty propagation are often based on Monte Carlo estimation, which can require large numbers of numerical simulations of the numerical model describing the system response to obtain estimates with acceptable accuracies. Thus, if the model is computationally expensive to evaluate, then Monte-Carlo-based uncertainty propagation methods can quickly become computationally intractable. We demonstrate that multifidelity methods can significantly speedup uncertainty propagation by leveraging low-cost low-fidelity models and establish accuracy guarantees by using occasional recourse to the expensive high-fidelity model. We focus on the multifidelity Monte Carlo method, which is a multifidelity approach that optimally distributes work among the models such that the mean-squared error of the multifidelity estimator is minimized for a given computational budget. The multifidelity Monte Carlo method is applicable to general types of low-fidelity models, including projection-based reduced models, data-fit surrogates, response surfaces, and simplified-physics models. We apply the multifidelity Monte Carlo method to a coupled aero-structural analysis of a wing and a flutter problem with a high-aspect-ratio wing. The low-fidelity models are data-fit surrogate models derived with standard procedures that are built in common software environments such as Matlab and numpy/scipy. Our results demonstrate speedups of orders of magnitude compared to using the high-fidelity model alone.United States. Air Force. Office of Scientific Research. Multidisciplinary University Research Initiative (Award FA9550-15-1-0038

Generalized information reuse for optimization under uncertainty with non-sample average estimators

In optimization under uncertainty for engineering design, the behavior of the system outputs due to uncertain inputs needs to be quantified at each optimization iteration, but this can be computationally expensive. Multi-fidelity techniques can significantly reduce the computational cost of Monte Carlo sampling methods for quantifying the effect of uncertain inputs, but existing multi-fidelity techniques in this context apply only to Monte Carlo estimators that can be expressed as a sample average, such as estimators of statistical moments. Information reuse is a particular multi-fidelity method that treats previous optimization iterations as lower-fidelity models. This work generalizes information reuse to be applicable to quantities with non-sample average estimators. The extension makes use of bootstrapping to estimate the error of estimators and the covariance between estimators at different fidelities. Specifically, the horsetail matching metric and quantile function are considered as quantities whose estimators are not sample-averages. In an optimization under uncertainty for an acoustic horn design problem, generalized information reuse demonstrated computational savings of over 60% compared to regular Monte Carlo sampling

Multifidelity approaches for optimization under uncertainty

It is important to design robust and reliable systems by accounting for uncertainty and variability in the design process. However, performing optimization in this setting can be computationally expensive, requiring many evaluations of the numerical model to compute statistics of the system performance at every optimization iteration. This paper proposes a multifidelity approach to optimization under uncertainty that makes use of inexpensive, low-fidelity models to provide approximate information about the expensive, high-fidelity model. The multifidelity estimator is developed based on the control variate method to reduce the computational cost of achieving a specified mean square error in the statistic estimate. The method optimally allocates the computational load between the two models based on their relative evaluation cost and the strength of the correlation between them. This paper also develops an information reuse estimator that exploits the autocorrelation structure of the high-fidelity model in the design space to reduce the cost of repeatedly estimating statistics during the course of optimization. Finally, a combined estimator incorporates the features of both the multifidelity estimator and the information reuse estimator. The methods demonstrate 90% computational savings in an acoustic horn robust optimization example and practical design turnaround time in a robust wing optimization problem.United States. Air Force Office of Scientific Research. Multidisciplinary University Research Initiative (Uncertainty Quantification Grant FA9550-09-0613

Optimal Model Management for Multifidelity Monte Carlo Estimation

This work presents an optimal model management strategy that exploits multifidelity surrogate models to accelerate the estimation of statistics of outputs of computationally expensive high-fidelity models. Existing acceleration methods typically exploit a multilevel hierarchy of surrogate models that follow a known rate of error decay and computational costs; however, a general collection of surrogate models, which may include projection-based reduced models, data-fit models, support vector machines, and simplified-physics models, does not necessarily give rise to such a hierarchy. Our multifidelity approach provides a framework to combine an arbitrary number of surrogate models of any type. Instead of relying on error and cost rates, an optimization problem balances the number of model evaluations across the high-fidelity and surrogate models with respect to error and costs. We show that a unique analytic solution of the model management optimization problem exists under mild conditions on the models. Our multifidelity method makes occasional recourse to the high-fidelity model; in doing so it provides an unbiased estimator of the statistics of the high-fidelity model, even in the absence of error bounds and error estimators for the surrogate models. Numerical experiments with linear and nonlinear examples show that speedups by orders of magnitude are obtained compared to Monte Carlo estimation that invokes a single model only

Reinforcement Learning for Machine Translation: from Simulations to Real-World Applications

If a machine translation is wrong, how we can tell the underlying model to fix it? Answering this question requires (1) a machine learning algorithm to define update rules, (2) an interface for feedback to be submitted, and (3) expertise on the side of the human who gives the feedback. This thesis investigates solutions for machine learning updates, the suitability of feedback interfaces, and the dependency on reliability and expertise for different types of feedback. We start with an interactive online learning scenario where a machine translation (MT) system receives bandit feedback (i.e. only once per source) instead of references for learning. Policy gradient algorithms for statistical and neural MT are developed to learn from absolute and pairwise judgments. Our experiments on domain adaptation with simulated online feedback show that the models can largely improve under weak feedback, with variance reduction techniques being very effective. In production environments offline learning is often preferred over online learning. We evaluate algorithms for counterfactual learning from human feedback in a study on eBay product title translations. Feedback is either collected via explicit star ratings from users, or implicitly from the user interaction with cross-lingual product search. Leveraging implicit feedback turns out to be more successful due to lower levels of noise. We compare the reliability and learnability of absolute Likert-scale ratings with pairwise preferences in a smaller user study, and find that absolute ratings are overall more effective for improvements in down-stream tasks. Furthermore, we discover that error markings provide a cheap and practical alternative to error corrections. In a generalized interactive learning framework we propose a self-regulation approach, where the learner, guided by a regulator module, decides which type of feedback to choose for each input. The regulator is reinforced to find a good trade-off between supervision effect and cost. In our experiments, it discovers strategies that are more efficient than active learning and standard fully supervised learning

Multidelity approaches for design under uncertainty

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2013.This electronic version was submitted and approved by the author's academic department as part of an electronic thesis pilot project. The certified thesis is available in the Institute Archives and Special Collections.Cataloged from department-submitted PDF version of thesis.Includes bibliographical references (p. 113-117).Uncertainties are present in many engineering applications and it is important to account for their effects during engineering design to achieve robust and reliable systems. One approach is to represent uncertainties as random inputs to the numerical model of the system and investigate the probabilistic behaviour of the model outputs. However, performing optimization in this setting can be computationally expensive, requiring many evaluations of the numerical model to compute the statistics of the system metrics, such as the mean and the variance of the system performance. Fortunately, in many engineering applications, there are one or more lower fidelity models that approximate the original (high-fidelity) numerical model at lower computational costs. This thesis presents rigorous multifidelity approaches to leverage cheap low-fidelity models and other approximations of the expensive high-fidelity model to reduce the computational expense of optimization under uncertainty. Solving an optimization under uncertainty problem can require estimates of the statistics at many different design points, incurring a significant number of expensive high-fidelity model evaluations. The multifidelity estimator is developed based on the control variate method to reduce the computational cost of achieving a specified root mean square error in the statistic estimate by making use of the correlation between the outputs of the expensive high-fidelity model and the outputs of the cheap low-fidelity model. The method optimally relegates some of the computational load to the low-fidelity model based on the relative model evaluation cost and the strength of the correlation. It has demonstrated 85% computational savings in an acoustic horn robust optimization example. When the model is sufficiently smooth in the design space in the sense that a small change in the design variables produces a small change in the model outputs, it has an autocorrelation structure that can be exploited by the control variate method. The information reuse estimator is developed to reduce the computational cost of achieving a specified root mean square error in the statistic estimate by making use of the correlation between the high-fidelity model outputs at one design point and those at a previously visited design point. As the optimization progresses towards the optimum in the design space, the steps taken in the design space often become shorter, increasing the correlation and making the information reuse estimator more efficient. To further reduce the computational cost, the combined estimator is developed to incorporate the features of both the multifidelity estimator and the information reuse estimator. It has demonstrated 90% computational savings in the acoustic horn robust optimization example. The methods developed in this thesis are applied to two practical aerospace applications. In conceptual aircraft design, there are often uncertainties about the future developments of the underlying technologies. The information reuse estimator can be used to efficiently generate a Pareto front to study the trade off between the expected performance and the risk induced by the uncertainties in the different aircraft designs. In a large-scale wing robust optimization problem with uncertainties in material properties and flight conditions, the combined estimator demonstrated a reasonable solution turnaround time of 9.7 days on a 16-processor desktop machine, paving the way to a larger scale wing optimization problem with distributed uncertainties to account for degradation or damage.by Leo Wai-Tsun Ng.Ph.D