Ranking and Selection under Input Uncertainty: Fixed Confidence and Fixed Budget

In stochastic simulation, input uncertainty (IU) is caused by the error in estimating the input distributions using finite real-world data. When it comes to simulation-based Ranking and Selection (R&S), ignoring IU could lead to the failure of many existing selection procedures. In this paper, we study R&S under IU by allowing the possibility of acquiring additional data. Two classical R&S formulations are extended to account for IU: (i) for fixed confidence, we consider when data arrive sequentially so that IU can be reduced over time; (ii) for fixed budget, a joint budget is assumed to be available for both collecting input data and running simulations. New procedures are proposed for each formulation using the frameworks of Sequential Elimination and Optimal Computing Budget Allocation, with theoretical guarantees provided accordingly (e.g., upper bound on the expected running time and finite-sample bound on the probability of false selection). Numerical results demonstrate the effectiveness of our procedures through a multi-stage production-inventory problem

Robust Prediction Error Estimation with Monte-Carlo Methodology

In predictive modeling with simulation or machine learning, it is critical to assess the quality of estimated values through output analysis accurately. In recent decades output analysis has become enriched with methods that quantify the impact of input data uncertainty in the model outputs to increase robustness. However, most developments apply when the input data can be parametrically parameterized. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling. This framework provides nonparametric quantification of the variance and bias induced in the outputs with higher-order accuracy. Our new bias-corrected estimation from the model outputs leverages the extension of fast iterative bootstrap sampling and higher-order influence functions. For the scalability of the proposed estimation methods, we devise budget-optimal rules and leverage control variates for variance reduction. Our numerical results demonstrate a clear advantage in building better and more robust confidence intervals for both simulation and machine learning frameworks

Sequential Design for Gaussian Process Surrogates in Noisy Level Set Estimation

We consider the problem of learning the level set for which a noisy black-box function exceeds a given threshold. To efficiently reconstruct the level set, we investigate Gaussian process (GP) metamodels and sequential design frameworks. Our focus is on strongly stochastic samplers, in particular with heavy-tailed simulation noise and low signal-to-noise ratio. We introduce the use of four GP-based metamodels in level set estimation that are robust to noise misspecification, and evaluate the performance of them. In conjunction with these metamodels, we develop several acquisition functions for guiding the sequential experimental designs, extending existing stepwise uncertainty reduction criteria to the stochastic contour-finding context. This also motivates our development of (approximate) updating formulas to efficiently compute such acquisition functions for the proposed metamodels. To expedite sequential design in stochastic experiments, we also develop adaptive batching designs, which are natural extensions of sequential design heuristics with the benefit of replication growing as response features are learned, inputs concentrate, and the metamodeling overhead rises. We develop four novel schemes that simultaneously or sequentially determine the sequential design inputs and the respective number of replicates. Our schemes are benchmarked by using synthetic examples and an application in quantitative finance (Bermudan option pricing)

Capturing Risk in Capital Budgeting

NPS NRP Technical ReportThis proposed research has the goal of proposing novel, reusable, extensible, adaptable, and comprehensive advanced analytical process and Integrated Risk Management to help the (DOD) with risk-based capital budgeting, Monte Carlo risk-simulation, predictive analytics, and stochastic optimization of acquisitions and programs portfolios with multiple competing stakeholders while subject to budgetary, risk, schedule, and strategic constraints. The research covers topics of traditional capital budgeting methodologies used in industry, including the market, cost, and income approaches, and explains how some of these traditional methods can be applied in the DOD by using DOD-centric non-economic, logistic, readiness, capabilities, and requirements variables. Stochastic portfolio optimization with dynamic simulations and investment efficient frontiers will be run for the purposes of selecting the best combination of programs and capabilities is also addressed, as are other alternative methods such as average ranking, risk metrics, lexicographic methods, PROMETHEE, ELECTRE, and others. The results include actionable intelligence developed from an analytically robust case study that senior leadership at the DOD may utilize to make optimal decisions. The main deliverables will be a detailed written research report and presentation brief on the approach of capturing risk and uncertainty in capital budgeting analysis. The report will detail the proposed methodology and applications, as well as a summary case study and examples of how the methodology can be applied.N8 - Integration of Capabilities & ResourcesThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

Policy design in a model with swings in risk appetite

This paper studies the policy implications of habits and cyclical changes in agents' appetite for risk-taking. To do so, it analyses the non-linear solution of a New Keynesian (NK) model, in which slow-moving habits help match the cyclical properties of risk-premia. Our findings suggest that the presence of habits and swings in risk appetite can materially affect policy prescriptions. As in Ljungqvist and Uhlig (2000), a counter-cyclical fiscal instrument can eliminate habit-related externalities. Alternatively, monetary policy can partially curb the associated overconsumption by responding to risk premia. Specifically, periods in which risk premia are elevated (compressed) merit a looser (tighter) policy stance. However, the associated welfare gains appear quantitatively small