Asymptotic Validity of the Bayes-Inspired Indifference Zone Procedure: The Non-Normal Known Variance Case

We consider the indifference-zone (IZ) formulation of the ranking and selection problem in which the goal is to choose an alternative with the largest mean with guaranteed probability, as long as the difference between this mean and the second largest exceeds a threshold. Conservatism leads classical IZ procedures to take too many samples in problems with many alternatives. The Bayes-inspired Indifference Zone (BIZ) procedure, proposed in Frazier (2014), is less conservative than previous procedures, but its proof of validity requires strong assumptions, specifically that samples are normal, and variances are known with an integer multiple structure. In this paper, we show asymptotic validity of a slight modification of the original BIZ procedure as the difference between the best alternative and the second best goes to zero,when the variances are known and finite, and samples are independent and identically distributed, but not necessarily normal

Applications of stochastic simulation in two-stage multiple comparisons with the best problem and time average variance constant estimation

In this dissertation, we study two problems. In the first part, we consider the two-stage methods for comparing alternatives using simulation. Suppose there are a finite number of alternatives to compare, with each alternative having an unknown parameter that is the basis for comparison. The parameters are to be estimated using simulation, where the alternatives are simulated independently. We develop two-stage selection and multiple-comparison procedures for simulations under a general framework. The assumptions are that each alternative has a parameter estimation process that satisfies a random- time-change central limit theorem (CLT), and there is a weakly consistent variance estimator (WCVE) for the variance constant appearing in the CLT. The framework encompasses comparing means of independent populations, functions of means, and steady-state means. One problem we consider of considerable practical interest and not handled in previous work on two-stage multiple-comparison procedures is comparing quantiles of alternative populations. We establish the asymptotic validity of our procedures as the prescribed width of the confidence intervals or indifference-zone parameter shrinks to zero. Also, for the steady-state simulation context, we compare our procedures based on WCVEs with techniques that instead use standardized time series methods. In the second part, we propose a new technique of estimating the variance parameter of a wide variety of stochastic processes. This new technique is better than the existing techniques for some standard stochastic processes in terms of bias and variance properties, since it reduces bias at the cost of no significant increase in variance

Advances in ranking and selection: variance estimation and constraints

In this thesis, we first show that the performance of ranking and selection (R&S) procedures in steady-state simulations depends highly on the quality of the variance estimates that are used. We study the performance of R&S procedures using three variance estimators --- overlapping area, overlapping Cramer--von Mises, and overlapping modified jackknifed Durbin--Watson estimators --- that show better long-run performance than other estimators previously used in conjunction with R&S procedures for steady-state simulations. We devote additional study to the development of the new overlapping modified jackknifed Durbin--Watson estimator and demonstrate some of its useful properties. Next, we consider the problem of finding the best simulated system under a primary performance measure, while also satisfying stochastic constraints on secondary performance measures, known as constrained ranking and selection. We first present a new framework that allows certain systems to become dormant, halting sampling for those systems as the procedure continues. We also develop general procedures for constrained R&S that guarantee a nominal probability of correct selection, under any number of constraints and correlation across systems. In addition, we address new topics critical to efficiency of the these procedures, namely the allocation of error between feasibility check and selection, the use of common random numbers, and the cost of switching between simulated systems.Ph.D.Committee Co-chairs: Sigrun Andradottir, Dave Goldsman and Seong-Hee Kim; Committee Members:Shabbir Ahmed and Brani Vidakovi

Economic Analysis of Simulation Selection Problems

Ranking and selection procedures are standard methods for selecting the best of a finite number of simulated design alternatives based on a desired level of statistical evidence for correct selection. But the link between statistical significance and financial significance is indirect, and there has been little or no research into it. This paper presents a new approach to the simulation selection problem, one that maximizes the expected net present value of decisions made when using stochastic simulation. We provide a framework for answering these managerial questions: When does a proposed system design, whose performance is unknown, merit the time and money needed to develop a simulation to infer its performance? For how long should the simulation analysis continue before a design is approved or rejected? We frame the simulation selection problem as a “stoppable” version of a Bayesian bandit problem that treats the ability to simulate as a real option prior to project implementation. For a single proposed system, we solve a free boundary problem for a heat equation that approximates the solution to a dynamic program that finds optimal simulation project stopping times and that answers the managerial questions. For multiple proposed systems, we extend previous Bayesian selection procedures to account for discounting and simulation-tool development costs

'Some tactical problems in digital simulation' for the next 10 years

In his influential 1963 paper ‘Some Tactical Problems in Digital Simulation’, Conway identified important issues that became the pillars of research in simulation analysis methodology. Naturally these ‘problems’ were a product of the applications of interest at the time, as well as the state of simulation and computing, much of which has changed dramatically. In light of those changes, we attempt to identify the tactical problems that might occupy simulation researchers for the next 10 years

Finite Simulation Budget Allocation for Ranking and Selection

We consider a simulation-based ranking and selection (R&S) problem under a fixed budget setting. Existing budget allocation procedures focus either on asymptotic optimality or on one-step-ahead allocation efficiency. Neither of them depends on the fixed simulation budget, the ignorance of which could lead to an inefficient allocation, especially when the simulation budget is finite. In light of this, we develop a finite-budget allocation rule that is adaptive to the simulation budget. Theoretical results show that the budget allocation strategies are distinctively different between a finite budget and a sufficiently large budget. Our proposed allocation rule can dynamically determine the ratio of budget allocated to designs according to different simulation budget and is optimal when the simulation budget goes to infinity, indicating it not only possesses desirable finite-budget properties but also achieves asymptotic optimality. Based on the proposed allocation rule, two efficient finite simulation budget allocation algorithms are developed. In the numerical experiments, we use both synthetic examples and a case study to show the superior efficiency of our proposed allocation rule