Validation of Trace-Driven Simulation Models: A Novel Regression Test

This paper argues that it is wrong to require that regressing the outputs of a trace-driven simulation on the observed real outcomes should give a 45° (unit slope) line through the origin (zero intercept). This note proposes instead an alternative requirement: the responses of the simulated and the real systems should have the same means and the same variances. To test statistically whether this requirement is satisfied, a novel procedure is derived: regress the differences between simulated and real responses on their associated sums, and test whether the resulting intercept and slope are both zero. This novel but simple test assumes identically, independently, and normally distributed outputs of the real system and the simulated system. The old and the new procedures are investigated in extensive Monte Carlo experiments that simulate M/M/1 queueing systems. The conclusions are: (i) the naive intuitive test rejects a valid simulation model substantially more often than the novel test does; (ii) the naive test shows "perverse" behavior within a certain domain: the worse the simulation model, the higher its estimated probability of acceptance; and (iii) the novel test does not reject a valid simulation model too often (its type I error probability is correct), provided the queueing response is transformed appropriately to obtain (nearly) normally distributed responses.Correlation, Paired Observations, Goodness-of-Fit, Type II Error, Power, Simultaneous Tests

An upper bound for sstress

Nonmetric scaling, multidimensional scaling, loss functions, distance geometry,

Full Screening Another View at One-Factor-At-A-Time Designs

Textbooks on Design Of Experiments invariably start by explaining why one-factor-at-a -time (OAT) is an inferior method. Here we will show that in a model with all interactions a variant of OAT is extremely efficient, provided that we only have non-negative parameters and that there are only a few large parameters. In the extreme, this means that there is one positive parameter. In that case, for m variables, that is, for 2m-1 parameters, the procedure only needs m+2 observations to find. In other examined cases, the proposed method is also very fast.

Validation of trace-driven simulation models: bootstrap tests

Trace-driven (or correlated inspection) simulation means that the simulated and the real systems have some common inputs (say, historical arrival times) so that the two systems' outputs are cross-correlated. To validate such a simulation, this paper focuses on the difference between the average simulated and real responses. To evaluate this validation statistic, the paper develops a novel bootstrap technique--based on replicated runs. This validation statistic and the bootstrap technique are evaluated in extensive Monte Carlo experiments with specific single-server queues. These experiments show acceptable Type-I and Type-II error probabilities

Finding the Important Factors in Large Discrete-Event Simulation: Sequential Bifurcation and its Applications

This contribution discusses experiments with many factors: the case study includes a simulation model with 92 factors.The experiments are guided by sequential bifurcation.This method is most efficient and effective if the true input/output behavior of the simulation model can be approximated through a first-order polynomial possibly augmented with two-factor interactions.The method is explained and illustrated through three related discrete-event simulation models.These models represent three supply chain configurations, studied for an Ericsson factory in Sweden.After simulating 21 scenarios (factor combinations) each replicated five times to account for noise a shortlist with the 11 most important factors is identified for the biggest of the three simulation models.