Arguments for and against the use of multiple comparison control in stochastic simulation studies

Pick up any of the standard discrete-event simulation textbooks and you will find that the output analysissection includes a note on multiple comparison control (MCC). These procedures aim to mitigate theproblem of inflating the probability of making a single type I error when comparing many simulatedscenarios simultaneously. We consider the use of MCC in stochastic simulation studies and present anargument discouraging its use in the classical sense. In particular, we focus on the impracticality ofprocedures, the benefits of common random numbers and that simulation is very different from empiricalstudies where MCC has its roots. We then consider in what instances would abandoning MCC altogetherbe problematic and what alternatives are available. We present an argument for medium to largeexploratory studies to move their attention away from classical Type I errors and instead control asubtlety different quantity: the rate of false positives amongst all ‘discoveries’

Item statistics derived from three-option versions of multiple-choice questions are usually as robust as four- or five-option versions: implications for exam design.

Different versions of multiple-choice exams were administered to an undergraduate class in human physiology as part of normal testing in the classroom. The goal was to evaluate whether the number of options (possible answers) per question influenced the effectiveness of this assessment. Three exams (each with three versions) were given to each of two sections during an academic quarter. All versions were equally long, with 30 questions: 10 questions with 3 options, 10 questions with 4, and 10 questions with 5 (always one correct answer plus distractors). Each question appeared in all three versions of an exam, with a different number of options in each version (three, four, or five). Discrimination (point biserial and upper-lower discrimination indexes) and difficulty were evaluated for each question. There was a small increase in difficulty (a lower average score on a question) when more options were provided. The upper-lower discrimination index indicated a small improvement in assessment of student learning with more options, although the point biserial did not. The total length of a question (number of words) was associated with a small increase in discrimination and difficulty, independent of the number of options. Quantitative questions were more likely to show an increase in discrimination with more options than nonquantitative questions, but this effect was very small. Therefore, for these testing conditions, there appears to be little advantage in providing more than three options per multiple-choice question, and there are disadvantages, such as needing more time for an exam

Rating and ranking of multiple-aspect alternatives using fuzzy sets

A method is proposed to deal with multiple-alternative decision problems under uncertainty. It is assumed that all the alternatives in the choice set can be characterized by a number of aspects, and that information is available to assign weights to these aspects and to construct a rating scheme for the various aspects of each alternative. The method basically consists of computing weighted final ratings for each alternative and comparing the weighted final ratings. The uncertainty that is assumed to be inherent in the assessments of the ratings and weights is accounted for by considering each of these variables as fuzzy quantities, characterized by appropriate membership functions. Accordingly, the final evaluation of the alternatives consists of a degree of membership in the fuzzy set of alternatives ranking first. A practical method is given to compute membership functions of fuzzy sets induced by mappings, and applied to the problem at hand. A number of examples are worked out. The method is compared to another one proposed by Kahne who approaches the problem probabilistically

Kolmogorov-Smirnov Tests For Distribution Function Similarity With Applications To Portfolios of Common Stock

If the elements of the choice set in a decision model involving randomness are not arbitrary, but restricted appropriately, an expected utility ordering of them can be represented by a mean standard deviation ranking function. These restrictions can apply to the form of, or can specify relationships among, the distribution functions. A particularly useful restriction is one which requires that elements in the choice set, when normalized to have a zero mean and unit variance, be identically distributed. No restriction is placed on the form of any individual distribution function. This research empirically tests for this and other useful restrictions on the relationships among the elements of a set of random variables. Observations from the random variables are used to test whether or not they have distribution functions which are appropriately related to one another. The tests are applied to rate of return data for portfolios of common stock. The tests indicate that one cannot reject the hypothesis that the distribution functions of these portfolios are sufficiently similar to imply that the efficient set of portfolios for any risk averse expected utility maximizer is contained in the mean-standard deviation efficient set.

Asymptotic Validity of the Bayes-Inspired Indifference Zone Procedure: The Non-Normal Known Variance Case

We consider the indifference-zone (IZ) formulation of the ranking and selection problem in which the goal is to choose an alternative with the largest mean with guaranteed probability, as long as the difference between this mean and the second largest exceeds a threshold. Conservatism leads classical IZ procedures to take too many samples in problems with many alternatives. The Bayes-inspired Indifference Zone (BIZ) procedure, proposed in Frazier (2014), is less conservative than previous procedures, but its proof of validity requires strong assumptions, specifically that samples are normal, and variances are known with an integer multiple structure. In this paper, we show asymptotic validity of a slight modification of the original BIZ procedure as the difference between the best alternative and the second best goes to zero,when the variances are known and finite, and samples are independent and identically distributed, but not necessarily normal

ISER Working Paper

In communities that largely depend on the extraction of natural resources, attitudes towards conservation and development may seem at odds or particularly rigid. With an unprecedented wealth of natural capital, a growing mining sector, strong oil and gas industry, and a politically conservative population, Alaska serves as a case study to measure such attitudes. This research was motivated by a lack of primary ecosystem service valuation studies in Alaska that could be used to assess the public’s perceived value of ecosystem services in order to guide future land use decisions and incentivize land use decisions that minimize negative externalities. A choice experiment was conducted with 224 households in the Matanuska-Susitna Borough, the fastest growing region in Alaska and one of the fastest growing regions in the U.S. Rapid development with few restrictions has led to changes for local ecosystems particularly important to salmon, negative effects on access related to recreation and tourism, and caused conversion of valuable farmland. Study results show that attitudes and values vary regarding future land use and economic development efforts. On average, policy action to improve conditions for local salmon stocks are most valuable to local residents followed by protecting farm and ranch lands as well as public access to recreation sites. Conversely, residents show negative preferences towards rapid population growth and developing local mining, oil and gas, and timber resources but support developing a professional and technical services sector. The quantified welfare changes related to different development scenarios show that focusing on conserving valuable ecosystem services is in the public’s best interest

Combining and Comparing Consumers' Stated Preference Ratings and Choice Responses

In this study we develop and test an econometric model for combining choice and preference ratings data collected from the same set of individuals.Choice data are modeled using a multinomial logit framework, while preference data are modeled using an ordered response equation.Individual heterogeneity is allowed for via random coefficients providing a link between the choice and ratings data.Parameters are estimated by Simulated Maximum Likelihood.An application of the model to consumer yoghurt choice in The Netherlands found that ratings based preference estimates differ significantly from choice based estimates, but the correlation between random coefficients driving the two is very strong.econometric models;preferences;consumer choice;maximum likelihood;JEL classifications;C35;M31