Validating simulated interaction for retrieval evaluation

A searcher’s interaction with a retrieval system consists of actions such as query formulation, search result list interaction and document interaction. The simulation of searcher interaction has recently gained momentum in the analysis and evaluation of interactive information retrieval (IIR). However, a key issue that has not yet been adequately addressed is the validity of such IIR simulations and whether they reliably predict the performance obtained by a searcher across the session. The aim of this paper is to determine the validity of the common interaction model (CIM) typically used for simulating multi-query sessions. We focus on search result interactions, i.e., inspecting snippets, examining documents and deciding when to stop examining the results of a single query, or when to stop the whole session. To this end, we run a series of simulations grounded by real world behavioral data to show how accurate and responsive the model is to various experimental conditions under which the data were produced. We then validate on a second real world data set derived under similar experimental conditions. We seek to predict cumulated gain across the session. We find that the interaction model with a query-level stopping strategy based on consecutive non-relevant snippets leads to the highest prediction accuracy, and lowest deviation from ground truth, around 9 to 15% depending on the experimental conditions. To our knowledge, the present study is the first validation effort of the CIM that shows that the model’s acceptance and use is justified within IIR evaluations. We also identify and discuss ways to further improve the CIM and its behavioral parameters for more accurate simulations

Unique bid auctions: Equilibrium solutions and experimental evidence

Two types of auction were introduced on the Internet a few years ago and have rapidly been gaining widespread popularity. In both auctions, players compete for an exogenously determined prize by independently choosing an integer in some finite and common strategy space specified by the auctioneer. In the unique lowest (highest) bid auction, the winner of the prize is the player who submits the lowest (highest) bid, provided that it is unique. We construct the symmetric mixed-strategy equilibrium solutions to the two auctions, and then test them in a sequence of experiments that vary the number of bidders and size of the strategy space. Our results show that the aggregate bids, but only a minority of the individual bidders, are accounted for quite accurately by the equilibrium solutions.

Recall termination in free recall

Although much is known about the dynamics of\ud memory search in the free recall task, relatively little is\ud known about the factors related to recall termination. Rean-\ud alyzing individual trial data from 14 prior studies (1,079\ud participants in 28,015 trials) and defining termination as\ud occurring when a final response is followed by a long\ud nonresponse interval, we observed that termination proba-\ud bility increased throughout the recall period and that retriev-\ud al was more likely to terminate following an error than\ud following a correct response. Among errors, termination\ud probability was higher following prior-list intrusions and\ud repetitions than following extralist intrusions. To verify that\ud this pattern of results can be seen in a single study, we report\ud a new experiment in which 80 participants contributed recall\ud data from a total of 9,122 trials. This experiment replicated\ud the pattern observed in the aggregate analysis of the prior\ud studies.\u

A Statistical View of Learning in the Centipede Game

In this article we evaluate the statistical evidence that a population of students learn about the sub-game perfect Nash equilibrium of the centipede game via repeated play of the game. This is done by formulating a model in which a player's error in assessing the utility of decisions changes as they gain experience with the game. We first estimate parameters in a statistical model where the probabilities of choices of the players are given by a Quantal Response Equilibrium (QRE) (McKelvey and Palfrey, 1995, 1996, 1998), but are allowed to change with repeated play. This model gives a better fit to the data than similar models previously considered. However, substantial correlation of outcomes of games having a common player suggests that a statistical model that captures within-subject correlation is more appropriate. Thus we then estimate parameters in a model which allows for within-player correlation of decisions and rates of learning. Through out the paper we also consider and compare the use of randomization tests and posterior predictive tests in the context of exploratory and confirmatory data analyses

Testing Game Theory in the Field: Swedish LUPI Lottery Games

Game theory is usually difficult to test precisely in the field because predictions typically depend sensitively on features that are not controlled or observed. We conduct one such test using field data from the Swedish lowest unique positive integer (LUPI) game. In the LUPI game, players pick positive integers and whoever chose the lowest unique number wins a fixed prize. Theoretical equilibrium predictions are derived assuming Poisson- distributed uncertainty about the number of players, and tested using both field and laboratory data. The field and lab data show similar patterns. Despite various deviations from equilibrium, there is a surprising degree of convergence toward equilibrium. Some of the deviations from equilibrium can be rationalized by a cognitive hierarchy model