Evaluating reinforcement learning for game theory application learning to price airline seats under competition

Applied Game Theory has been criticised for not being able to model real decision making situations. A game's sensitive nature and the difficultly in determining the utility payoff functions make it hard for a decision maker to rely upon any game theoretic results. Therefore the models tend to be simple due to the complexity of solving them (i.e. finding the equilibrium).In recent years, due to the increases of computing power, different computer modelling techniques have been applied in Game Theory. A major example is Artificial Intelligence methods e.g. Genetic Algorithms, Neural Networks and Reinforcement Learning (RL). These techniques allow the modeller to incorporate Game Theory within their models (or simulation) without necessarily knowing the optimal solution. After a warm up period of repeated episodes is run, the model learns to play the game well (though not necessarily optimally). This is a form of simulation-optimization.The objective of the research is to investigate the practical usage of RL within a simple sequential stochastic airline seat pricing game. Different forms of RL are considered and compared to the optimal policy, which is found using standard dynamic programming techniques. The airline game and RL methods displays various interesting phenomena, which are also discussed. For completeness, convergence proofs for the RL algorithms were constructed

What to bid and when to stop

Negotiation is an important activity in human society, and is studied by various disciplines, ranging from economics and game theory, to electronic commerce, social psychology, and artificial intelligence. Traditionally, negotiation is a necessary, but also time-consuming and expensive activity. Therefore, in the last decades there has been a large interest in the automation of negotiation, for example in the setting of e-commerce. This interest is fueled by the promise of automated agents eventually being able to negotiate on behalf of human negotiators.Every year, automated negotiation agents are improving in various ways, and there is now a large body of negotiation strategies available, all with their unique strengths and weaknesses. For example, some agents are able to predict the opponent's preferences very well, while others focus more on having a sophisticated bidding strategy. The problem however, is that there is little incremental improvement in agent design, as the agents are tested in varying negotiation settings, using a diverse set of performance measures. This makes it very difficult to meaningfully compare the agents, let alone their underlying techniques. As a result, we lack a reliable way to pinpoint the most effective components in a negotiating agent.There are two major advantages of distinguishing between the different components of a negotiating agent's strategy: first, it allows the study of the behavior and performance of the components in isolation. For example, it becomes possible to compare the preference learning component of all agents, and to identify the best among them. Second, we can proceed to mix and match different components to create new negotiation strategies., e.g.: replacing the preference learning technique of an agent and then examining whether this makes a difference. Such a procedure enables us to combine the individual components to systematically explore the space of possible negotiation strategies.To develop a compositional approach to evaluate and combine the components, we identify structure in most agent designs by introducing the BOA architecture, in which we can develop and integrate the different components of a negotiating agent. We identify three main components of a general negotiation strategy; namely a bidding strategy (B), possibly an opponent model (O), and an acceptance strategy (A). The bidding strategy considers what concessions it deems appropriate given its own preferences, and takes the opponent into account by using an opponent model. The acceptance strategy decides whether offers proposed by the opponent should be accepted.The BOA architecture is integrated into a generic negotiation environment called Genius, which is a software environment for designing and evaluating negotiation strategies. To explore the negotiation strategy space of the negotiation research community, we amend the Genius repository with various existing agents and scenarios from literature. Additionally, we organize a yearly international negotiation competition (ANAC) to harvest even more strategies and scenarios. ANAC also acts as an evaluation tool for negotiation strategies, and encourages the design of negotiation strategies and scenarios.We re-implement agents from literature and ANAC and decouple them to fit into the BOA architecture without introducing any changes in their behavior. For each of the three components, we manage to find and analyze the best ones for specific cases, as described below. We show that the BOA framework leads to significant improvements in agent design by wining ANAC 2013, which had 19 participating teams from 8 international institutions, with an agent that is designed using the BOA framework and is informed by a preliminary analysis of the different components.In every negotiation, one of the negotiating parties must accept an offer to reach an agreement. Therefore, it is important that a negotiator employs a proficient mechanism to decide under which conditions to accept. When contemplating whether to accept an offer, the agent is faced with the acceptance dilemma: accepting the offer may be suboptimal, as better offers may still be presented before time runs out. On the other hand, accepting too late may prevent an agreement from being reached, resulting in a break off with no gain for either party. We classify and compare state-of-the-art generic acceptance conditions. We propose new acceptance strategies and we demonstrate that they outperform the other conditions. We also provide insight into why some conditions work better than others and investigate correlations between the properties of the negotiation scenario and the efficacy of acceptance conditions.Later, we adopt a more principled approach by applying optimal stopping theory to calculate the optimal decision on the acceptance of an offer. We approach the decision of whether to accept as a sequential decision problem, by modeling the bids received as a stochastic process. We determine the optimal acceptance policies for particular opponent classes and we present an approach to estimate the expected range of offers when the type of opponent is unknown. We show that the proposed approach is able to find the optimal time to accept, and improves upon all existing acceptance strategies.Another principal component of a negotiating agent's strategy is its ability to take the opponent's preferences into account. The quality of an opponent model can be measured in two different ways. One is to use the agent's performance as a benchmark for the model's quality. We evaluate and compare the performance of a selection of state-of-the-art opponent modeling techniques in negotiation. We provide an overview of the factors influencing the quality of a model and we analyze how the performance of opponent models depends on the negotiation setting. We identify a class of simple and surprisingly effective opponent modeling techniques that did not receive much previous attention in literature.The other way to measure the quality of an opponent model is to directly evaluate its accuracy by using similarity measures. We review all methods to measure the accuracy of an opponent model and we then analyze how changes in accuracy translate into performance differences. Moreover, we pinpoint the best predictors for good performance. This leads to new insights concerning how to construct an opponent model, and what we need to measure when optimizing performance.Finally, we take two different approaches to gain more insight into effective bidding strategies. We present a new classification method for negotiation strategies, based on their pattern of concession making against different kinds of opponents. We apply this technique to classify some well-known negotiating strategies, and we formulate guidelines on how agents should bid in order to be successful, which gives insight into the bidding strategy space of negotiating agents. Furthermore, we apply optimal stopping theory again, this time to find the concessions that maximize utility for the bidder against particular opponents. We show there is an interesting connection between optimal bidding and optimal acceptance strategies, in the sense that they are mirrored versions of each other.Lastly, after analyzing all components separately, we put the pieces back together again. We take all BOA components accumulated so far, including the best ones, and combine them all together to explore the space of negotiation strategies.We compute the contribution of each component to the overall negotiation result, and we study the interaction between components. We find that combining the best agent components indeed makes the strongest agents. This shows that the component-based view of the BOA architecture not only provides a useful basis for developing negotiating agents but also provides a useful analytical tool. By varying the BOA components we are able to demonstrate the contribution of each component to the negotiation result, and thus analyze the significance of each. The bidding strategy is by far the most important to consider, followed by the acceptance conditions and finally followed by the opponent model.Our results validate the analytical approach of the BOA framework to first optimize the individual components, and then to recombine them into a negotiating agent

Rationalist causes of war : mechanisms, experiments, and East Asian wars

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Political Science, 2013.Cataloged from PDF version of thesis.Includes bibliographical references.This dissertation specifies and tests rationalist mechanisms of war. Why would rational states fight each other despite their incentives for peaceful bargains that would avoid the costs of war? In the rationalist theory of war, private information and the commitment problem are the key causes of war. I study the effects of these factors - and the mechanisms regulating their effects - through randomized experiments, historical analysis of the decision processes in three wars, and a comparative study of all international wars fought in East Asia in the last century. This is the first integrated study of rationalist causes of war that combines randomized experiments with historical cases. Despite a wide theoretical literature, there are few empirical tests of rationalist explanations for war. I use experimental and historical evidence to show that the commitment problem has strong positive effects on conflict. The effects of private information are less clear. Next, I specify six mechanisms that regulate the effects of the commitment problem and the private-information problem: three mechanisms (exogenous, endogenous, and inadvertent enforcement) for the first problem and three mechanisms (signaling with sunk cost, implementation cost, and salient contradiction) for the second. The experimental and historical evidence largely converge. Each of the three enforcement mechanisms calms the commitment problem and reduces the risk of conflict. Evidence for the three signaling mechanisms is mixed. Finally, I use the case universe of East Asian wars to assess the relevance of the mechanisms, suggest theoretical refinements, and infer alternative theories of war.by Ch-yuan Kaiy Quek.Ph.D

Towards More Nuanced Patient Management: Decomposing Readmission Risk with Survival Models

Unplanned hospital readmissions are costly and associated with poorer patient outcomes. Overall readmission rates have also come to be used as performance metrics in reimbursement in healthcare policy, further motivating hospitals to identify and manage high-risk patients. Many models predicting readmission risk have been developed to facilitate the equitable measurement of readmission rates and to support hospital decision-makers in prioritising patients for interventions. However, these models consider the overall risk of readmission and are often restricted to a single time point. This work aims to develop the use of survival models to better support hospital decision-makers in managing readmission risk. First, semi-parametric statistical and nonparametric machine learning models are applied to adult patients admitted via the emergency department at Gold Coast University Hospital (n = 46,659) and Robina Hospital (n = 23,976) in Queensland, Australia. Overall model performance is assessed based on discrimination and calibration, as measured by time-dependent concordance and D-calibration. Second, a framework based on iterative hypothesis development and model fitting is proposed for decomposing readmission risk into persistent, patient-specific baselines and transient, care-related components using a sum of exponential hazards structure. Third, criteria for patient prioritisation based on the duration and magnitude of care-related risk components are developed. The extensibility of the framework and subsequent prioritisation criteria are considered for alternative populations, such as outpatient admissions and specific diagnosis groups, and different modelling techniques. Time-to-event models have rarely been applied for readmission modelling but can provide a rich description of the evolution of readmission risk post-discharge and support more nuanced patient management decisions than simple classification models

Pre-competition achievement goals within young sports performers

This thesis attempted to develop a clearer understanding of the pre-competition achievement goal perspectives that are held by young performers. The programme of research moves through three transitional stages incorporating three different methodologies. Specifically, the first two investigations which comprised Study 1 adopted a quantitative research methodology; Study 2 incorporated qualitative techniques; and the final investigation addressed the research question on an idiographic basis via a single subject design study. Firstly, an attempt is made to identify the major antecedents or precursors to states of goal involvement prior to a specific competitive situation. The first study examined the antecedents of pre-competition state goals within adolescent swimmers from an interactionist perspective. Results showed how levels of task and ego involvement prior to a specific race were related to both dispositional tendencies and situational factors within the race context. However, task orientation appeared to play a more powerful role than ego orientation in predicting their respective goal states. Furthermore, ego involvement was more strongly predicted by situational factors. The second investigation extended this question by investigating a sample of elite junior tennis players prior to a competitive match at the National Championships. In this way, the nature of the competitive context, with respect to goal or reward structure, changed from being more task-involving (individualistic-focused) to being more ego-involving (competitive-focused). Results showed how the players' goal states were related much more to perceptions of the context than to their reported goal orientation. Furthermore, task orientation did not emerge as a significant predictor of goal involvement. With these results in mind, the second stage of the thesis involved investigating, to a much greater depth, the motivational criteria which appeared to contribute to the development of goal orientation and the activation of goal involvement in the context of competition. For this purpose, qualitative interview techniques and an inductive content analysis were applied to a sample of seventeen elite junior tennis players. The findings suggested that the development of goal orientation and activation of pre-competition goal involvement rested on a complex interaction of internal and environmental factors. Specific general dimensions of influence included cognitive-developmental skills and experience, the motivational climate conveyed by significant others, the social and structural nature of tennis, and the match context. The information gathered from this study provided the impetus, rationale and theoretical foundation for the final study in this thesis. Employing a single subject multiple baseline across subjects design, the study investigated the effects of a structured environmental and task-based intervention programme which sought to influence precompetition goal involvement and related competitive cognitions within a small sample of adolescent national standard tennis players. Following a three month intervention period, the three targeted players reported pre-competition goal states which showed increased activation of the self-referent conception of achievement. Furthermore, each player fostered an attitude which valued the challenge of winning matches for internal reasons, as opposed to reasons associated with favourable social approval. These findings reinforced the practicability of educationlaction-based interventions designed to develop more adaptive motivational responses to competitive situations. The programme of research conducted in this thesis, therefore, highlights how precompetition achievement goal perspectives within young performers may be influenced, provided that one has a detailed understanding of the antecedents of this process. In so doing, this thesis alerts future research to the importance of working within an interactionist paradigm and with a measurement technology which can accurately assess goal states in a diverse number of sporting situations. In this way, our understanding of goal involvement, as an important achievement-related attentional state, may be greatly facilitated

Exploring Hopes And Fears From Supply Chain Innovations: An Analysis Of Antecedents And Consequences Of Supply Chain Knowledge Exchanges

This dissertation sheds light on severalhopes and fears from supply chain innovation in three distinct papers. Paper one introduces the concept of Process Innovation Propagation as an appropriation technique helping to extract the most returns out of a process innovation by exporting to supply chain partners. Paper two devises and empirically tests knowledge properties that best lead to radical and incremental supply chain innovative capabilities. Lastly, paper three conducts an exploratory study that introduces factors affecting a firm’s optimum supply chain innovation strategy. The dissertation makes a strong argument that supply chain innovation is most prominently governed by power asymmetry that may either help or hurt innovative performance

Using MapReduce Streaming for Distributed Life Simulation on the Cloud

Distributed software simulations are indispensable in the study of large-scale life models but often require the use of technically complex lower-level distributed computing frameworks, such as MPI. We propose to overcome the complexity challenge by applying the emerging MapReduce (MR) model to distributed life simulations and by running such simulations on the cloud. Technically, we design optimized MR streaming algorithms for discrete and continuous versions of Conway’s life according to a general MR streaming pattern. We chose life because it is simple enough as a testbed for MR’s applicability to a-life simulations and general enough to make our results applicable to various lattice-based a-life models. We implement and empirically evaluate our algorithms’ performance on Amazon’s Elastic MR cloud. Our experiments demonstrate that a single MR optimization technique called strip partitioning can reduce the execution time of continuous life simulations by 64%. To the best of our knowledge, we are the first to propose and evaluate MR streaming algorithms for lattice-based simulations. Our algorithms can serve as prototypes in the development of novel MR simulation algorithms for large-scale lattice-based a-life models.https://digitalcommons.chapman.edu/scs_books/1014/thumbnail.jp