Rational bidding using reinforcement learning: an application in automated resource allocation

The application of autonomous agents by the provisioning and usage of computational resources is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic resource provisioning and usage of computational resources, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a framework for supporting consumers and providers in technical and economic preference elicitation and the generation of bids. Secondly, we introduce a consumer-side reinforcement learning bidding strategy which enables rational behavior by the generation and selection of bids. Thirdly, we evaluate and compare this bidding strategy against a truth-telling bidding strategy for two kinds of market mechanisms – one centralized and one decentralized

Variation in Decision Making

publication-status: PublishedVariation in how organisms allocate their behavior over their lifetimes is key to determining Darwinian fitness, and thus the evolution of human and non-human decision making. In this chapter, we explore how decision making varies across biologically and societally significant scales and what role such variation plays when trying to understand decision making from an evolutionary perspective. In the process, we highlight the importance of explicitly considering variation both when attempting to predict economically and socially important patterns of behavior, and to obtain a deeper understanding of the fundamental biological processes involved. We conclude by identifying key elements of a framework for incorporating variation into a general theory of Darwinian decision making

Q-Strategy: A Bidding Strategy for Market-Based Allocation of Grid Services

The application of autonomous agents by the provisioning and usage of computational services is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic service provisioning and usage of Grid services, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a bidding agent framework for implementing artificial bidding agents, supporting consumers and providers in technical and economic preference elicitation as well as automated bid generation by the requesting and provisioning of Grid services. Secondly, we introduce a novel consumer-side bidding strategy, which enables a goal-oriented and strategic behavior by the generation and submission of consumer service requests and selection of provider offers. Thirdly, we evaluate and compare the Q-strategy, implemented within the presented framework, against the Truth-Telling bidding strategy in three mechanisms – a centralized CDA, a decentralized on-line machine scheduling and a FIFO-scheduling mechanisms

Reinstated episodic context guides sampling-based decisions for reward.

How does experience inform decisions? In episodic sampling, decisions are guided by a few episodic memories of past choices. This process can yield choice patterns similar to model-free reinforcement learning; however, samples can vary from trial to trial, causing decisions to vary. Here we show that context retrieved during episodic sampling can cause choice behavior to deviate sharply from the predictions of reinforcement learning. Specifically, we show that, when a given memory is sampled, choices (in the present) are influenced by the properties of other decisions made in the same context as the sampled event. This effect is mediated by fMRI measures of context retrieval on each trial, suggesting a mechanism whereby cues trigger retrieval of context, which then triggers retrieval of other decisions from that context. This result establishes a new avenue by which experience can guide choice and, as such, has broad implications for the study of decisions

Contextual Simulated Annealing Q-Learning for Pre-negotiation of Agent-Based Bilateral Negotiations

Electricity markets are complex environments, which have been suffering continuous transformations due to the increase of renewable based generation and the introduction of new players in the system. In this context, players are forced to re-think their behavior and learn how to act in this dynamic environment in order to get as much benefit as possible from market negotiations. This paper introduces a new learning model to enable players identifying the expected prices of future bilateral agreements, as a way to improve the decision-making process in deciding the opponent players to approach for actual negotiations. The proposed model introduces a con-textual dimension in the well-known Q-Learning algorithm, and includes a simulated annealing process to accelerate the convergence process. The proposed model is integrated in a multi-agent decision support system for electricity market players negotiations, enabling the experimentation of results using real data from the Iberian electricity market.This work has received funding from the European Union's Horizon 2020 research and innovation programme under project DOMINOES (grant agreement No 771066) and from FEDER Funds through COMPETE program and from National Funds through FCT under the project UID/EEA/00760/2019.info:eu-repo/semantics/publishedVersio

Volatile Decision Dynamics: Experiments, Stochastic Description, Intermittency Control, and Traffic Optimization

The coordinated and efficient distribution of limited resources by individual decisions is a fundamental, unsolved problem. When individuals compete for road capacities, time, space, money, goods, etc., they normally make decisions based on aggregate rather than complete information, such as TV news or stock market indices. In related experiments, we have observed a volatile decision dynamics and far-from-optimal payoff distributions. We have also identified ways of information presentation that can considerably improve the overall performance of the system. In order to determine optimal strategies of decision guidance by means of user-specific recommendations, a stochastic behavioural description is developed. These strategies manage to increase the adaptibility to changing conditions and to reduce the deviation from the time-dependent user equilibrium, thereby enhancing the average and individual payoffs. Hence, our guidance strategies can increase the performance of all users by reducing overreaction and stabilizing the decision dynamics. These results are highly significant for predicting decision behaviour, for reaching optimal behavioural distributions by decision support systems, and for information service providers. One of the promising fields of application is traffic optimization.Comment: For related work see http://www.helbing.or

Comonotonic Independence: The Critical Test between Classical and Rank-Dependent Utility Theories

This article compares classical expected utility (EU) with the more general rank-dependent utility (RDU) models. The difference between the independence condition for preferences of EU and its comonotonic generalization in RDU provides the exact demarcation between EU and rank-dependent models. Other axiomatic differences are not essential. An experimental design is described that tests this difference between independence and comonotonic independence in its most basic form and is robust against violations of other assumptions that may confound the results, in particular the reduction principle and transitivity. It is well known that in the classical counterexamples to EU, comonotonic independence performs better than full-force independence. For our more general choice pairs, however, we find that comonotonic independence does not perform better. This is contrary to our prior expectation and suggests that rank-dependent models, in full generality, do not provide a descriptive improvement over EU. For rank-dependent models to have a future, submodels and choice situations need to be identified for which rank-dependence does contribute descriptively

Simulating complex social behaviour with the genetic action tree kernel

The concept of genetic action trees combines action trees with genetic algorithms. In this paper, we create a multi-agent simulation on the base of this concept and provide the interested reader with a software package to apply genetic action trees in a multi-agent simulation to simulate complex social behaviour. An example model is introduced to conduct a feasibility study with the described method. We find that our library can be used to simulate the behaviour of agents in a complex setting and observe a convergence to a global optimum in spite of the absence of stable states