Rational bidding using reinforcement learning: an application in automated resource allocation

The application of autonomous agents by the provisioning and usage of computational resources is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic resource provisioning and usage of computational resources, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a framework for supporting consumers and providers in technical and economic preference elicitation and the generation of bids. Secondly, we introduce a consumer-side reinforcement learning bidding strategy which enables rational behavior by the generation and selection of bids. Thirdly, we evaluate and compare this bidding strategy against a truth-telling bidding strategy for two kinds of market mechanisms – one centralized and one decentralized

Using Multi-Agent Reinforcement Learning in Auction Simulations

Game theory has been developed by scientists as a theory of strategic interaction among players who are supposed to be perfectly rational. These strategic interactions might have been presented in an auction, a business negotiation, a chess game, or even in a political conflict aroused between different agents. In this study, the strategic (rational) agents created by reinforcement learning algorithms are supposed to be bidder agents in various types of auction mechanisms such as British Auction, Sealed Bid Auction, and Vickrey Auction designs. Next, the equilibrium points determined by the agents are compared with the outcomes of the Nash equilibrium points for these environments. The bidding strategy of the agents is analyzed in terms of individual rationality, truthfulness (strategy-proof), and computational efficiency. The results show that using a multi-agent reinforcement learning strategy improves the outcomes of the auction simulations

Q-Strategy: A Bidding Strategy for Market-Based Allocation of Grid Services

The application of autonomous agents by the provisioning and usage of computational services is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic service provisioning and usage of Grid services, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a bidding agent framework for implementing artificial bidding agents, supporting consumers and providers in technical and economic preference elicitation as well as automated bid generation by the requesting and provisioning of Grid services. Secondly, we introduce a novel consumer-side bidding strategy, which enables a goal-oriented and strategic behavior by the generation and submission of consumer service requests and selection of provider offers. Thirdly, we evaluate and compare the Q-strategy, implemented within the presented framework, against the Truth-Telling bidding strategy in three mechanisms – a centralized CDA, a decentralized on-line machine scheduling and a FIFO-scheduling mechanisms

Boundedly rational versus optimization-based models of strategic thinking and learning in games

The paper is a comment on the article by R. Harstad and R. Selten and considers the tradeoff between bounded rationality and optimization models in the game-theoretic context. The author shows that in most of the models elements of opimization are still retained and that it is thus more productive to further improve the optimization-based modeling rather than to abandon them altogether in favour of bounded rationality

User evaluation of a market-based recommender system

Recommender systems have been developed for a wide variety of applications (ranging from books, to holidays, to web pages). These systems have used a number of different approaches, since no one technique is best for all users in all situations. Given this, we believe that to be effective, systems should incorporate a wide variety of such techniques and then some form of overarching framework should be put in place to coordinate them so that only the best recommendations (from whatever source) are presented to the user. To this end, in our previous work, we detailed a market-based approach in which various recommender agents competed with one another to present their recommendations to the user. We showed through theoretical analysis and empirical evaluation with simulated users that an appropriately designed marketplace should be able to provide effective coordination. Building on this, we now report on the development of this multi-agent system and its evaluation with real users. Specifically, we show that our system is capable of consistently giving high quality recommendations, that the best recommendations that could be put forward are actually put forward, and that the combination of recommenders performs better than any constituent recommende

Learning users' interests by quality classification in market-based recommender systems

Recommender systems are widely used to cope with the problem of information overload and, to date, many recommendation methods have been developed. However, no one technique is best for all users in all situations. To combat this, we have previously developed a market-based recommender system that allows multiple agents (each representing a different recommendation method or system) to compete with one another to present their best recommendations to the user. In our system, the marketplace encourages good recommendations by rewarding the corresponding agents who supplied them according to the users’ ratings of their suggestions. Moreover, we have theoretically shown how our system incentivises the agents to bid in a manner that ensures only the best recommendations are presented. To do this effectively in practice, however, each agent needs to be able to classify its recommendations into different internal quality levels, learn the users’ interests for these different levels, and then adapt its bidding behaviour for the various levels accordingly. To this end, in this paper we develop a reinforcement learning and Boltzmann exploration strategy that the recommending agents can exploit for these tasks. We then demonstrate that this strategy does indeed help the agents to effectively obtain information about the users’ interests which, in turn, speeds up the market convergence and enables the system to rapidly highlight the best recommendations

When Do Groups Perform Better than Individuals? A Company Takeover Experiment

It is still an open question when groups will perform better than individuals in intellectual tasks. We report that in a company takeover experiment, groups placed better bids than individuals and substantially reduced the winner’s curse. This improvement was mostly due to peer pressure over the minority opinion and to group learning. Learning took place from interacting and negotiating consensus with others, not simply from observing their bids. When there was disagreement within a group, what prevailed was not the best proposal but the one of the majority. Groups underperformed with respect to a “truth wins” benchmark although they outperformed individuals deciding in isolation.

Multi-Unit Auctions to Allocate Water Scarcity Simulating Bidding Behaviour with Agent Based Models

Multi-unit auctions are promising mechanisms for the reallocation of water. The main advantage of such auctions is to avoid the lumpy bid issue. However, there is great uncertainty about the best auction formats when multi-unit auctions are used. The theory can only supply the structural properties of equilibrium strategies and the multiplicity of equilibria makes comparisons across auction formats difficult. Empirical studies and experiments have improved our knowledge of multi- unit auctions but they remain scarce and most experiments are restricted to two bidders and two units. Moreover, they demonstrate that bidders have limited rationality and learn through experience. This paper constructs an agent-based model of bidders to compare the performance of alternative auction formats under circumstances where bidders submit continuous bid supply functions and learn over time to adjust their bids to improve their net incomes. We demonstrate that under the generalized Vickrey, simulated bids converge towards truthful bids as predicted by the theory and that bid shading is the rule for the uniform and discriminatory auctions. Our study allows us to assess the potential gains from agent-based modelling approaches in the assessment of the dynamic performance of multi-unit procurement auctions. Some recommendations on the desirable format of water auctions are provided.Multi-unit auctions, Learning, Multi-agent models, Water allocation