Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising

Real-time advertising allows advertisers to bid for each impression for a visiting user. To optimize specific goals such as maximizing revenue and return on investment (ROI) led by ad placements, advertisers not only need to estimate the relevance between the ads and user's interests, but most importantly require a strategic response with respect to other advertisers bidding in the market. In this paper, we formulate bidding optimization with multi-agent reinforcement learning. To deal with a large number of advertisers, we propose a clustering method and assign each cluster with a strategic bidding agent. A practical Distributed Coordinated Multi-Agent Bidding (DCMAB) has been proposed and implemented to balance the tradeoff between the competition and cooperation among advertisers. The empirical study on our industry-scaled real-world data has demonstrated the effectiveness of our methods. Our results show cluster-based bidding would largely outperform single-agent and bandit approaches, and the coordinated bidding achieves better overall objectives than purely self-interested bidding agents

Real-time bidding with multi-agent reinforcement learning in display advertising

Real-time advertising allows advertisers to bid for each impression for a visiting user. To optimize specific goals such as maximizing revenue and return on investment (ROI) led by ad placements, advertisers not only need to estimate the relevance between the ads and user's interests, but most importantly require a strategic response with respect to other advertisers bidding in the market. In this paper, we formulate bidding optimization with multi-agent reinforcement learning. To deal with a large number of advertisers, we propose a clustering method and assign each cluster with a strategic bidding agent. A practical Distributed Coordinated Multi-Agent Bidding (DCMAB) has been proposed and implemented to balance the tradeoff between the competition and cooperation among advertisers. The empirical study on our industry-scaled real-world data has demonstrated the effectiveness of our methods. Our results show cluster-based bidding would largely outperform single-agent and bandit approaches, and the coordinated bidding achieves better overall objectives than purely self-interested bidding agents

A Rule-driven Approach for Defining the Behavior of Negotiating Software Agents

One problem with existing agent-mediated negotiation systems is that they rely on ad hoc, static, non-adaptive, and hardcoded schemes to represent the behaviour of agents. This limitation is probably due to the complexity of the negotiation task itself. Indeed, while negotiating, software (human) agents face tough decisions. These decisions are based not only on the information made available by the negotiation server, but on the behaviour of the other participants in the negotiation process as well. The information and the behaviour in question are constantly changing and highly uncertain. In the first part of the paper, we propose a rule-driven approach to represent, manage and explore negotiation strategies and coordination information. For that, we divide the behaviour of negotiating agents into protocols, strategies and coordination. Among the many advantages of the proposed solution, we can cite the high level of abstraction, the closeness to human understanding, the versatility, and the possibility to modify the agents' behaviour during the negotiation process. To validate our solution, we ran many agent tournaments, and used the rule-driven approach to implement bidding strategies that are common in the English and Dutch auctions. We also implemented simple coordination schemes across several auctions. The ongoing validation work is detailed and discussed in the second part of the paper. Un des inconvénients qu'on retrouve fréquemment dans les systèmes de négociation par agents est qu'ils reposent sur des schémas ad-hoc, non adaptatifs et figés dans le code pour représenter le comportement des agents. Cette limitation est probablement due à la complexité de l'activité de négociation elle-même. En effet, au cours de la négociation, les agents logiciels (humains) ont des décisions difficiles à prendre. Ces décisions ne sont pas seulement basées sur l'information disponible sur le serveur de négociation, mais aussi sur le comportement des autres participants durant le processus de négociation. L'information et le comportement en question changent constamment et sont très incertains. Dans la première partie de l'article, nous proposons une approche à base de règles pour représenter, gérer et explorer les stratégies de négociation ainsi que l'information de coordination. Parmi les nombreux avantages de la solution proposée, on peut citer le haut niveau d'abstraction, la proximité avec la compréhension humaine, la souplesse d'utilisation et la possibilité de modifier le comportement des agents durant le processus de négociation. Pour valider notre solution, nous avons effectué plusieurs tournois entre agents et utilisé l'approche à base de règles pour implémenter des stratégies simples applicables à l'enchère anglaise et à l'enchère hollandaise. Nous avons aussi implémenté des schémas simples de coordination impliquant plusieurs enchères. Le travail de validation, en cours, est détaillé et discuté dans la seconde partie de l'article.e-negotiation, online auction, software agent, negotiation strategy, coordination, rule-based system, rule engine, Négociation électronique, enchères en ligne, agents logiciels, stratégie de négociation, coordination, système à base de règles, moteur de règles

Towards a quantitative concession-based classification method of negotiation strategies

In order to successfully reach an agreement in a negotiation, both parties rely on each other to make concessions. The willingness to concede also depends in large part on the opponent. A concession by the opponent may be reciprocated, but the negotiation process may also be frustrated if the opponent does not concede at all.This process of concession making is a central theme in many of the classic and current automated negotiation strategies. In this paper, we present a quantitative classification method of negotiation strategies that measures the willingness of an agent to concede against different types of opponents. The method is then applied to classify some well-known negotiating strategies, including the agents of ANAC 2010. It is shown that the technique makes it easy to identify the main characteristics of negotiation agents, and can be used to group negotiation strategies into categories with common negotiation characteristics. We also observe, among other things, that different kinds of opponents call for a different approach in making concession