AdCraft: An Advanced Reinforcement Learning Benchmark Environment for Search Engine Marketing Optimization

We introduce AdCraft, a novel benchmark environment for the Reinforcement Learning (RL) community distinguished by its stochastic and non-stationary properties. The environment simulates bidding and budgeting dynamics within Search Engine Marketing (SEM), a digital marketing technique utilizing paid advertising to enhance the visibility of websites on search engine results pages (SERPs). The performance of SEM advertisement campaigns depends on several factors, including keyword selection, ad design, bid management, budget adjustments, and performance monitoring. Deep RL recently emerged as a potential strategy to optimize campaign profitability within the complex and dynamic landscape of SEM but it requires substantial data, which may be costly or infeasible to acquire in practice. Our customizable environment enables practitioners to assess and enhance the robustness of RL algorithms pertinent to SEM bid and budget management without such costs. Through a series of experiments within the environment, we demonstrate the challenges imposed by sparsity and non-stationarity on agent convergence and performance. We hope these challenges further encourage discourse and development around effective strategies for managing real-world uncertainties

User Response in Ad Auctions: An MDP Formulation of Long-Term Revenue Optimization

We propose a new Markov Decision Process (MDP) model for ad auctions to capture the user response to the quality of ads, with the objective of maximizing the long-term discounted revenue. By incorporating user response, our model takes into consideration all three parties involved in the auction (advertiser, auctioneer, and user). The state of the user is modeled as a user-specific click-through rate (CTR) with the CTR changing in the next round according to the set of ads shown to the user in the current round. We characterize the optimal mechanism for this MDP as a Myerson's auction with a notion of modified virtual value, which relies on the value distribution of the advertiser, the current user state, and the future impact of showing the ad to the user. Moreover, we propose a simple mechanism built upon second price auctions with personalized reserve prices and show it can achieve a constant-factor approximation to the optimal long term discounted revenue

Essays on auction mechanisms and resource allocation in keyword advertising

textAdvances in information technology have created radically new business models, most notably the integration of advertising with keyword-based targeting, or "keyword advertising." Keyword advertising has two main variations: advertising based on keywords employed by users in search engines, often known as "sponsored links," and advertising based on keywords embedded in the content users view, often known as "contextual advertising." Keyword advertising providers such as Google and Yahoo! use auctions to allocate advertising slots. This dissertation examines the design of keyword auctions. It consists of three essays. The first essay "Ex-Ante Information and the Design of Keyword Auctions" focuses on how to incorporate available information into auction design. In our keyword auction model, advertisers bid their willingness-to-pay per click on their advertisements, and the advertising provider can weigh advertisers' bids differently and require different minimum bids based on advertisers' click-generating potential. We study the impact and design of such weighting schemes and minimum-bids policies. We find that weighting scheme determines how advertisers with different click-generating potential match in equilibrium. Minimum bids exclude low-valuation advertisers and at the same time may distort the equilibrium matching. The efficient design of keyword auctions requires weighting advertisers' bids by their expected click-through-rates, and requires the same minimum weighted bids. The revenue-maximizing weighting scheme may or may not favor advertisers with low click-generating potential. The revenue-maximizing minimum-bid policy differs from those prescribed in the standard auction design literature. Keyword auctions that employ the revenue-maximizing weighting scheme and differentiated minimum bid policy can generate higher revenue than standard fixed-payment auctions. The dynamics of bidders' performance is examined in the second essay, "Keyword Auctions, Unit-price Contracts, and the Role of Commitment." We extend earlier static models by allowing bidders with lower performance levels to improve their performance at a certain cost. We examine the impact of the weighting scheme on overall bidder performance, the auction efficiency, and the auctioneer's revenue, and derive the revenue-maximizing and efficient policy accordingly. Moreover, the possible upgrade in bidders' performance levels gives the auctioneer an incentive to modify the auction rules over time, as is confirmed by the practice of Yahoo! And Google. We thus compare the auctioneer's revenue-maximizing policies when she is fully committed to the auction rule and when not, and show that she should give less preferential treatment to low-performance advertisers when she is fully committed. In the third essay, "How to Slice the Pie? Optimal Share Structure Design in Keyword Auctions," we study the design of share structures in keyword auctions. Auctions for keyword advertising resources can be viewed as share auctions in which the highest bidder gets the largest share, the second highest bidder gets the second largest share, and so on. A share structure problem arises in such a setting regarding how much resources to set aside for the highest bidder, for the second highest bidder, etc. We address this problem under a general specification and derive implications on how the optimal share structure should change with bidders' price elasticity of demand for exposure, their valuation distribution, total resources, and minimum bids.Managemen

Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting

The most significant progress in recent years in online display advertising is what is known as the Real-Time Bidding (RTB) mechanism to buy and sell ads. RTB essentially facilitates buying an individual ad impression in real time while it is still being generated from a user’s visit. RTB not only scales up the buying process by aggregating a large amount of available inventories across publishers but, most importantly, enables direct targeting of individual users. As such, RTB has fundamentally changed the landscape of digital marketing. Scientifically, the demand for automation, integration and optimisation in RTB also brings new research opportunities in information retrieval, data mining, machine learning and other related fields. In this monograph, an overview is given of the fundamental infrastructure, algorithms, and technical solutions of this new frontier of computational advertising. The covered topics include user response prediction, bid landscape forecasting, bidding algorithms, revenue optimisation, statistical arbitrage, dynamic pricing, and ad fraud detection

Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting

The most significant progress in recent years in online display advertising is what is known as the Real-Time Bidding (RTB) mechanism to buy and sell ads. RTB essentially facilitates buying an individual ad impression in real time while it is still being generated from a user’s visit. RTB not only scales up the buying process by aggregating a large amount of available inventories across publishers but, most importantly, enables direct targeting of individual users. As such, RTB has fundamentally changed the landscape of digital marketing. Scientifically, the demand for automation, integration and optimisation in RTB also brings new research opportunities in information retrieval, data mining, machine learning and other related fields. In this monograph, an overview is given of the fundamental infrastructure, algorithms, and technical solutions of this new frontier of computational advertising. The covered topics include user response prediction, bid landscape forecasting, bidding algorithms, revenue optimisation, statistical arbitrage, dynamic pricing, and ad fraud detection