Real-Time Bidding by Reinforcement Learning in Display Advertising

The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks.Comment: WSDM 201

StreamBed: capacity planning for stream processing

StreamBed is a capacity planning system for stream processing. It predicts, ahead of any production deployment, the resources that a query will require to process an incoming data rate sustainably, and the appropriate configuration of these resources. StreamBed builds a capacity planning model by piloting a series of runs of the target query in a small-scale, controlled testbed. We implement StreamBed for the popular Flink DSP engine. Our evaluation with large-scale queries of the Nexmark benchmark demonstrates that StreamBed can effectively and accurately predict capacity requirements for jobs spanning more than 1,000 cores using a testbed of only 48 cores.Comment: 14 pages, 11 figures. This project has been funded by the Walloon region (Belgium) through the Win2Wal project GEPICIA

The persistence of drought impacts across growing seasons: a dynamic stochastic analysis

Agricultural producers throughout much of the United States experienced one of the most severe droughts in the last 100 years during the years 1999-2006. The prolonged nature of this drought highlights a need to better understand the impacts and management of drought across growing seasons, rather than just within a growing season. Producers express specific concern about the tendency of drought impacts to persist even after drought itself has subsided. The persistence of drought impacts has received limited attention in the economics literature. The objectives of this study are two-fold: 1) to determine whether inter-year dynamics, in the form of agronomic constraints and financial flows, can cause persistence of a drought's impact in years subsequent to the drought, and 2) to determine whether the impact of one year of drought can alter the impact of a subsequent year of drought. A multi-year, dynamic and stochastic decision model is developed in a discrete stochastic programming framework and solved to address the objectives. The structure and parameters of the farm-level model are based on irrigated row crop farms in eastern Oregon, USA. Analysis of the model's solution reveals the following results: 1) the impact of a drought can persist long after the drought subsides, and 2) the impact of one year of drought can alter the impact of a subsequent year of drought. Potential implications for the administration of drought-related assistance are discussed briefly.Drought, preparedness, response, uncertainty, dynamics, discrete stochastic programming, agriculture, irrigation, eastern Oregon, row crops, crop rotation, Crop Production/Industries, Risk and Uncertainty,

Assessing farmer behaviour as affected by policy and technological innovations: bio-economic farm models

Farm Management,

Chippe : a system for constraint driven behavioral synthesis

This report describes the Chippe system, gives some background previous work and describes several sample design runs of the system. Also presented are the sources of the design tradeoffs used by Chippe, and overview of the internal design model, and experiences using the system

Learning and Production of Movement Sequences: Behavioral, Neurophysiological, and Modeling Perspectives

A growing wave of behavioral studies, using a wide variety of paradigms that were introduced or greatly refined in recent years, has generated a new wealth of parametric observations about serial order behavior. What was a mere trickle of neurophysiological studies has grown to a more steady stream of probes of neural sites and mechanisms underlying sequential behavior. Moreover, simulation models of serial behavior generation have begun to open a channel to link cellular dynamics with cognitive and behavioral dynamics. Here we summarize the major results from prominent sequence learning and performance tasks, namely immediate serial recall, typing, 2XN, discrete sequence production, and serial reaction time. These populate a continuum from higher to lower degrees of internal control of sequential organization. The main movement classes covered are speech and keypressing, both involving small amplitude movements that are very amenable to parametric study. A brief synopsis of classes of serial order models, vis-à-vis the detailing of major effects found in the behavioral data, leads to a focus on competitive queuing (CQ) models. Recently, the many behavioral predictive successes of CQ models have been joined by successful prediction of distinctively patterend electrophysiological recordings in prefrontal cortex, wherein parallel activation dynamics of multiple neural ensembles strikingly matches the parallel dynamics predicted by CQ theory. An extended CQ simulation model-the N-STREAMS neural network model-is then examined to highlight issues in ongoing attemptes to accomodate a broader range of behavioral and neurophysiological data within a CQ-consistent theory. Important contemporary issues such as the nature of working memory representations for sequential behavior, and the development and role of chunks in hierarchial control are prominent throughout.Defense Advanced Research Projects Agency/Office of Naval Research (N00014-95-1-0409); National Institute of Mental Health (R01 DC02852

Model checking control communication of a FACTS device

This thesis concerns the design and verification of a real-time communication protocol for sensor data collection and processing between an embedded computer and a DSP. In such systems, a certain amount of data loss without recovery may be tolerated. The key issue is to design and verify the correctness in the presence of these lost data frames under real-time constraints. This thesis describes a temporal verification that if the end processes do not detect that too many frames are lost, defined by comparison of error counters against given threshold values, then there will be a bounded delay between transmission of data frames and reception of control frames. This verification and others presented herein were performed with the model checkers SPIN and RT-SPIN --Abstract, page iii

A High Density Micro-Electrocorticography Device for a Rodent Model

Electrocorticography (ECoG) is a methodology for stable mapping of the brain surface using local field potentials (LFPs) with a wide cortical region, high signal fidelity, and minimal invasiveness to brain tissue. To compare surface ECoG signals with inter-cortical neuronal activity, we fabricated a flexible handcrafted ECoG electrode made with economically available materials. This research is on a Lewis rat implanted with a handcrafted 256-channel, non-penetrative ECoG electrode covering an area of 7mm x 7mm on the cortical surface. This device was placed on the motor and somatosensory cortex of the brain to record signals with an active animal. The recordings are acquired by using the Synapse Software and the Tucker-Davis Technologies acquisition system to monitor and analyze electrophysiological signals within the amplitude range of 200µV for local field potentials. This demonstrates how reactive channels and their spatiotemporal and frequency-specific characteristics can be identified by means of this method