Rational bidding using reinforcement learning: an application in automated resource allocation

The application of autonomous agents by the provisioning and usage of computational resources is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic resource provisioning and usage of computational resources, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a framework for supporting consumers and providers in technical and economic preference elicitation and the generation of bids. Secondly, we introduce a consumer-side reinforcement learning bidding strategy which enables rational behavior by the generation and selection of bids. Thirdly, we evaluate and compare this bidding strategy against a truth-telling bidding strategy for two kinds of market mechanisms – one centralized and one decentralized

Q-Strategy: A Bidding Strategy for Market-Based Allocation of Grid Services

The application of autonomous agents by the provisioning and usage of computational services is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic service provisioning and usage of Grid services, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a bidding agent framework for implementing artificial bidding agents, supporting consumers and providers in technical and economic preference elicitation as well as automated bid generation by the requesting and provisioning of Grid services. Secondly, we introduce a novel consumer-side bidding strategy, which enables a goal-oriented and strategic behavior by the generation and submission of consumer service requests and selection of provider offers. Thirdly, we evaluate and compare the Q-strategy, implemented within the presented framework, against the Truth-Telling bidding strategy in three mechanisms – a centralized CDA, a decentralized on-line machine scheduling and a FIFO-scheduling mechanisms

Evolutionary Multiagent Transfer Learning With Model-Based Opponent Behavior Prediction

This article embarks a study on multiagent transfer learning (TL) for addressing the specific challenges that arise in complex multiagent systems where agents have different or even competing objectives. Specifically, beyond the essential backbone of a state-of-the-art evolutionary TL framework (eTL), this article presents the novel TL framework with prediction (eTL-P) as an upgrade over existing eTL to endow agents with abilities to interact with their opponents effectively by building candidate models and accordingly predicting their behavioral strategies. To reduce the complexity of candidate models, eTL-P constructs a monotone submodular function, which facilitates to select Top-K models from all available candidate models based on their representativeness in terms of behavioral coverage as well as reward diversity. eTL-P also integrates social selection mechanisms for agents to identify their better-performing partners, thus improving their learning performance and reducing the complexity of behavior prediction by reusing useful knowledge with respect to their partners' mind universes. Experiments based on a partner-opponent minefield navigation task (PO-MNT) have shown that eTL-P exhibits the superiority in achieving higher learning capability and efficiency of multiple agents when compared to the state-of-the-art multiagent TL approaches

Enhancing Intelligent Agents with Episodic Memory.

In this dissertation, we explore the effects of adding an episodic memory to an intelligent agent. First, we define the design space for episodic memory systems and the properties that any implementation must have in order to be integrated into a cognitive architecture. We then describe our exploration of this space including two major implementations of an architectural episodic memory as well as several refinements to those implementations and their impact on agent performance. We also present a series of cognitive capabilities that are facilitated by virtue of an agent possessing an episodic memory. We hypothesize that these capabilities improve an agent’s ability to effectively sense its environment, reason and learn. We then demonstrate five of these cognitive capabilities using a specific task in one of two different virtual environments.Ph.D.Computer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/57720/2/anuxoll_1.pd

Goal-oriented Behaviour for Intelligent Game Agents

This thesis concerns our innovation in game AI techniques, mainly game agents' modeling, planning and learning. The research topic involves the development of a game design software --- Gameme. Our work mainly focus on the development of the core AI module. In this thesis, after discussing the system design of Gameme, we explain our contributions in two parts: off-line design and real-time processing. In off-line design, we present goal-oriented behaviour design and related modeling methodology for game agents. The goal-oriented design provides not only an intuitive behaviour design methodology for non-professional game designers but also efficient support for real-time behaviour control. In particular, the goal-oriented design can be used in modeling agents in different games. The real-time processing component includes planning and learning mechanisms for game agents. These mechanisms are placed in a layered architecture. Basically, a procedural planning mechanism allows game agents to have the ability of fast reaction to their environment. Then, the creative transfer and adaptive learning mechanism trains game agents to learn from their experience and cooperate in teamwork. Furthermore, the unique emergent learning mechanism can allow game agents to have the ability to analyze different PCs' behaviour patterns and to find the suitable strategy to defeat PCs in real-time. Most of the experiments in this thesis are performed in fighting scenarios. We connected the core AI module with a 3D graphics engine in order to have visual testing results. All test cases show that our goal-oriented behaviour design along with planning and learning mechanisms can provide fast, autonomous, collaborative and adaptive behaviour instructions for game agent in real-time game play

Automated Bidding in Computing Service Markets. Strategies, Architectures, Protocols

This dissertation contributes to the research on Computational Mechanism Design by providing novel theoretical and software models - a novel bidding strategy called Q-Strategy, which automates bidding processes in imperfect information markets, a software framework for realizing agents and bidding strategies called BidGenerator and a communication protocol called MX/CS, for expressing and exchanging economic and technical information in a market-based scheduling system

Distributed Reinforcement Learning for Network Intrusion Response

The increasing adoption of technologies and the exponential growth of networks has made the area of information technology an integral part of our lives, where network security plays a vital role. One of the most serious threats in the current Internet is posed by distributed denial of service (DDoS) attacks, which target the availability of the victim system. Such an attack is designed to exhaust a server's resources or congest a network's infrastructure, and therefore renders the victim incapable of providing services to its legitimate users or customers. To tackle the distributed nature of these attacks, a distributed and coordinated defence mechanism is necessary, where many defensive nodes, across different locations cooperate in order to stop or reduce the flood. This thesis investigates the applicability of distributed reinforcement learning to intrusion response, specifically, DDoS response. We propose a novel approach to respond to DDoS attacks called Multiagent Router Throttling. Multiagent Router Throttling provides an agent-based distributed response to the DDoS problem, where multiple reinforcement learning agents are installed on a set of routers and learn to rate-limit or throttle traffic towards a victim server. One of the novel characteristics of the proposed approach is that it has a decentralised architecture and provides a decentralised coordinated response to the DDoS problem, thus being resilient to the attacks themselves. Scalability constitutes a critical aspect of a defence system since a non-scalable mechanism will never be considered, let alone adopted, for wide deployment by a company or organisation. We propose Coordinated Team Learning (CTL) which is a novel design to the original Multiagent Router Throttling approach based on the divide-and-conquer paradigm, that uses task decomposition and coordinated team rewards. To better scale-up CTL is combined with a form of reward shaping. The scalability of the proposed system is successfully demonstrated in experiments involving up to 1000 reinforcement learning agents. The significant improvements on scalability and learning speed lay the foundations for a potential real-world deployment