Utility-based Reinforcement Learning for Reactive Grids

International audienceLarge scale production grids are an important case for autonomic computing. They follow a mutualization paradigm: decision-making (human or automatic) is distributed and largely independent, and, at the same time, it must implement the highlevel goals of the grid management. This paper deals with the scheduling problem with two partially conflicting goals: fairshare and Quality of Service (QoS). Fair sharing is a wellknown issue motivated by return on investment for participating institutions. Differentiated QoS has emerged as an important and unexpected requirement in the current usage of production grids. In the framework of the EGEE grid (one of the largest existing grids), applications from diverse scientific communities require a pseudo-interactive response time. More generally, seamless integration of the grid power into everyday use calls for unplanned and interactive access to grid resources, which defines reactive grids. The major result of this paper is that the combination of utility functions and reinforcement learning (RL) provides a general and efficient method for dynamically allocating grid resources in order to satisfy both end users with differentiated requirements and participating institutions. Combining RL methods and utility functions for resource allocation was pioneered by Tesauro and Vengerov. While the application contexts are different, the resource allocation issues are very similar. The main difference in our work is that we consider a multi-criteria optimization problem that includes a fair-share objective. A first contribution of our work is the definition of a set of variables describing states and actions that allows us to formulate the grid scheduling problem as a continuous action-state space reinforcement learning problem. To capture the immediate goals of end users and the long-term objectives of administrators, we propose automatically derived utility functions. Finally, our experimental results on a synthetic workload and a real EGEE trace show that RL clearly outperforms the classical schedulers, so it is a realistic alternative to empirical scheduler design

Economy of Spectrum Access in Timy Varying Multichannel Networks

We consider a wireless network consisting of two classes of potentially mobile users: primary users and secondary users. Primary users license frequency channels and transmit in their respective bands as required. Secondary users resort to unlicensed access of channels that are not used by their primary users. Primaries impose access fees on the secondaries which depend on access durations and may be different for different primary channels and different available communication rates in the channels. The available rates to the secondaries change with time depending on the usage status of the primaries and the random access quality of channels. Secondary users seek to minimize their total access cost subject to stabilizing their queues whenever possible. Our first contribution is to present a dynamic link scheduling policy that attains this objective. The computation time of this policy, however, increases exponentially with the size of the network. We next present an approximate scheduling scheme based on graph partitioning that is distributed and attains arbitrary trade-offs between aggregate access cost and computation times of the schedules, irrespective of the size of the network. Our performance guarantees hold for general arrival and primary usage statistics and multihop networks. Each secondary user is, however, primarily interested in minimizing the cost it incurs, rather than in minimizing the aggregate cost. Thus, it will schedule its transmissions so as to minimize the aggregate cost only if it perceives that the aggregate cost is shared among the users as per a fair cost sharing scheme. Using concepts from cooperative game theory, we develop a rational basis for sharing the aggregate cost among secondary sessions and present a cost sharing mechanism that conforms to the above basis

An economic market for the brokering of time and budget guarantees

Grids offer best effort services to users. Service level agreements offer the opportunity to provide guarantees upon services offered, in such a way that it captures the users’ requirements, while also considering concerns of the service providers. This is achieved via a process of converging requirements and service cost values from both sides towards an agreement. This paper presents the intelligent scheduling for quality of service market-oriented mechanism for brokering guarantees upon completion time and cost for jobs submitted to a batch-oriented compute service. Web Services agreement (negotiation) is used along with the planning of schedules in determining pricing, ensuring that jobs become prioritised depending on their budget constraints. An evaluation is performed to demonstrate how market mechanisms can be used to achieve this, whilst also showing the effects that scheduling algorithms can have upon the market in terms of rescheduling. The evaluation is completed with a comparison of the broker’s capabilities in relation to the literature

A Framework for Service Differentiation and Optimization in Multi-hop Wireless Networks

In resource-constrained networks such as multi-hop wireless networks (MHWNs), service differentiation algorithms designed to address end users' interests (e.g. user satisfaction, QoS, etc.) should also consider efficient utilization of the scarce network resources in order to maximize the network's interests (e.g. revenue). For this very reason, service differentiation in MHWNs is quite different from the wired network scenario. We propose a service differentiation tool called the ``Investment Function'', which essentially captures the network's cumulative resource investment in a given packet at a given time. This investment value can be used by the network algorithm to implement specific service differentiation principles. As proof-of-concept, we use the investment function to improve fairness among simultaneous flows that traverse varying number of hops in a MHWN (multihop flow fairness). However, to attain the optimal value of a specific service differentiation objective, optimal service differentiation and investment function parameters may need to be computed. The optimal parameters can be computed by casting the service differentiation problem as a network flow problem in MHWNs, with the goal of optimizing the service differentiation objective. The capacity constraints for these problems require knowledge of the adjacent-node interference values, and constructing these constraints could be very expensive based on the transmission scheduling scheme used. As a result, even formulating the optimization problem may take unacceptable computational effort or memory or both. Under optimal scheduling, the adjacent node interference values (and thus the capacity constraints) are not only very expensive to compute, but also cannot be expressed in polynomial form. Therefore, existing optimization techniques cannot be directly applied to solve optimization problems in MHWNs. To develop an efficient optimization framework, we first model the MHWN as a Unit Disk Graph (UDG). The optimal transmission schedule in the MHWN is related to the chromatic number of the UDG, which is very expensive to compute. However, the clique number, which is a lower bound on the chromatic number, can be computed in polynomial time in UDGs. Through an empirical study, we obtain tighter bounds on the ratio of the chromatic number to clique number in UDGs, which enables us to leverage existing polynomial time clique-discovery algorithms to compute very close approximations to the chromatic number value. This approximation not only allows us to quickly formulate the capacity constraints in polynomial form, but also allows us to significantly deviate from the traditional approach of discovering all or most of the constraints \textit{a priori}; instead, we can discover the constraints as needed. We have integrated this approach of constraint-discovery into an active-set optimization algorithm (Gradient Projection method) to solve network flow problems in multi-hop wireless networks. Our results show significant memory and computational savings when compared to existing methods