Using Pilot Systems to Execute Many Task Workloads on Supercomputers

High performance computing systems have historically been designed to support applications comprised of mostly monolithic, single-job workloads. Pilot systems decouple workload specification, resource selection, and task execution via job placeholders and late-binding. Pilot systems help to satisfy the resource requirements of workloads comprised of multiple tasks. RADICAL-Pilot (RP) is a modular and extensible Python-based pilot system. In this paper we describe RP's design, architecture and implementation, and characterize its performance. RP is capable of spawning more than 100 tasks/second and supports the steady-state execution of up to 16K concurrent tasks. RP can be used stand-alone, as well as integrated with other application-level tools as a runtime system

Design and Experimental Validation of a Software-Defined Radio Access Network Testbed with Slicing Support

Network slicing is a fundamental feature of 5G systems to partition a single network into a number of segregated logical networks, each optimized for a particular type of service, or dedicated to a particular customer or application. The realization of network slicing is particularly challenging in the Radio Access Network (RAN) part, where multiple slices can be multiplexed over the same radio channel and Radio Resource Management (RRM) functions shall be used to split the cell radio resources and achieve the expected behaviour per slice. In this context, this paper describes the key design and implementation aspects of a Software-Defined RAN (SD-RAN) experimental testbed with slicing support. The testbed has been designed consistently with the slicing capabilities and related management framework established by 3GPP in Release 15. The testbed is used to demonstrate the provisioning of RAN slices (e.g. preparation, commissioning and activation phases) and the operation of the implemented RRM functionality for slice-aware admission control and scheduling

Learning Scheduling Algorithms for Data Processing Clusters

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective such as minimizing average job completion time. Off-the-shelf RL techniques, however, cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves the average job completion time over hand-tuned scheduling heuristics by at least 21%, achieving up to 2x improvement during periods of high cluster load

Building a Truly Distributed Constraint Solver with JADE

Real life problems such as scheduling meeting between people at different locations can be modelled as distributed Constraint Satisfaction Problems (CSPs). Suitable and satisfactory solutions can then be found using constraint satisfaction algorithms which can be exhaustive (backtracking) or otherwise (local search). However, most research in this area tested their algorithms by simulation on a single PC with a single program entry point. The main contribution of our work is the design and implementation of a truly distributed constraint solver based on a local search algorithm using Java Agent DEvelopment framework (JADE) to enable communication between agents on different machines. Particularly, we discuss design and implementation issues related to truly distributed constraint solver which might not be critical when simulated on a single machine. Evaluation results indicate that our truly distributed constraint solver works well within the observed limitations when tested with various distributed CSPs. Our application can also incorporate any constraint solving algorithm with little modifications.Comment: 7 page

Automated Negotiation for Provisioning Virtual Private Networks Using FIPA-Compliant Agents

This paper describes the design and implementation of negotiating agents for the task of provisioning virtual private networks. The agents and their interactions comply with the FIPA specification and they are implemented using the FIPA-OS agent framework. Particular attention is focused on the design and implementation of the negotiation algorithms

An Approach to Agent-Based Service Composition and Its Application to Mobile

This paper describes an architecture model for multiagent systems that was developed in the European project LEAP (Lightweight Extensible Agent Platform). Its main feature is a set of generic services that are implemented independently of the agents and can be installed into the agents by the application developer in a flexible way. Moreover, two applications using this architecture model are described that were also developed within the LEAP project. The application domain is the support of mobile, virtual teams for the German automobile club ADAC and for British Telecommunications

The Repast Simulation/Modelling System for Geospatial Simulation

The use of simulation/modelling systems can simplify the implementation of agent-based models. Repast is one of the few simulation/modelling software systems that supports the integration of geospatial data especially that of vector-based geometries. This paper provides details about Repast specifically an overview, including its different development languages available to develop agent-based models. Before describing Repast’s core functionality and how models can be developed within it, specific emphasis will be placed on its ability to represent dynamics and incorporate geographical information. Once these elements of the system have been covered, a diverse list of Agent-Based Modelling (ABM) applications using Repast will be presented with particular emphasis on spatial applications utilizing Repast, in particular, those that utilize geospatial data

Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management

As energy demand continues to increase, demand response (DR) programs in the electricity distribution grid are gaining momentum and their adoption is set to grow gradually over the years ahead. Demand response schemes seek to incentivise consumers to use green energy and reduce their electricity usage during peak periods which helps support grid balancing of supply-demand and generate revenue by selling surplus of energy back to the grid. This paper proposes an effective energy management system for residential demand response using Reinforcement Learning (RL) and Fuzzy Reasoning (FR). RL is considered as a model-free control strategy which learns from the interaction with its environment by performing actions and evaluating the results. The proposed algorithm considers human preference by directly integrating user feedback into its control logic using fuzzy reasoning as reward functions. Q-learning, a RL strategy based on a reward mechanism, is used to make optimal decisions to schedule the operation of smart home appliances by shifting controllable appliances from peak periods, when electricity prices are high, to off-peak hours, when electricity prices are lower without affecting the customer’s preferences. The proposed approach works with a single agent to control 14 household appliances and uses a reduced number of state-action pairs and fuzzy logic for rewards functions to evaluate an action taken for a certain state. The simulation results show that the proposed appliances scheduling approach can smooth the power consumption profile and minimise the electricity cost while considering user’s preferences, user’s feedbacks on each action taken and his/her preference settings. A user-interface is developed in MATLAB/Simulink for the Home Energy Management System (HEMS) to demonstrate the proposed DR scheme. The simulation tool includes features such as smart appliances, electricity pricing signals, smart meters, solar photovoltaic generation, battery energy storage, electric vehicle and grid supply.Peer reviewe

Integrated engineering environments for large complex products

An introduction is given to the Engineering Design Centre at the University of Newcastle upon Tyne, along with a brief explanation of the main focus towards large made-to-order products. Three key areas of research at the Centre, which have evolved as a result of collaboration with industrial partners from various sectors of industry, are identified as (1) decision support and optimisation, (2) design for lifecycle, and (3) design integration and co-ordination. A summary of the unique features of large made-to-order products is then presented, which includes the need for integration and co-ordination technologies. Thus, an overview of the existing integration and co-ordination technologies is presented followed by a brief explanation of research in these areas at the Engineering Design Centre. A more detailed description is then presented regarding the co-ordination aspect of research being conducted at the Engineering Design Centre, in collaboration with the CAD Centre at the University of Strathclyde. Concurrent Engineering is acknowledged as a strategy for improving the design process, however design coordination is viewed as a principal requirement for its successful implementation. That is, design co-ordination is proposed as being the key to a mechanism that is able to maximise and realise any potential opportunity of concurrency. Thus, an agentoriented approach to co-ordination is presented, which incorporates various types of agents responsible for managing their respective activities. The co-ordinated approach, which is implemented within the Design Co-ordination System, includes features such as resource management and monitoring, dynamic scheduling, activity direction, task enactment, and information management. An application of the Design Co-ordination System, in conjunction with a robust concept exploration tool, shows that the computational design analysis involved in evaluating many design concepts can be performed more efficiently through a co-ordinated approach