An Intelligent Robust Mouldable Scheduler for HPC & Elastic Environments

Traditional scheduling techniques are of a by-gone era and do not cater for the dynamism of new and emerging computing paradigms. Budget constraints now push researchers to migrate their workloads to public clouds or to buy into shared computing services as funding for large capital expenditures are few and far between. The sites still hosting large or shared computing infrastructure have to ensure that the system utilisation and efficiency is as high as ossible. However, the efficiency can not come at the cost of quality of service as the availability of public clouds now means that users can move away. This thesis presents a novel scheduling system to improve job turn-around-time. The Robust Mouldable Scheduler outlined in these pages utilises real application benchmarks to profile system performance and predict job execution times at different allocations, something no other scheduler does at present. The system is able to make an allocation decisions ensuring the jobs can fit into spaces available on the system using fewer resources without delaying the job completion time. The results demonstrate significant improvement in workload turn-around-times using real High Performance Computing (HPC) trace logs. Utilising three years of the University of Huddersfield trace logs the mouldable scheduler consistently simulated faster workload completion. Further, the results establish that by not relying on the user to suggest resource allocations for jobs the system is able to mitigate bad-put into the system leading to improved efficiency. A thorough investigation of Research Computing Systems (RCS), workload management systems, scheduling algorithms and strategies, benchmarking and profiling toolkits, and simulators is presented to establish the state of the art. Within this thesis a method to profile applications and workloads that leverages common open-source tools on HPC systems is presented. The resultant toolkit is used to profile the University of Huddersfield workload. This workload forms the basis to evaluate the mouldable scheduler. The research includes advance computing paradigms such as utilising Artificial Intelligence methods to improve the efficiency of the scheduler, or Surge Computing, where workloads are scaled beyond institutional firewalls through elastic compute systems

Proceedings of the Third International Workshop on Sustainable Ultrascale Computing Systems (NESUS 2016) Sofia, Bulgaria

Proceedings of: Third International Workshop on Sustainable Ultrascale Computing Systems (NESUS 2016). Sofia (Bulgaria), October, 6-7, 2016

Numerical simulation of the structural behaviour of concrete since its early ages

Tese de doutoramento. Engenharia Civil. Faculdade de Engenharia. Universidade do Porto, School of Enegineering. University of Tokyo. 200

Robust mouldable intelligent scheduling using application benchmarking for elastic environments

In a green IT obsessed world hardware efficiency and usage of computer systems becomes essential. There is a multiplier effect when this is applied to High Performance Computing systems. With an average compute rack consuming between 7 and 25kW it is essential that resources be utilised in the most optimum way possible. Currently the batch schedulers employed to manage these multi-user multi-application environments are nothing more than match making and service level agreement (SLA) enforcing tools. System Administrators strive to get maximum “usage efficiency” from the systems by fine-tuning and restricting queues to get a predictable performance characteristic, e.g. any software package running in queue X will take N number of cores and run for a maximum of T time. These fixed approximations of performance characteristics are used then to schedule queued jobs in the system, in the hope of achieving 100% utilisation. Choosing which queue to place a job in, falls on the user. A savvy user may use trial an error to establish which queue is best suited to his/her needs, but most users will find a queue that gives them results and stick to it – even if they change the model being simulated. This usually leads to a job receiving either an over or under allocation of resources, resulting in either hardware failure or inefficient utilisation of the system. Ideally the system should know how a particular application with a particular dataset would behave when run. Benchmarking Schemes have historically been used as marketing and administration tools. Some schemes like Standard Performance Evaluation Corporation (SPEC) and Perfect Benchmark used “real” applications with generic datasets to test a systems performance. This way a scientist looking for a cluster computer could ask questions such as “How well will my software run?” rather than “How many FLOPS can I get out of this system?” If adapted to include an API to plug in any software to benchmark and to pass results to other software, these toolkits can be used for purposes other than sales and marketing. If a job scheduler can get access to performance characteristic curves for every application on the system, optimal resource allocation and scheduling/queuing decisions can be made at submit time by the system rather than the user. This would further improve the performance of Mouldable schedulers that currently follow the Downey model. Along with the decision-making regarding resource allocation and scheduling, if the scheduler is able to collect a historic record of simulations by the particular users, then further optimisation is possible. This would lead to better and safer utilisation of the system. Currently AI is used in some decision making in Mouldable schedulers. Given a user inputted variance of resources required the scheduler makes a decision on resource allocation by selecting from the available range. If the user supplied range is incorrect, the scheduler is powerless to adapt, and on a next run cannot learn from previous mistakes or successes. This project aims to adapt an open-framework benchmarking scheme to feed information to a job scheduler. This job scheduler will also use gathered heuristic data to make scheduling decisions and optimise the resource allocation and the system utilisation. This work will be further expanded to include elastic or even shared resource environments where the scheduler can expand the size of its world based on either financial or SLA driven decision

Analysis of the water distribution main replacement conundrum in Durban.

Thesis (M.Sc.Eng.)-University of KwaZulu-Natal, Durban, 2012.The optimisation of the decision of when to replace water distribution mains is a complex task. There are numerous drivers in the decision making process (informed by financial data, performance data and water quality data) and hundreds of variables and performance indicators that can be considered when trying to reach an optimised decision. Most of the assets under consideration are buried and the internal and external pipe conditions are not easily assessable, forcing the Utility to rely on the available direct and indirect variables from which conclusions on the reliability of the mains are to be inferred. The cost of mains replacement is relatively low but the assessment cost, if carried out can be relatively high. The total value of a metropolitan distribution network typically runs into billions of rands but the impact of an individual pipe failure is generally low. The distribution network is comprised of many different pipe materials and components, of different pressure classes, made by different manufacturers, installed by numerous contractors with different skill levels under differing quality control regimens over many years. To add to this complexity, various parts of the network are operated at different static pressures and varying velocities. Some sections of the network are isolated more often than others and at times there can be large pressure surges that the network is subjected to by either the Utility or Consumer. These pressure surges are known to have a marked detrimental effect on the network. False markers also exist that can give rise to totally incorrect decisions and therefore performance data cannot be accepted at face value and needs to be scrutinised and cleansed to increase its reliability prior to being utilised in decision making process. This important step has been missed by much of the research carried out to date. In the Durban context, a further complication is caused by consumers tampering with the water mains and also not reporting leaks. This has a negative effect on the performance of the water main that can cause it to be flagged for replacement, but its replacement will not result in an increase in performance if the social issues are not resolved first. The aim of this research is to make recommendations on the methodology to be employed to improve network performance and thereby delay the point at which the water mains are to be replaced for as long as possible. These recommended activities will be carried out to remove false markers and improve upon the quality and reliability of the data available on the network performance. A further output is to make recommendations regarding the minimum data that can be reasonably collected and analysed in order to determine an optimised result. The recommendation of which mains should be targeted for replacement should result in the highest benefit for the utility as well as the consumers. By implication, this will result lowest long term capital and operational expenditure and thus the lowest long term tariffs charged to the consumers whilst complying with the water quality criteria and service level targets