On-demand distributed image processing over an adaptive Campus-Grid

This thesis explores how scientific applications, which are based upon short jobs (seconds and minutes) can capitalize upon the idle workstations of a Campus-Grid. These resources are donated on a voluntary basis, and consequently, the Campus-Grid is constantly adapting and the availability of workstations changes. Typically, to utilize these resources a Condor system or equivalent would be used. However, such systems are designed with different trade-offs and incentives in mind and therefore do not provide intrinsic support for short jobs. The motivation for creating a provisioning scenario for short jobs is that Image Processing, as well as other areas of scientific analysis, are typically composed of short running jobs, but still require parallel solutions. Much of the literature in this area comments on the challenges of performing such analysis efficiently and effectively even when dedicated resources are in use. The main challenges are: latency and scheduling penalties, granularity and the potential for very short jobs. A volunteer Grid retains these challenges but also adds further challenges. These can be summarized as: unpredictable re source availability and longevity, multiple machine owners and administrators who directly affect the operating environment. Ultimately, this creates the requirement for well conceived and effective fault management strategies. However, these are typically not in place to enable transparent fault-free job administration for the user. This research demonstrates that these challenges are answerable, and that in doing so opportunistically sourced Campus-Grid resources can host disparate applications constituted of short running jobs, of as little as one second in length. This is demonstrated by the significant improvements in performance when the system presented here was compared to a well established Condor system. Here, improvements are increased job efficiency from 60–70% to 95%–100%, up to a 99% reduction in application makespan and up to a 13000% increase in the efficiency of resource utilization. The Condor pool in use is approximately 1,600 workstations distributed across 27 administrative domains of Cardiff University. The application domain of this research is Matlab-based image processing, and the application area used to demonstrate the approach is the analysis of Magnetic Resonance Imagery (MRI). However, the presented approach is generalizable to any application domain with similar characteristics

Speeding up systems biology simulations of biochemical pathways using condor

This is the accepted version of the following article: Speeding up Systems Biology Simulations of Biochemical Pathways using Condor". Concurrency and Computation: Practice and Experience Volume 26, Issue 17, pages 2727–2742, 10 December 2014 which has been published in final form at http://onlinelibrary.wiley.com/doi/10.1002/cpe.3161/abstractSystems biology is a scientific field that uses computational modelling to study biological and biochemical systems. The simulation and analysis of models of these systems typically explore behaviour over a wide range of parameter values; as such, they are usually characterised by the need for nontrivial amounts of computing power. Grid computing provides access to such computational resources. In previous research, we created the grid-enabled biochemical networks simulation environment to attempt to speed up system biology simulations over a grid (the UK National Grid Service and ScotGrid). Following on from this work, we have created the simulation modelling of the epidermal growth factor receptor microtubule-associated protein kinase pathway utility, a standalone simulation tool dedicated to the modelling and analysis of the epidermal growth factor receptor microtubule-associated protein kinase pathway. This builds on experiences from biochemical networks simulation environment by decoupling the simulation modelling elements from the Grid middleware. This new utility enables us to interface with different grid technologies. This paper therefore describes the new SIMAP utility and an empirical investigation of its performance when deployed over a desktop grid based on the high throughput computing middleware Condor. We present our results based on a case study with a model of the mammalian ErbB signalling pathway, a pathway strongly linked to cance

SLA Establishment Decisions: Minimizing the Risk of SLA Violations

This thesis presents an approach for service providers to select an SLA portfolio that minimizes the SLA violation risk. It considers constraints on expected profit and available resources. The problem is addressed by applying decision theory and risk measures, especially by adapting the concept of portfolio selection by Harry Markowitz and the semi-variance. In order to capture a decision maker\u27s attitude towards risk, utility theory and the concept of risk aversion are used

Distributed image processing over an adaptive Campus Grid

A system implemented in MATLAB is described, which may be deployed over a Campus Grid utilizing the Condor job management system. Our approach can re-distribute jobs as node availability changes. The architecture of the system, its components, and their deployment across the Cardiff University Campus Grid (consisting of 2500 machines) are presented. Challenges in image processing applications that can be deployed over such infrastructure are presented, along with performance results that demonstrate the use of our system alongside a standard Condor deployment, demonstrating a significant increase in throughput using our approach