Wildfire: distributed, Grid-enabled workflow construction and execution

BACKGROUND: We observe two trends in bioinformatics: (i) analyses are increasing in complexity, often requiring several applications to be run as a workflow; and (ii) multiple CPU clusters and Grids are available to more scientists. The traditional solution to the problem of running workflows across multiple CPUs required programming, often in a scripting language such as perl. Programming places such solutions beyond the reach of many bioinformatics consumers. RESULTS: We present Wildfire, a graphical user interface for constructing and running workflows. Wildfire borrows user interface features from Jemboss and adds a drag-and-drop interface allowing the user to compose EMBOSS (and other) programs into workflows. For execution, Wildfire uses GEL, the underlying workflow execution engine, which can exploit available parallelism on multiple CPU machines including Beowulf-class clusters and Grids. CONCLUSION: Wildfire simplifies the tasks of constructing and executing bioinformatics workflows

Leveraging HTC for UK eScience with very large Condor pools: demand for transforming untapped power into results

We provide an insight into the demand from the UK eScience community for very large HighThroughput Computing resources and provide an example of such a resource in current productionuse: the 930-node eMinerals Condor pool at UCL. We demonstrate the significant benefits thisresource has provided to UK eScientists via quickly and easily realising results throughout a rangeof problem areas. We demonstrate the value added by the pool to UCL I.S infrastructure andprovide a case for the expansion of very large Condor resources within the UK eScience Gridinfrastructure. We provide examples of the technical and administrative difficulties faced whenscaling up to institutional Condor pools, and propose the introduction of a UK Condor/HTCworking group to co-ordinate the mid to long term UK eScience Condor development, deploymentand support requirements, starting with the inaugural UK Condor Week in October 2004

Grid Portal Development

The project consists of the analysis, design and implementation of a user account creation system and a notification system for the P-GRADE Grid Portal. The user account creation system expedites the process of accessing a portal by automating many administrative tasks. The notification system provides a useful feature to users of the Portal by alerting them in real time of the status of their workflows. Both systems serve to enhance a user\u27s experience with the Portal

Adaptive Process Distribution at the Edge of IoT using the Integration of BPMS and Containerization

Täna levivad pilvepõhised värkvõrgu (asjade interneti) süsteemid tuginevad protsesside halduseks kaugel asuvatel andmekeskustel, mis toob endaga kaasa latentsusprobleeme. Vastusena sellele probleemile on varem välja pakutud servaarvutuse lähenemine, kus arvutused viiakse läbi asjade interneti süsteemi võrgule füüsiliselt lähemal. Mitmete servaarvutuse metoodikate seas on uduarvutus lähenemine, kus rõhk on arvutuste liigutamisel värkvõrgu seadmetele endile. Ehkki uduarvutusel põhinev arhitektuur on paljutõotav, tõstatab see küsimuse – kuidas värkvõrgu protsessihaldussüsteemid (BPMS4IoT-süsteemid) äriprotsesse heterogeensetele värkvõrgu seadmetele jaotama peaksid? Levinud on lähenemine, kus protsesside töövooülesannete käituseks tuginetakse ühisele platvormile. Näiteks, kui haldusserver defineerib teatud töövoo ülesandena Pythoni skripti ja määrab selle seadmele, siis peab seadme töövookäitusmootor toetama vastavat mehhanismi skriptide jooksutamiseks. Selline nõue ei ole paindlik, arvestades värkvõrgu seadmete heterogeensust. Käesolevas magistritöös pakub autor välja raamistiku, mis eraldab töövoo ülesannete käitusmeetodi käitusmootorist kasutades selleks konteinertehnoloogiat. Töö käigus arendati välja raamistiku prototüüp ning viidi läbi katseid mikroarvutitel põhinevail seadmetel. Lisaks võrreldi väljapakutud uduarvutuse raamistiku jõudlust pilvearvutusel põhineva süsteemiga.Emerging cloud-centric Internet of Things (IoT) system relies on distant data centers to manage the entire processes, which raises the issue of latency. To address the issue, researchers have introduced the Edge computing methodologies that carry out computation closer to the edge network of IoT system. Among the numerous Edge computing approaches, Mist computing paradigm emphasises the mechanism that moves the computation further to the front-end IoT devices. Although the architecture of Mist computing is promising, it raises a new challenge in how the Business Process Management System for IoT (BPMS4IoT) distributes the business process workflow to the heterogeneous IoT devices? In general, executing business process workflows relies on the common platform for executing customized tasks. For example, if the management server defines a Python script task in a workflow, which has been allocated to an IoT device, the workflow engine of the IoT device must have the compatible execution method. Such a requirement is less flexible when one considers the heterogeneity of the IoT devices. Therefore, in this thesis, the author proposes a framework to decouple the workflow task execution method from the workflow engines using the containerization technology. A proof-of-concept prototype has been developed and has been tested on several single-board computers-based IoT devices. Further, a case study has been performed to demonstrate the performance of the proposed framework comparing to the cloud-centric system

Cluster de alto desempenho para uso na disciplina de computação paralela e distribuída utilizando contêineres

Trabalho de Conclusão de Curso, apresentado para obtenção do grau de Bacharel no Curso de Ciência da Computação da Universidade do Extremo Sul Catarinense, UNESC.Nas mais diversas áreas, cientificas e profissionais, é crescente a necessidade por de se obter cada vez mais poder computacional, seja para manter uma enorme quantidade de serviços ativos, ou resolver problemas científicos e matemáticos complexos, porém o alto custo de um supercomputador faz com que se busque alternativas de se obter esse desempenho, sendo uma delas a utilização de clusters como por exemplo o tipo Beowulf que tem como proposta, unir recursos computacionais local ou geograficamente dispersos trazendo vantagem inclusive em relação aos supercomputadores. Porém, mesmo sendo estes mais viáveis financeiramente, ainda representam gastos elevados, levando as empresas e universidades optarem por utilizar ambientes compartilhados na intenção de reduzir os custos, surgindo com isso, problemas oriundos do compartilhamento de recursos, conflitos de dependências entre outras. Como solução para esses problemas, surgem técnicas de virtualização, sendo uma delas os chamados contêineres, que são mais leves e capazes de isolar as aplicações, passando a ser uma boa alternativa para resolver muitos destes problemas. Utilizando essa ideia, o presente trabalho busca alternativas de implementar um cluster de alto desempenho baseado em contêineres, para ser utilizado na disciplina de computação paralela e distribuída, e com isso evitar problemas de compatibilidade, falta de infraestrutura necessária para a implementação de uma cluster físico e evitar conflitos de configuração devido a uma instalação malsucedida, além de acelerar o processo de instalação do cluster, e com isso dar aos alunos a experiência de desenvolverem programas paralelos, em um ambiente próximo ao real

Cluster Computing: A Novel Peer-to-Peer Cluster for Generic Application Sharing

Ph.DDOCTOR OF PHILOSOPH

Enhancing reliability with Latin Square redundancy on desktop grids.

Computational grids are some of the largest computer systems in existence today. Unfortunately they are also, in many cases, the least reliable. This research examines the use of redundancy with permutation as a method of improving reliability in computational grid applications. Three primary avenues are explored - development of a new redundancy model, the Replication and Permutation Paradigm (RPP) for computational grids, development of grid simulation software for testing RPP against other redundancy methods and, finally, running a program on a live grid using RPP. An important part of RPP involves distributing data and tasks across the grid in Latin Square fashion. Two theorems and subsequent proofs regarding Latin Squares are developed. The theorems describe the changing position of symbols between the rows of a standard Latin Square. When a symbol is missing because a column is removed the theorems provide a basis for determining the next row and column where the missing symbol can be found. Interesting in their own right, the theorems have implications for redundancy. In terms of the redundancy model, the theorems allow one to state the maximum makespan in the face of missing computational hosts when using Latin Square redundancy. The simulator software was developed and used to compare different data and task distribution schemes on a simulated grid. The software clearly showed the advantage of running RPP, which resulted in faster completion times in the face of computational host failures. The Latin Square method also fails gracefully in that jobs complete with massive node failure while increasing makespan. Finally an Inductive Logic Program (ILP) for pharmacophore search was executed, using a Latin Square redundancy methodology, on a Condor grid in the Dahlem Lab at the University of Louisville Speed School of Engineering. All jobs completed, even in the face of large numbers of randomly generated computational host failures