Decentralised Orchestration of Service-Oriented Scientific Workflows

Service-oriented architectures, optimisation, scientific workflows, data deluge Service-oriented workflows in the scientific domain are commonly composed as Directed Acyclic Graphs (DAGs), formed from a collection of vertices and directed edges. When orchestrating service-oriented DAGs, intermediate data are typically routed through a single centralised engine, which results in unnecessary data transfer, increasing the execution time of a workflow and causing the engine to become a performance bottleneck. This paper introduces an architecture for deploying and executing a service-oriented DAG-based workflows across a peer-to-peer proxy network. A workflow is divided into a set of vertices, disseminated to a group of proxies and executed without centralised control over a peer-to-peer proxy network. Through a Web services implementation, we demonstrate across PlanetLab that by reducing intermediate data transfer and by sharing the workload between distributed proxies, end-to-end workflows are sped up. Furthermore, our architecture is non-intrusive: Web services owned and maintained by different institutions do not have to be altered prior to execution.