Productivity equation and the m distributions of information processing in workflows

This research investigates an equation of productivity for workflows regarding its robustness towards the definition of workflows as probabilistic distributions. The equation was formulated across its derivations through a theoretical framework about information theory, probabilities and complex adaptive systems. By defining the productivity equation for organism-object interactions, workflows mathematical derivations can be predicted and monitored without strict empirical methods and allows workflow flexibility for organism-object environments.Comment: 6 pages, 0 figure

Idle block based methods for cloud workflow scheduling with preemptive and non-preemptive tasks

[EN] Complex workflow applications are widely used in scientific computing and economic analysis, which commonly include both preemptive and non-preemptive tasks. Cloud computing provides a convenient way for users to access different resources based on the ¿pay-as-you-go¿ model. However, different resource renting alternatives (reserved, on-demand or spot) are usually provided by the service provider. The spot instances provide a dynamic and cheaper alternative comparing to the on-demand one. However, failures often occur due to the fluctuations of the price of the instance. It is a big challenge to determine the appropriate amount of spot and on-demand resources for workflow applications with both preemptive and non-preemptive tasks. In this paper, the workflow scheduling problem with both spot and on-demand instances is considered. The objective is to minimize the total renting cost under deadline constrains. An idle time block-based method is proposed for the considered problem. Different idle time block-based searing and improving strategies are developed to construct schedules for workflow applications. Schedules are improved by a forward and backward moving mechanism. Experimental and statistical results demonstrate the effectiveness of the proposed algorithm over a lot of tests with different sizes.This work is supported by the National Natural Science Foundation of China (No. 61572127, 61272377), the National Key Research and Development Program of China (No. 2017YFB1400800). Ruben Ruiz is partially supported by the Spanish Ministry of Economy and Competitiveness, under the project "SCHEYARD - Optimization of Scheduling Problems in Container Yards" (No. DPI2015-65895-R) financed by FEDER funds.Chen, L.; Li, X.; Ruiz García, R. (2018). Idle block based methods for cloud workflow scheduling with preemptive and non-preemptive tasks. Future Generation Computer Systems. 89:659-669. https://doi.org/10.1016/j.future.2018.07.037S6596698

Stochastic scheduling and workload allocation : QoS support and profitable brokering in computing grids

Abstract: The Grid can be seen as a collection of services each of which performs some functionality. Users of the Grid seek to use combinations of these services to perform the overall task they need to achieve. In general this can be seen as aset of services with a workflow document describing how these services should be combined. The user may also have certain constraints on the workflow operations, such as execution time or cost ----t~ th~ user, specified in the form of a Quality of Service (QoS) document. The users . submit their workflow to a brokering service along with the QoS document. The brokering service's task is to map any given workflow to a subset of the Grid services taking the QoS and state of the Grid into account -- service availability and performance. We propose an approach for generating constraint equations describing the workflow, the QoS requirements and the state of the Grid. This set of equations may be solved using Mixed-Integer Linear Programming (MILP), which is the traditional method. We further develop a novel 2-stage stochastic MILP which is capable of dealing with the volatile nature of the Grid and adapting the selection of the services during the lifetime of the workflow. We present experimental results comparing our approaches, showing that the . 2-stage stochastic programming approach performs consistently better than other traditional approaches. Next we addresses workload allocation techniques for Grid workflows in a multi-cluster Grid We model individual clusters as MIMIk. queues and obtain a numerical solutio~ for missed deadlines (failures) of tasks of Grid workflows. We also present an efficient algorithm for obtaining workload allocations of clusters. Next we model individual cluster resources as G/G/l queues and solve an optimisation problem that minimises QoS requirement violation, provides QoS guarantee and outperforms reservation based scheduling algorithms. Both approaches are evaluated through an experimental simulation and the results confirm that the proposed workload allocation strategies combined with traditional scheduling algorithms performs considerably better in terms of satisfying QoS requirements of Grid workflows than scheduling algorithms that don't employ such workload allocation techniques. Next we develop a novel method for Grid brokers that aims at maximising profit whilst satisfying end-user needs with a sufficient guarantee in a volatile utility Grid. We develop a develop a 2-stage stochastic MILP which is capable of dealing with the volatile nature . of the Grid and obtaining cost bounds that ensure that end-user cost is minimised or satisfied and broker's profit is maximised with sufficient guarantee. These bounds help brokers know beforehand whether the budget limits of end-users can be satisfied and. if not then???????? obtain appropriate future leases from service providers. Experimental results confirm the efficacy of our approach.Imperial Users onl

Modeling and verification of web service composition based interorganizational workflows

Interorganisationale Workflows sind Arbeitsabläufe, welche die Grenzen einer Organisation verlassen und einen Rahmen für Kooperationen der verschiedenen autonomen Organisationen zur Verfügung stellen. Ein wichtiger Punkt für den Entwurf solcher Workflows ist die Balance zwischen Offenheit und Abgrenzung, wobei erstere für Kooperationen und letztere die für den Schutz von Know-how benötigt wird. Workflow Sichten stellen ein effizientes Werkzeug für diesen Zweck zur Verfügung. Durch Offenlegung von bestimmten Teilen eines Prozesses, können Organisationen sowohl kooperieren als auch das Know-how schützen. Diese Dissertation präsentiert nun eine Methode für die korrekte Konstruktion von Workflow Sichten. Es wird angenommen, dass Organisationen Web Service orientierte Technologien zur Modellierung und Implementierung von interorganisationalen Workflows verwenden. Die Anwendung von Web Services bietet Organisationen viele Vorteile. Den eigentlichen Mehrwert von Web Services stellt aber die Kompositionsfähigkeit dar. Verfügbare Web Services können dadurch von anderen Choreographien und Orchestrationen (wieder-)verwendet werden. Die Notwendigkeit der Implementierung von Systemen von Null weg kann minimiert werden. Die zentralen Anforderungen sind einerseits eine Architektur mit adäquatem Potential, andererseits die Verifikation der Korrektheit. Diese Dissertation präsentiert nun eine Architektur zur Modellierung von Web Service Composition basierten interorganisationalen Workflows, genannt föderierte Choreographien, die verglichen mit anderen Architekturen verschiedene Vorteile anbieten. Darüber hinaus werden Algorithmen und Techniken zur Verifikation der strukturellen und temporalen Korrektheit vorgestellt. Strukturelle Korrektheit prüft, ob die Strukturen der beteiligten Prozesse zusammenpassen. Temporale Korrektheit überprüft, ob ein interorganisationaler Workflow, der aus mehreren Choreographien und Orchestrationen besteht hinsichtlich der lokalen und globalen Bedingungen fehlerfrei ist. Mit Hilfe dieser Techniken kann die strukturelle und temporale Konformität des Modells zur Designzeit überprüft werden. Falls das Modell nicht strukturell oder temporal konform ist, können nötige Änderungen durchgeführt werden, sodass die korrekte Ausführung zur Laufzeit garantiert werden kann. Die Überprüfung der Konformität zur Designzeit reduziert die Prozesskosten vor allem wegen den folgenden zwei Gründen: Erstens, die entdeckten Fehler zur Designzeit sind normalerweise billiger als jene, die zur Laufzeit entdeckt werden und zweitens, Fehlerbehandlungsmechanismen können verhindert werden, die wiederum Zusatzkosten verursachen. Zusätzlich zu der vorgestellten Architektur wird eine allgemeinere Architektur zusammen mit den passenden Konformitätsprüfungsalgorithmen präsentiert. Der Ansatz ist Platform- und sprachunabhängig und die Algorithmen sind verteilt.Interorganizational workflows are workflows that cross the boundaries of a single organization and provide a framework for cooperation of different autonomous organizations. An important issue when designing such workflows is the balance between the openness needed for cooperation and the privacy needed for protection of business know-how. Workflow views provide an efficient tool for this aim. By exposure of only selected parts of a process, organizations can both cooperate and protect their business logic. This dissertation presents a technique for a correct construction of workflow views. It is assumed that organizations and partners use web services and web service related technology to model and implement interorganizational workflows. Application of web services offers several advantages for organizations. The real surplus of web services is their capability of being composed to more complex systems. Available web services can be reused by other choreographies and orchestrations and the need for development of new systems from scratch can be minimized. The essential requirements are on the one hand an architecture with adequate capabilities and on the other hand, verification of correctness. This dissertation proposes an architecture for modeling web service composition based interorganizational workflows, called \emph{federated choreographies}, that provides several advantages compared to existing proposals. Moreover, algorithms and techniques for verification of structural and temporal correctness of interorganizational workflows are proposed. Structural conformance checks if the structures of the involved processes match. Temporal conformance checks if an interorganizational workflow composed of choreographies and orchestrations is temporally error-free with respect to local and global temporal constraints. The proposed algorithms can be applied for checking the structural and temporal conformance of the federated choreographies at design-time. If the model is not structurally or temporally conformant, necessary modifications can be done such that the correct execution of the flow at run-time can be guaranteed. The conformance checking at design time reduces the cost of process because of two reasons: first, errors detected at design time are normally cheaper than those detected at run time and second, exception handling mechanisms can be avoided which are, in turn, coupled with additional costs. In addition to the proposed architecture, a more general architecture together with the conformance checking algorithms and techniques for interorganizational workflows are presented. The presented approach is language and platform independent and algorithms work in a distributed manner

Trusted resource allocation in volunteer edge-cloud computing for scientific applications

Data-intensive science applications in fields such as e.g., bioinformatics, health sciences, and material discovery are becoming increasingly dynamic and demanding with resource requirements. Researchers using these applications which are based on advanced scientific workflows frequently require a diverse set of resources that are often not available within private servers or a single Cloud Service Provider (CSP). For example, a user working with Precision Medicine applications would prefer only those CSPs who follow guidelines from HIPAA (Health Insurance Portability and Accountability Act) for implementing their data services and might want services from other CSPs for economic viability. With the generation of more and more data these workflows often require deployment and dynamic scaling of multi-cloud resources in an efficient and high-performance manner (e.g., quick setup, reduced computation time, and increased application throughput). At the same time, users seek to minimize the costs of configuring the related multi-cloud resources. While performance and cost are among the key factors to decide upon CSP resource selection, the scientific workflows often process proprietary/confidential data that introduces additional constraints of security postures. Thus, users have to make an informed decision on the selection of resources that are most suited for their applications while trading off between the key factors of resource selection which are performance, agility, cost, and security (PACS). Furthermore, even with the most efficient resource allocation across multi-cloud, the cost to solution might not be economical for all users which have led to the development of new paradigms of computing such as volunteer computing where users utilize volunteered cyber resources to meet their computing requirements. For economical and readily available resources, it is essential that such volunteered resources can integrate well with cloud resources for providing the most efficient computing infrastructure for users. In this dissertation, individual stages such as user requirement collection, user's resource preferences, resource brokering and task scheduling, in lifecycle of resource brokering for users are tackled. For collection of user requirements, a novel approach through an iterative design interface is proposed. In addition, fuzzy interference-based approach is proposed to capture users' biases and expertise for guiding their resource selection for their applications. The results showed improvement in performance i.e. time to execute in 98 percent of the studied applications. The data collected on user's requirements and preferences is later used by optimizer engine and machine learning algorithms for resource brokering. For resource brokering, a new integer linear programming based solution (OnTimeURB) is proposed which creates multi-cloud template solutions for resource allocation while also optimizing performance, agility, cost, and security. The solution was further improved by the addition of a machine learning model based on naive bayes classifier which captures the true QoS of cloud resources for guiding template solution creation. The proposed solution was able to improve the time to execute for as much as 96 percent of the largest applications. As discussed above, to fulfill necessity of economical computing resources, a new paradigm of computing viz-a-viz Volunteer Edge Computing (VEC) is proposed which reduces cost and improves performance and security by creating edge clusters comprising of volunteered computing resources close to users. The initial results have shown improved time of execution for application workflows against state-of-the-art solutions while utilizing only the most secure VEC resources. Consequently, we have utilized reinforcement learning based solutions to characterize volunteered resources for their availability and flexibility towards implementation of security policies. The characterization of volunteered resources facilitates efficient allocation of resources and scheduling of workflows tasks which improves performance and throughput of workflow executions. VEC architecture is further validated with state-of-the-art bioinformatics workflows and manufacturing workflows.Includes bibliographical references