Server‐side workflow execution using data grid technology for reproducible analyses of data‐intensive hydrologic systems

Many geoscience disciplines utilize complex computational models for advancing understanding and sustainable management of Earth systems. Executing such models and their associated data preprocessing and postprocessing routines can be challenging for a number of reasons including (1) accessing and preprocessing the large volume and variety of data required by the model, (2) postprocessing large data collections generated by the model, and (3) orchestrating data processing tools, each with unique software dependencies, into workflows that can be easily reproduced and reused. To address these challenges, the work reported in this paper leverages the Workflow Structured Object functionality of the Integrated Rule‐Oriented Data System and demonstrates how it can be used to access distributed data, encapsulate hydrologic data processing as workflows, and federate with other community‐driven cyberinfrastructure systems. The approach is demonstrated for a study investigating the impact of drought on populations in the Carolinas region of the United States. The analysis leverages computational modeling along with data from the Terra Populus project and data management and publication services provided by the Sustainable Environment‐Actionable Data project. The work is part of a larger effort under the DataNet Federation Consortium project that aims to demonstrate data and computational interoperability across cyberinfrastructure developed independently by scientific communities.Plain Language SummaryExecuting computational workflows in the geosciences can be challenging, especially when dealing with large, distributed, and heterogeneous data sets and computational tools. We present a methodology for addressing this challenge using the Integrated Rule‐Oriented Data System (iRODS) Workflow Structured Object (WSO). We demonstrate the approach through an end‐to‐end application of data access, processing, and publication of digital assets for a scientific study analyzing drought in the Carolinas region of the United States.Key PointsReproducibility of data‐intensive analyses remains a significant challengeData grids are useful for reproducibility of workflows requiring large, distributed data setsData and computations should be co‐located on servers to create executable Web‐resourcesPeer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/137520/1/ess271_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/137520/2/ess271.pd

Constructing distributed time-critical applications using cognitive enabled services

Time-critical analytics applications are increasingly making use of distributed service interfaces (e.g., micro-services) that support the rapid construction of new applications by dynamically linking the services into different workflow configurations. Traditional service-based applications, in fixed networks, are typically constructed and managed centrally and assume stable service endpoints and adequate network connectivity. Constructing and maintaining such applications in dynamic heterogeneous wireless networked environments, where limited bandwidth and transient connectivity are commonplace, presents significant challenges and makes centralized application construction and management impossible. In this paper we present an architecture which is capable of providing an adaptable and resilient method for on-demand decentralized construction and management of complex time-critical applications in such environments. The approach uses a Vector Symbolic Architecture (VSA) to compactly represent an application as a single semantic vector that encodes the service interfaces, workflow, and the time-critical constraints required. By extending existing services interfaces, with a simple cognitive layer that can interpret and exchange the vectors, we show how the required services can be dynamically discovered and interconnected in a completely decentralized manner. We demonstrate the viability of this approach by using a VSA to encode various time-critical data analytics workflows. We show that these vectors can be used to dynamically construct and run applications using services that are distributed across an emulated Mobile Ad-Hoc Wireless Network (MANET). Scalability is demonstrated via an empirical evaluation

Dynamic distributed orchestration of Node-RED IOT workflows using a vector symbolic architecture

There are a large number of workflow systems designed to work in various scientific domains, including support for the Internet of Things (IoT). One such workflow system is Node-RED, which is designed to bring workflow-based programming to IoT. However, the majority of scientific workflow systems, and specifically systems like Node-RED, are designed to operate in a fixed networked environment, which rely on a central point of coordination in order to manage the workflow. The main focus of the work described in this paper is to investigate means whereby we can migrate Node-RED workflows into a decentralized execution environment, so that such workflows can run on Edge networks, where nodes are extremely transient in nature. In this work, we demonstrate the feasibility of such an approach by showing how we can migrate a Node-RED based traffic congestion workflow into a decentralized environment. The traffic congestion algorithm is implemented as a set of Web services within Node-RED and we have architected and implemented a system that proxies the centralized Node-RED services using cognitively-aware wrapper services, designed to operate in a decentralized environment. Our cognitive services use a Vector Symbolic Architecture to semantically represent service descriptions and workflows in a way that can be unraveled on the fly without any central point of control. The VSA-based system is capable of parsing Node-RED workflows and migrating them to a decentralized environment for execution; providing a way to use Node-RED as a front-end graphical composition tool for decentralized workflow

A component-based framework for certification of components in a cloud of HPC services

HPC Shelfis a proposal of a cloud computing platform to provide component-oriented services for High Performance Computing (HPC) applications. This paper presents a Verification-as-a-Service (VaaS) framework for component certification onHPC Shelf. Certification is aimed at providing higher confidence that components of parallel computing systems ofHPC Shelfbehave as expected according to one or more requirements expressed in their contracts. To this end, new abstractions are introduced, starting with certifier components. They are designed to inspect other components and verify them for different types of functional, non-functional and behavioral requirements. The certification framework is naturally based on parallel computing techniques to speed up verification tasks.NORTE-01-0145- FEDER-000037

Efficient orchestration of Node-RED IoT workflows using a vector symbolic architecture

Numerous workflow systems span multiple scientific domains and environments, and for the Internet of Things (IoT), Node-RED offers an attractive Web based user interface to execute IoT service-based workflows. However, like most workflow systems, it coordinates the workflow centrally, and cannot run within more transient environments where nodes are mobile. To address this gap, we show how Node-RED workflows can be migrated into a decentralized execution environment for operation on mobile ad-hoc networks, and we demonstrate this by converting a Node-RED based traffic congestion detection workflow to operate in a decentralized environment. The approach uses a Vector Symbolic Architecture (VSA) to dynamically convert Node-Red applications into a compact semantic vector representation that encodes the service interfaces and the workflow in which they are embedded. By extending existing services interfaces, with a simple cognitive layer that can interpret and exchange the vectors, we show how the required services can be dynamically discovered and interconnected into the required workflow in a completely decentralized manner. The resulting system provides a convenient environment where the Node-RED front-end graphical composition tool can be used to orchestrate decentralized workflows. In this paper, we further extend this work by introducing a new dynamic VSA vector compression scheme that compresses vectors for on-the-wire communication, thereby reducing communication bandwidth while maintaining the semantic information content. This algorithm utilizes the holographic properties of the symbolic vectors to perform compression taking into consideration the number of combined vectors along with similarity bounds that determine conflict with other encoded vectors used in the same context. The resulting savings make this approach extremely efficient for discovery in service-based decentralized workflows

Publish/subscribe scientific workflow interoperability framework (PS-SWIF)

Different or similar workflow systems, hosted anywhere on a network, written in any language and running on different operating systems, can easily use the full range of PS-SWIF tools to interoperate with each other. The PS-SWIF approach provides interoperability among a wide range of scientific workflow systems

Semantic technologies for the domain specific and formal description of time series in databases

Messdaten werden zur effizienten Organisation und Weiterverarbeitung in relationalen Datenbanken gespeichert. Die in den letzten Jahren entstandenen Semantic Web Technologien bieten eine hervorragende Basis zur Wissensmodellierung und Beschreibung von Domäneninhalten in Form von Ontologien. Aufgrund der offenen Architektur dieses Ansatzes können leicht fremde Ontologien und Ressourcen mit eingebunden und berücksichtigt werden. Semantic Web Technologien stellen eine formale Modellierungsgrundlage dar. Mittels Reasoning kann deshalb aus Ontologien implizites Wissen abgeleitet werden. In dieser Arbeit werden semantische (Datenbank-) Annotationen und deren Interpretation fokussiert. Sie verknüpfen Datenbanken und das Semantic Web miteinander. Die Annotationen erlauben es, Inhalte von Datenbanken mit Semantic Web Technologien in verschiedenen Nutzungsszenarien zu beschreiben. Außerdem wird für die gemeinsame Behandlung und den Einsatz beider Technologien eine Architektur entwickelt. Auf dieser Basis werden Konzepte zur Visualisierung und Interaktion mit den Annotationen eingeführt. Weiterhin wird deren Einsatz zur formalen Modellierung von Ereignissen mittels Automaten betrachtet, sodass ein Reasoning zur Berechnung durchgeführt werden kann. Mittels einer Implementierung werden die eingeführten Konzepte demonstriert. Die Applikation Semantic Database Browser erlaubt die integrierte Verwendung von Messdaten und deren formaler Beschreibung. Modelle können ausgetauscht und wiederverwendet werden, sodass die Wiederverwendung von Wissen gefördert wird. Anhand des Beispiels von Ereignissen während Autofahrten wird demonstriert, wie auf Basis der formalen Beschreibung Schlussfolgerungen gezogen werden können. So können durch das Schlussfolgern ohne zusätzlichen Aufwand neue Erkenntnisse über auftretende Fahrmanöver generiert werden. Aufgrund des domänenunabhängigen Charakters der skizzierten Lösungsansätze wird gezeigt, dass diese sich leicht auf andere Anwendungsfälle anwenden lassen.Measurement data in form of time series of scientific experiments is stored in relational databases for efficient processing. Complementary, Semantic Web technologies have been developed in the last years for describing domain knowledge in form of ontologies. Due to their open architecture, foreign ontologies and resources can be easily referenced and integrated. Since Semantic Web technologies are based on predicate logic, they are suitable for formal modeling. Therefore, using reasoning implicit knowledge can be derived from ontologies. This work introduces semantic (database) annotations to link databases and ontologies to take advantage of both together by describing database contents with Semantic Web technologies. An architecture is developed for the combined handling and usage of these two technologies, which is designed in respect of scalability of large amounts of measurement data. Based on this architecture, concepts for visualizing and interacting with annotations are introduced. Furthermore, semantic annotations are used for formally modeling events in time series using finite state machines, which are computed using reasoning. An implementation is introduced to demonstrate the feasibility and advantages of the discussed concepts. The presented application Semantic Database Browser allows using semantic database annotations and interactively working with them for integrated handling of formally described measurement data. Formal models can be easily exchanged and reused to support reusability of knowledge and cooperation. By describing measurement data with models, data becomes much easier to understand. Using an example of events during driving, it is demonstrated how formal description can be used for automatic reasoning to generate additional knowledge about driving maneuvers without any additional effort. Because the presented approaches are domain independent, they can be easily adapted for other use cases

WS-RF workflow in Triana

This paper describes the integration of WS-RF functionality into the graphical workflow enactment engine Triana. The motivation for our work is the belief that standardized communication models as represented by the numerous Web services related specifications will enrich the semantic content of service intercommunication and are likely to achieve widespread adoption. These two features will help to facilitate the kind of dynamic binding of services advocated by Service Oriented Architectures. Furthermore we maintain that exposing the capabilities offered by these specifications to the end user in a simple and intuitive manner is essential if they are to be used intelligently and flexibly, particularly when the specifications do not address the functional semantics of a service as is the case with WS-RF. We examine the possibilities that WS-RF offers to workflow creation and execution, particularly how it enables a number of optimizations which are useful in scientific workflows to be performed in a standardized manner, and describe the visual interpretation of WS-RF, the underlying technologies that enable this interpretation as well as issues arising from the integration of WS-RF into Triana