Multi-tenant Pub/Sub processing for real-time data streams

Devices and sensors generate streams of data across a diversity of locations and protocols. That data usually reaches a central platform that is used to store and process the streams. Processing can be done in real time, with transformations and enrichment happening on-the-fly, but it can also happen after data is stored and organized in repositories. In the former case, stream processing technologies are required to operate on the data; in the latter batch analytics and queries are of common use. This paper introduces a runtime to dynamically construct data stream processing topologies based on user-supplied code. These dynamic topologies are built on-the-fly using a data subscription model defined by the applications that consume data. Each user-defined processing unit is called a Service Object. Every Service Object consumes input data streams and may produce output streams that others can consume. The subscription-based programing model enables multiple users to deploy their own data-processing services. The runtime does the dynamic forwarding of data and execution of Service Objects from different users. Data streams can originate in real-world devices or they can be the outputs of Service Objects. The runtime leverages Apache STORM for parallel data processing, that combined with dynamic user-code injection provides multi-tenant stream processing topologies. In this work we describe the runtime, its features and implementation details, as well as we include a performance evaluation of some of its core components.This work is partially supported by the European Research Council (ERC) un- der the EU Horizon 2020 programme (GA 639595), the Spanish Ministry of Economy, Industry and Competitivity (TIN2015-65316-P) and the Generalitat de Catalunya (2014-SGR-1051).Peer ReviewedPostprint (author's final draft

aDORe djatoka: An Open-Source Jpeg 2000 Image Server and Dissemination Service Framework

4th International Conference on Open RepositoriesThis presentation was part of the session : Conference PresentationsDate: 2009-05-19 03:00 PM – 04:30 PMThe JPEG 2000 image format has attracted considerable attention due to its rich feature set defined in a multi-part open ISO standard, and its potential use as a holy-grail preservation format providing both lossless compression and rich service format features. Until recently there was lack of an implementation agnostic (e.g., Kakadu, Aware, etc) API for JPEG 2000 compression and extraction, and an open-source service framework, upon which rich Web 2.0-style applications can be developed. Recently we engaged in the development of aDORe djatoka , an open-source JPEG 2000 image server and dissemination framework to help address some of these issues. The djatoka image server is geared towards Web 2.0 style reuse through URI-addressability of all image disseminations including regions, rotations, and format transformations. Djatoka also provides a JPEG 2000 compression / extraction API that serves as an abstraction layer from the underlying JPEG 2000 library (e.g., Kakadu, Aware, etc).  The initial release has attracted considerable interest and is already being used in production environments, such as at the Biodiversity Heritage Library , who uses djatoka to serve more than eleven million images. This presentation introduces the aDORe djatoka image server and describes various interoperability approaches with existing repository systems.  Djatoka was derived from a concrete need to introduce a solution to disseminate high-resolution images stored in an aDORe repository system.  Djatoka is able to disseminate images that reside either in a repository environment or that are Web-accessible at arbitrary URIs.  Since dynamic service requests pertaining to an identified resource (the entire JPEG 2000 image) are being made, the OpenURL Framework was selected to provide an extensible dissemination service framework. The OpenURL service layer simplifies development and provides exciting interoperability opportunities. The presentation will showcase the flexibility of this interface by introducing a mobile image collection viewer developed for the iPhone / iTouch platform

XRound : A reversible template language and its application in model-based security analysis

Successful analysis of the models used in Model-Driven Development requires the ability to synthesise the results of analysis and automatically integrate these results with the models themselves. This paper presents a reversible template language called XRound which supports round-trip transformations between models and the logic used to encode system properties. A template processor that supports the language is described, and the use of the template language is illustrated by its application in an analysis workbench, designed to support analysis of security properties of UML and MOF-based models. As a result of using reversible templates, it is possible to seamlessly and automatically integrate the results of a security analysis with a model. (C) 2008 Elsevier B.V. All rights reserved

Pattern-based software architecture for service-oriented software systems

Service-oriented architecture is a recent conceptual framework for service-oriented software platforms. Architectures are of great importance for the evolution of software systems. We present a modelling and transformation technique for service-centric distributed software systems. Architectural configurations, expressed through hierarchical architectural patterns, form the core of a specification and transformation technique. Patterns on different levels of abstraction form transformation invariants that structure and constrain the transformation process. We explore the role that patterns can play in architecture transformations in terms of functional properties, but also non-functional quality aspects

Microservices and Machine Learning Algorithms for Adaptive Green Buildings

In recent years, the use of services for Open Systems development has consolidated and strengthened. Advances in the Service Science and Engineering (SSE) community, promoted by the reinforcement of Web Services and Semantic Web technologies and the presence of new Cloud computing techniques, such as the proliferation of microservices solutions, have allowed software architects to experiment and develop new ways of building open and adaptable computer systems at runtime. Home automation, intelligent buildings, robotics, graphical user interfaces are some of the social atmosphere environments suitable in which to apply certain innovative trends. This paper presents a schema for the adaptation of Dynamic Computer Systems (DCS) using interdisciplinary techniques on model-driven engineering, service engineering and soft computing. The proposal manages an orchestrated microservices schema for adapting component-based software architectural systems at runtime. This schema has been developed as a three-layer adaptive transformation process that is supported on a rule-based decision-making service implemented by means of Machine Learning (ML) algorithms. The experimental development was implemented in the Solar Energy Research Center (CIESOL) applying the proposed microservices schema for adapting home architectural atmosphere systems on Green Buildings

Why (and How) Networks Should Run Themselves

The proliferation of networked devices, systems, and applications that we depend on every day makes managing networks more important than ever. The increasing security, availability, and performance demands of these applications suggest that these increasingly difficult network management problems be solved in real time, across a complex web of interacting protocols and systems. Alas, just as the importance of network management has increased, the network has grown so complex that it is seemingly unmanageable. In this new era, network management requires a fundamentally new approach. Instead of optimizations based on closed-form analysis of individual protocols, network operators need data-driven, machine-learning-based models of end-to-end and application performance based on high-level policy goals and a holistic view of the underlying components. Instead of anomaly detection algorithms that operate on offline analysis of network traces, operators need classification and detection algorithms that can make real-time, closed-loop decisions. Networks should learn to drive themselves. This paper explores this concept, discussing how we might attain this ambitious goal by more closely coupling measurement with real-time control and by relying on learning for inference and prediction about a networked application or system, as opposed to closed-form analysis of individual protocols

Interface refactoring in performance-constrained web services

This paper presents the development of REF-WS an approach to enable a Web Service provider to reliably evolve their service through the application of refactoring transformations. REF-WS is intended to aid service providers, particularly in a reliability and performance constrained domain as it permits upgraded ’non-backwards compatible’ services to be deployed into a performance constrained network where existing consumers depend on an older version of the service interface. In order for this to be successful, the refactoring and message mediation needs to occur without affecting functional compatibility with the services’ consumers, and must operate within the performance overhead expected of the original service, introducing as little latency as possible. Furthermore, compared to a manually programmed solution, the presented approach enables the service developer to apply and parameterize refactorings with a level of confidence that they will not produce an invalid or ’corrupt’ transformation of messages. This is achieved through the use of preconditions for the defined refactorings