Model-driven development of data intensive applications over cloud resources

The proliferation of sensors over the last years has generated large amounts of raw data, forming data streams that need to be processed. In many cases, cloud resources are used for such processing, exploiting their flexibility, but these sensor streaming applications often need to support operational and control actions that have real-time and low-latency requirements that go beyond the cost effective and flexible solutions supported by existing cloud frameworks, such as Apache Kafka, Apache Spark Streaming, or Map-Reduce Streams. In this paper, we describe a model-driven and stepwise refinement methodological approach for streaming applications executed over clouds. The central role is assigned to a set of Petri Net models for specifying functional and non-functional requirements. They support model reuse, and a way to combine formal analysis, simulation, and approximate computation of minimal and maximal boundaries of non-functional requirements when the problem is either mathematically or computationally intractable. We show how our proposal can assist developers in their design and implementation decisions from a performance perspective. Our methodology allows to conduct performance analysis: The methodology is intended for all the engineering process stages, and we can (i) analyse how it can be mapped onto cloud resources, and (ii) obtain key performance indicators, including throughput or economic cost, so that developers are assisted in their development tasks and in their decision taking. In order to illustrate our approach, we make use of the pipelined wavefront array

A Specification Language for Performance and Economical Analysis of Short Term Data Intensive Energy Management Services

Requirements of Energy Management Services include short and long term processing of data in a massively interconnected scenario. The complexity and variety of short term applications needs methodologies that allow designers to reason about the models taking into account functional and non-functional requirements. In this paper we present a component based specification language for building trustworthy continuous dataflow applications. Component behaviour is defined by Petri Nets in order to translate to the methodology all the advantages derived from a mathematically based executable model to support analysis, verification, simulation and performance evaluation. The paper illustrates how to model and reason with specifications of advanced dataflow abstractions such as smart grids

Application Driven MOdels for Resource Management in Cloud Environments

El despliegue y la ejecución de aplicaciones de gran escala en sistemas distribuidos con unos parametros de Calidad de Servicio adecuados necesita gestionar de manera eficiente los recursos computacionales. Para desacoplar los requirimientos funcionales y los no funcionales (u operacionales) de dichas aplicaciones, se puede distinguir dos niveles de abstracción: i) el nivel funcional, que contempla aquellos requerimientos relacionados con funcionalidades de la aplicación; y ii) el nivel operacional, que depende del sistema distribuido donde se despliegue y garantizará aquellos parámetros relacionados con la Calidad del Servicio, disponibilidad, tolerancia a fallos y coste económico, entre otros. De entre las diferentes alternativas del nivel operacional, en la presente tesis se contempla un entorno cloud basado en la virtualización de contenedores, como puede ofrecer Kubernetes.El uso de modelos para el diseño de aplicaciones en ambos niveles permite garantizar que dichos requerimientos sean satisfechos. Según la complejidad del modelo que describa la aplicación, o el conocimiento que el nivel operacional tenga de ella, se diferencian tres tipos de aplicaciones: i) aplicaciones dirigidas por el modelo, como es el caso de la simulación de eventos discretos, donde el propio modelo, por ejemplo Redes de Petri de Alto Nivel, describen la aplicación; ii) aplicaciones dirigidas por los datos, como es el caso de la ejecución de analíticas sobre Data Stream; y iii) aplicaciones dirigidas por el sistema, donde el nivel operacional rige el despliegue al considerarlas como una caja negra.En la presente tesis doctoral, se propone el uso de un scheduler específico para cada tipo de aplicación y modelo, con ejemplos concretos, de manera que el cliente de la infraestructura pueda utilizar información del modelo descriptivo y del modelo operacional. Esta solución permite rellenar el hueco conceptual entre ambos niveles. De esta manera, se proponen diferentes métodos y técnicas para desplegar diferentes aplicaciones: una simulación de un sistema de Vehículos Eléctricos descrita a través de Redes de Petri; procesado de algoritmos sobre un grafo que llega siguiendo el paradigma Data Stream; y el propio sistema operacional como sujeto de estudio.En este último caso de estudio, se ha analizado cómo determinados parámetros del nivel operacional (por ejemplo, la agrupación de contenedores, o la compartición de recursos entre contenedores alojados en una misma máquina) tienen un impacto en las prestaciones. Para analizar dicho impacto, se propone un modelo formal de una infrastructura operacional concreta (Kubernetes). Por último, se propone una metodología para construir índices de interferencia para caracterizar aplicaciones y estimar la degradación de prestaciones incurrida cuando dos contenedores son desplegados y ejecutados juntos. Estos índices modelan cómo los recursos del nivel operacional son usados por las applicaciones. Esto supone que el nivel operacional maneja información cercana a la aplicación y le permite tomar mejores decisiones de despliegue y distribución.<br /

An Edge Computing Based Smart Healthcare Framework for Resource Management

The revolution in information technologies, and the spread of the Internet of Things (IoT) and smart city industrial systems, have fostered widespread use of smart systems. As a complex, 24/7 service, healthcare requires efficient and reliable follow-up on daily operations, service and resources. Cloud and edge computing are essential for smart and efficient healthcare systems in smart cities. Emergency departments (ED) are real-time systems with complex dynamic behavior, and they require tailored techniques to model, simulate and optimize system resources and service flow. ED issues are mainly due to resource shortage and resource assignment efficiency. In this paper, we propose a resource preservation net (RPN) framework using Petri net, integrated with custom cloud and edge computing suitable for ED systems. The proposed framework is designed to model non-consumable resources and is theoretically described and validated. RPN is applicable to a real-life scenario where key performance indicators such as patient length of stay (LoS), resource utilization rate and average patient waiting time are modeled and optimized. As the system must be reliable, efficient and secure, the use of cloud and edge computing is critical. The proposed framework is simulated, which highlights significant improvements in LoS, resource utilization and patient waiting time

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications

Security in DevOps: understanding the most efficient way to integrate security in the agile software development process

Modern development methodologies follow a fast and dynamic pace, which gives great attention to customers’ satisfaction in the delivery of new releases. On the other hand, the work pursued to secure a system, if not adapted to the new development trend, can risk to slow down the delivery of new software and the adaptability typical for an Agile environment. Therefore, it is paramount to think about a new way to integrate security into the development framework, in order to secure the software in the best way without slowing down the pace of the developers. Moreover, the implementation of automatic and repeatable security controls inside the development pipeline can help to catch the presence of vulnerabilities as early as possible, thus reducing costs, comparing to solving the issues at later stages. The thesis presents a series of recommendations on how to best deploy a so called DevSecOps approach and applies the theory to the use case of Awake.AI, a Finnish startup company focusing its business on the maritime industry. It is not always easy and feasible to practically apply all the suggestions presented in the literature to a real case scenario, but rather the recommendations need to be adapted and forged in a way that best suits the situation and the current target. It is undeniable that the presence of a strong and efficient secure development framework can give substantial advantage to the success of a company. In fact, not only it makes sure that the delivery of good quality code to the customers is not slowed down, but it also dramatically reduces the risk of incurring in expensive security incidents. Lastly, it is valuable to also mention that, being able to show a clean and efficient approach to security, the framework improves the reputation and trustfulness of the company under the eyes of the customers

Leveraging Cloud-based NFV and SDN Platform Towards Quality-Driven Next-Generation Mobile Networks

Network virtualization has become a key approach for Network Service Providers (NSPs) to mitigate the challenge of the continually increasing demands for network services. Tightly coupled with their software components, legacy network devices are difficult to upgrade or modify to meet the dynamically changing end-user needs. To virtualize their infrastructure and mitigate those challenges, NSPs have started to adopt Software Defined Networking (SDN) and Network Function Virtualization (NFV). To this end, this thesis addresses the challenges faced on the road of transforming the legacy networking infrastructure to a more dynamic and agile virtualized environment to meet the rapidly increasing demand for network services and serve as an enabler for key emerging technologies such as the Internet of Things (IoT) and 5G networking. The thesis considers different approaches and platforms to serve as an NFV/SDN based cloud applications while closely considering how such an environment deploys its virtualized services to optimize the network and reducing their costs. The thesis starts first by defining the standards of adopting microservices as architecture for NFV. Then, it focuses on the latency-aware deployment approach of virtual network functions (VNFs) forming service function chains (SFC) in a cloud environment. This approach ensures that NSPs still meet their strict quality of service and service level agreements while considering both functional and non-functional constraints of the NFV-based applications such as, delay, resource allocation, and intercorrelation between VNF instances. In addition, the thesis proposes a detailed approach on recovering and handling of those instances by optimizing the decision of migrating or re-instantiating the virtualized services upon a sudden event (failure/overload…). All the proposed approaches contribute to the orchestration of NFV applications to meet the requirements of the IoT and NGNs era

Development and management of collective network and cloud computing infrastructures

Pla de Doctorat industrial de la Generalitat de CatalunyaIn the search and development of more participatory models for infrastructure development and management, in this dissertation, we investigate models for the financing, deployment, and operation of network and cloud computing infrastructures. Our main concern is to overcome the inherent exclusion in participation in the processes of development and management and in the right of use in the current dominant models. Our work starts by studying in detail the model of Guifi.net, a successful bottom-up initiative for building network infrastructure, generally referred to as a community networks. We pay special attention to its governance system and economic organisation because we argue that these are the key components of the success of this initiative. Then, we generalise our findings for any community network, aiming at becoming sustainable and scalable, and we explore the suitability of the Guifi.net model to the cloud computing infrastructure. As a result of both, we coin the attribute extensible to refer to infrastructure that is relatively easy to expand and maintain in contrast to those naturally limited or hard to expand, such as natural resources or highly complex or advanced artificial systems. We conclude proposing a generic model which, in our opinion, is suitable, at least, for managing extensible infrastructure. The Guifi.net model is deeply rooted in the commons; thus, the research in this field, in general, and Elinor Ostrom’s work, in particular, have left a profound imprint in our work. Our results show that the \guifinet model meets almost entirely the principles of long-enduring commons identified by E. Ostrom. This work has been developed as an industrial doctorate. As such, it combines academic research with elements of practice and pursues an effective knowledge transfer between academia and the private sector. Given that the private sector’s partner is a not-for-profit organisation, the effort to create social value has prevailed over the ambition to advance the development of a specific industrial product or particular technology.En la recerca i desenvolupament de models més participatius per al desenvolupament i gestió d'infraestructura, en aquesta tesi investiguem sobre models per al finançament, desplegament i operació d'infraestructures de xarxa i de computació al núvol. La nostra preocupació principal és fer front a l’exclusió inherent dels models dominants actualment pel que fa a la participació en els processos de desenvolupament i gestió i, també, als drets d’us. El nostre treball comença amb un estudi detallat del model de Guifi.net, un cas d'èxit d'iniciativa ciutadana en la construcció d'infraestructura de xarxa, iniciatives que es coneixen com a xarxes comunitàries. En fer-ho, parem una atenció especial al sistema de governança i a l’organització econòmica perquè pensem que són els dos elements claus de l'èxit d'aquesta iniciativa. Tot seguit passem a analitzar d'altres xarxes comunitàries per abundar en la comprensió dels factors determinants per a la seva sostenibilitat i escalabilitat. Després ampliem el nostre estudi analitzant la capacitat i el comportament del model de Guifi.net en el camp de les infraestructures de computació al núvol. A resultes d'aquests estudis, proposem l'atribut extensible per a descriure aquelles infraestructures que són relativament fàcil d'ampliar i gestionar, en contraposició a les que o bé estan limitades de forma natural o be són difícils d'ampliar, com ara els recursos naturals o els sistemes artificials avançats o complexos. Finalitzem aquest treball fent una proposta de model genèric que pensem que és d'aplicabilitat, com a mínim, a tot tipus d'infraestructura extensible. El model de Guifi.net està fortament vinculat als bens comuns. És per això que la recerca en aquest àmbit, en general, i els treballs de Elinor Ostrom en particular, han deixat una forta empremta en el nostre treball. Els resultats que hem obtingut mostren que el model Guifi.net s'ajusta molt bé als principis que segons Ostrom han de complir els béns comuns per ser sostenibles. Aquest treball s'ha desenvolupat com a doctorat industrial. Com a tal, combina la investigació acadèmica amb elements de practica i persegueix una transferència efectiva de coneixement entre l'àmbit acadèmic i el sector privat. Ates que el soci del sector privat és una organització sense ànim de lucre, l’esforç per crear valor social ha prevalgut en l’ambició d’avançar en el desenvolupament d'un producte industrial específic o d'una tecnologia particularPostprint (published version