Data distribution and exploitation in a global microservice artefact observatory

​© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Cloud computing and specifically the microservice architecture pattern is becoming an increasingly prominent paradigm in computer science. Many modern cloud applications are composed of a variety of different microservices, each potentially built in different languages, using different technologies and a different software artefact structure. What is needed is the capability to monitor this rapidly expanding field and leverage the data to enable further research and development of microservice architectures. Drawing inspiration from the global observatories used in geoscience and astronomy, the aim of this research initiative is the establishment of a global observatory for microservice artefacts, allowing the aggregation of data from different hubs and the execution of dynamic analysis on them

A DEVSECOPS APPROACH FOR DEVELOPING AND DEPLOYING CONTAINERIZED CLOUD-BASED SOFTWARE ON SUBMARINES

There are unique challenges for using secure cloud services in disconnected resource-constrained environments and with controlled data. To address those challenges, this thesis introduces a tactical-edge platform-as-a-service (PaaS) solution with a declarative-delivery method for submarine Consolidated Afloat Network Enterprise Services (CANES) operating systems. The PaaS is adapted from the Department of Defense’s Big Bang core elements for submarine-focused outcomes. Using the Team Submarine Project Blue initiative as a case study, this thesis consists of a feasibility study for running containerized applications on different submarine-compatible baselines and applying a prototype declarative software-delivery method called ZARF. We demonstrated the feasibility of using ZARF for packaging and automated deployment of the Project Blue PaaS and its software to the submarine CANES infrastructure. This research culminated in successful integration tests on a current and future submarine hardware and software baseline. The thesis documents the execution of the research, lessons learned, and recommendations for the Navy’s path forward for development of secure software and declarative deployment in air-gapped environments.Outstanding ThesisLieutenant, United States NavyApproved for public release. Distribution is unlimited

Deployment and Operation of Complex Software in Heterogeneous Execution Environments

This open access book provides an overview of the work developed within the SODALITE project, which aims at facilitating the deployment and operation of distributed software on top of heterogeneous infrastructures, including cloud, HPC and edge resources. The experts participating in the project describe how SODALITE works and how it can be exploited by end users. While multiple languages and tools are available in the literature to support DevOps teams in the automation of deployment and operation steps, still these activities require specific know-how and skills that cannot be found in average teams. The SODALITE framework tackles this problem by offering modelling and smart editing features to allow those we call Application Ops Experts to work without knowing low level details about the adopted, potentially heterogeneous, infrastructures. The framework offers also mechanisms to verify the quality of the defined models, generate the corresponding executable infrastructural code, automatically wrap application components within proper execution containers, orchestrate all activities concerned with deployment and operation of all system components, and support on-the-fly self-adaptation and refactoring

Software defined applications: a DevOps approach to monitoring

Dissertação de mestrado integrado em Informatics EngineeringDevOps presents a mix of agile methodologies that allow an application’s release cycle to be shortened. This translates into a faster delivery of value to the stakeholders. However, the value creation chain does not finish at the end of that cycle. It is necessary to monitor the artifacts produced at a system level, and at the application level, in order to ensure the compliance of the functional and non functional requirements. Today, there seems to be a clear separation between the monitoring process and the application development process. As the development and operations processes have merged in DevOps, this dissertation pretends to investigate how to integrate several aspects of monitoring into the regular lifecycle of an application’s development. The inclusion of external services further emphasizes the need to include an observability component into an infrastructure. The main goal of this dissertation is to develop a solution for the deployment of an infrastructure using stateof- the-art technologies and frameworks, while also providing observability to the system and to the applications running on it. To do so, it required the investigation of the methodologies and concepts that are the base of the software development lifecycle, focusing on the latter stages of that process: the deployment, and monitoring phases. These methodologies and concepts were complemented with the study of state-of-the-art technologies and frameworks that aim to ease the burden of setting up an infrastructure quickly and with the necessary tools to evolve it after the initial setup and with each new software release. Furthermore, it also involved the research of tools that enable the collection of metrics from applications, as well as processing such data and displaying it in useful ways for operators and stakeholders. In this context, this dissertation aims to provide a solution for the deployment of MobileID applications at INESC TEC, using the Mobile Driving Licence as the primary case study. The proposed design and implementation with a container orchestration framework and CI/CD pipelines, enables faster development of different MobileID applications, while also providing continuous monitoring to the deployments. With this implementation, it was possible to assess how container orchestration frameworks provide greater flexibility to applications, and how this observability can be augmented with the use of dedicated monitoring systems.DevOps baseia-se na utilização de um conjunto de metolodogias ágeis que permitem encurtar o ciclo de desenvolvimento de uma aplicação de forma a que as alterações efetuadas pelos programadores se traduzam no valor desejado pelas partes interessadas. No entanto, a criação de valor não termina na parte final desse ciclo. É necessário monitorizar os artefactos produzidos tanto a nível de sistema, como a nível aplicacional, de forma a garantir o cumprimento de requisitos funcionais e não funcionais. Todavia, parece existir uma separação entre o processo de monitorização e o processo de desenvolvimento de aplicações. Tal como os processos de desenvolvimento e de operações se uniram no conceito de DevOps, pretende-se também investigar como será possível integrar vários aspetos de monitorização no ciclo normal de desenvolvimento de uma aplicação. O principal objetivo desta dissertação é desenvolver uma solução de operacionalização de infraestruturas de suporte a aplicações com recurso às tecnologias e ferramentas mais adequadas. Esta solução deverá ser acompanhada, em paralelo, por mecanismos de observabilidade dessa infraestrutura e das aplicações que nela são executadas. Para isso, foi necessária a investigação de metodologias e conceitos que formam a base do processo de desenvolvimento de software. O foco esteve nas partes finais do processo: a fase de deployment e a de monitorização. Estas metodolodogias e conceitos foram complementados com o estudo de tecnologias e ferramentas que pretendem facilitar o processo de montar uma infraestrutura rapidamente, bem como permitir a evolução da arquitetura inicial consoante os subsequentes lançamentos de aplicações. Para além disso, também envolveu a pesquisa de ferramentas que permitem extrair e armazenar métricas de aplicações, bem como processar essa informação e disponibilizá-la em formato útil quer para operadores, quer para outras partes interessadas. Neste contexto, esta dissertação pretende desenvolver uma solução que permita efetuar o deployment de aplicações de Identidade Digital no INESC TEC, utilizando a Carta de Condução Móvel como caso de estudo. A arquitetura proposta, e a respetiva implementação com recurso a um orquestrador de containers e pipelines de CI/CD, permite o desenvolvimento mais ágil de novas aplicações de Identidade Digital, e proporciona monitorização contínua a cada iteração do desenvolvimento. A partir do resultado prático obtido, foi possível aferir de que forma os orquestradores de containers permitem melhorar a observabilidade de aplicações, e de que forma ela pode ser aumentada com recurso a sistemas dedicados de monitorização contínua

Short Papers of the 8th Conference on Cloud Computing Conference, Big Data & Emerging Topics (JCC-BD&ET 2020)

Compilación de los short papers presentados en las 8vas Jornadas de Cloud Computing, Big Data & Emerging Topics (JCC-BD&ET2020), llevadas a cabo en modalidad virtual durante septiembre de 2020 y organizadas por el Instituto de Investigación en Informática LIDI (III-LIDI) y la Secretaría de Posgrado de la Facultad de Informática de la UNLP en colaboración con universidades de Argentina y del exterior.Facultad de Informátic

Monitoring data distribution and exploitation in a global-scale microservice artefact observatory

Erstveröffentlichung basierend auf EuroDW-2020-Präsentation.Reusable microservice artefacts are often deployed as black or grey boxes, with little concern for their properties and quality, beyond a syntactical interface description. This leads application developers to chaotic and opportunistic assumptions about how a composite application will behave in the real world. Systematically analyzing and tracking these publicly available artefacts will grant much needed predictability to microservice-based deployments. By establishing a distributed observatory and knowledge base, it is possible to track microservice repositories and analyze the artefacts reliably, and provide insights on their properties and quality to developers and researchers alike. This position paper argues for a federated research infrastructure with consensus voting among participants to establish and preserve ground truth about the insights

Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices

Cloud-native applications constitute a recent trend for designing large-scale software systems. This thesis introduces the Theodolite benchmarking method, allowing researchers and practitioners to conduct empirical scalability evaluations of cloud-native applications, their frameworks, configurations, and deployments. The benchmarking method is applied to event-driven microservices, a specific type of cloud-native applications that employ distributed stream processing frameworks to scale with massive data volumes. Extensive experimental evaluations benchmark and compare the scalability of various stream processing frameworks under different configurations and deployments, including different public and private cloud environments. These experiments show that the presented benchmarking method provides statistically sound results in an adequate amount of time. In addition, three case studies demonstrate that the Theodolite benchmarking method can be applied to a wide range of applications beyond stream processing