Explorar kubernetes e devOps num contexto de IoT

Containerized solutions and container orchestration technologies have recently been of great interest to organizations as a way of accelerating both software development and delivery processes. However, adopting these is a rather complex shift that may impact an organization and teams that were already established. This is where development cultures such as DevOps emerge to ease such shift amongst teams, promoting collaboration and automation of development and deployment processes throughout. The purpose of the current dissertation is to illustrate the path that led to the use of DevOps and containerization as means to support the development and deployment of a proof of concept system, Firefighter Sync – an Internet of Things based solution applied to a firefighting monitoring scenario. The goal, besides implementing Firefighter Sync, was to propose and deploy a development and operations ecosystem based on DevOps practices to achieve a full automation pipeline for both the development and operations processes. Firefighter Sync enabled the exploration of such state-of-the-art solutions such as Kubernetes to support container-based deployment and Jenkins for a fully automated CI/CD pipeline. Firefighter Sync clearly illustrates that addressing the development of a system from a DevOps perspective from the very beginning, although it requires an accentuated learning curve due to the large range of concepts and technologies addressed throughout, has illustrated to effectively impact the development process as well as ease the solution for future evolution. A good example is the automation process pipeline, that whilst allowing an easy integration of new features within a DevOps process – implies addressing the development and operations as a whole – it abstracts specific technological concerns turning these transversals to the traditional stages from development to deployment.Soluções de contentores e orquestração de contentores têm vindo a tornar-se de grande interesse para as organizações como uma forma de acelerar os processos de desenvolvimento e entrega de software. No entanto, adotá-las é uma mudança bastante complexa que pode impactar uma organização e equipas já estabelecidas. É aqui que surgem culturas como o DevOps para facilitar essa mudança, promovendo a colaboração e a automação dos processos de desenvolvimento e deployment entre equipas. O objetivo desta dissertação é ilustrar o caminho que levou ao uso de DevOps e à conteinerização de modo a apoiar o desenvolvimento e o deployment de um sistema como prova de conceito, o Firefighter Sync – uma solução baseada na Internet das Coisas aplicada a um cenário de monitorização de combate a incêndios. Além de implementar o Firefighter Sync, o objetivo era também propor e implementar um ecossistema de desenvolvimento e operações com base nas práticas de DevOps para alcançar uma pipeline de automação completa para os processos de desenvolvimento e operações. O Firefighter Sync permitiu explorar soluções que constituem o estado da arte neste contexto, como o Kubernetes para apoiar o deployment baseado em contentores e o Jenkins para suportar a pipeline de CI/CD totalmente automatizada. O Firefighter Sync ilustra claramente que abordar o desenvolvimento de um sistema a partir da perspectiva de DevOps, embora exija uma curva de aprendizagem acentuada devido à grande variedade de conceitos e tecnologias inerentes ao longo do processo, demonstrou tornar mais eficiente o processo de desenvolvimento, bem como facilitar evolução futura. Um exemplo é a pipeline de automação, que permite uma fácil integração de novos recursos dentro de um processo de DevOps – que implica abordar o desenvolvimento e as operações como um todo – abstraindo assim preocupações tecnológicas específicas, transformando essas transversais nas fases tradicionais do desenvolvimento ao deployment.Mestrado em Engenharia Informátic

BlogForever: D3.1 Preservation Strategy Report

This report describes preservation planning approaches and strategies recommended by the BlogForever project as a core component of a weblog repository design. More specifically, we start by discussing why we would want to preserve weblogs in the first place and what it is exactly that we are trying to preserve. We further present a review of past and present work and highlight why current practices in web archiving do not address the needs of weblog preservation adequately. We make three distinctive contributions in this volume: a) we propose transferable practical workflows for applying a combination of established metadata and repository standards in developing a weblog repository, b) we provide an automated approach to identifying significant properties of weblog content that uses the notion of communities and how this affects previous strategies, c) we propose a sustainability plan that draws upon community knowledge through innovative repository design

Microservice-based Reference Architecture for Semantics-aware Measurement Systems

Cloud technologies have become more important than ever with the rising need for scalable and distributed software systems. A pattern that is used in many such systems is a microservice-based architecture (MSA). MSAs have become a blueprint for many large companies and big software systems. In many scientific fields like energy and environmental informatics, efficient and scalable software systems with a primary focus on measurement data are a core requirement. Nowadays, there are many ways to solve research questions using data-driven approaches. Most of them have a need for large amounts of measurement data and according metadata. However, many measurement systems still follow deprecated guidelines such as monolithic architectures, classic relational database principles and are missing semantic awareness and interpretation of data. These problems and the resulting requirements are tackled by the introduction of a reference architecture with a focus on measurement systems that utilizes the principles of microservices. The thesis first presents the systematic design of the reference architecture by using the principles of Domain-driven Design (DDD). This process ensures that the reference architecture is defined in a modular and sustainable way in contrast to complex monolithic software systems. An extensive scientific analysis leads to the core parts of the concept consisting of the data management and semantics for measurement systems. Different data services define a concept for managing measurement data, according meta data and master data describing the business objects of the application implemented by using the reference architecture. Further concepts allow the reference architecture to define a way for the system to understand and interpret the data using semantic information. Lastly, the introduction of a frontend framework for dashboard applications represents an example for visualizing the data managed by the microservices

A cloud, microservice-based digital twin for the Oil industry a flow assurance case study

Digital Twins are one of the top recent technology trends, considered a front-runner in enabling the next generation of intelligent, digitalized industries. A Digital Twin is basi cally a real enterprise asset mirrored into the virtual world, created entirely in software, and utilized to innovate business, generate new revenue, and create value-producing op portunities. Naturally, building such demanding systems is not an easy task. Many organi zational and technological concerns should be addressed, like real-time processing, data management, cybersecurity, and low latency communication. Moreover, Digital Twins should be extensible and support data integration with its physical counterpart, allowing enterprises to streamline their operations more safely. Considering there are no standard methodologies or technologies to design a Digital Twin, software developers struggle to develop a robust, reliable Digital Twin architecture that meets customers’ needs. Therefore, in consideration of these technical challenges, we decided to explore solutions in the oil and gas industry. Oil plants are known for having notoriously complex pro cesses and an oppressive ecosystem when it comes to the adoption of new technologies, especially due to its intricate and ever-changing landscape. Despite these difficulties, this industry would certainly benefit from a well-designed Digital Twin. Hence, this work proposes a solution for an Oil Well Digital Twin architecture. We decided to use the flow assurance process of an Oil Well as a base to build a Digital Twin, aiming to study and fulfill those requirements and technical limitations. We implemented and tested several APIs, each one representing the virtual version of a real Oil Well component, enforcing the importance of microservices and cloud-native patterns in its design. When function ing together, these microservices comprise an entire Oil Well Digital twin solution, cov ering basic storage, communication, and monitoring fundamentals. This proof of concept demonstrates how a complex asset — like an Oil Well — can be built and organized in a completely digital manner. Lastly, this design is extensible and opens up further develop ment opportunities.Gêmeos Digitais são uma das principais tendências tecnológicas recentes, considerados pioneiros na habilitação da próxima geração de indústrias inteligentes e digitalizadas. Um Gêmeo Digital é basicamente um ativo corporativo espelhado no mundo virtual, criado inteiramente em software e destinado a inovar os modelos operacionais e de negócios, ao mesmo tempo em que fornece novas oportunidades de geração de receita e valor. Naturalmente, construir sistemas tão exigentes não é uma tarefa fácil. Muitas preocupações organizacionais e tecnológicas devem ser abordadas, como processamento em tempo real, gerenciamento de dados, segurança cibernética e comunicação de baixa latência. Além disso, os Gêmeos Digitais também devem ser extensíveis e oferecer suporte à integração de dados com sua contraparte física, permitindo que as empresas agilizem suas opera ções com mais segurança. Considerando que não existem metodologias ou tecnologias padrão para projetar um Gêmeo Digital, os desenvolvedores de software tem dificuldade em desenvolver uma arquitetura de Gêmeo Digital robusta e confiável que atenda às ne cessidades dos clientes. Além desses desafios técnicos, a indústria de óleo e gás é conhecida por ter processos notoriamente complexos e um ecossistema opressor na adoção de novas tecnologias, prin cipalmente devido ao seu cenário intrincado e em constante mudança. Apesar dessas dificuldades, essa indústria certamente se beneficiaria de um Gêmeo Digital bem proje tado. Assim, este trabalho propõe uma solução de uma arquitetura para um Gêmeo Digital de um Poço de Petróleo. Decidimos usar o processo de garantia de vazão de um poço de petróleo como base para construir um Gêmeo Digital, visando estudar e atender a esses requisitos e limitações técnicas. Implementamos e testamos várias APIs, cada uma repre sentando a versão virtual de um componente real de um poço de petróleo, reforçando a importância de microsserviços e padrões nativos de nuvem em seu design. Ao funciona rem juntos, esses microsserviços compreendem uma solução completa de gêmeos digitais de poços de petróleo, abrangendo os fundamentos básicos de armazenamento, comunica ção e monitoramento. Esta prova de conceito demonstra como um ativo complexo – como um poço de petróleo – pode ser construído e organizado de forma totalmente digital. Por fim, esse design é extensível e abre mais oportunidades de desenvolvimento

Semantic Systems. In the Era of Knowledge Graphs

This open access book constitutes the refereed proceedings of the 16th International Conference on Semantic Systems, SEMANTiCS 2020, held in Amsterdam, The Netherlands, in September 2020. The conference was held virtually due to the COVID-19 pandemic

Automated deployment of machine learning applications to the cloud

The use of machine learning (ML) as a key technology in artificial intelligence (AI) is becoming more and more important in the increasing digitalization of business processes. However, the majority of the development effort of ML applications is not related to the programming of the ML model, but to the creation of the server structure, which is responsible for a highly available and error-free productive operation of the ML application. The creation of such a server structure by the developers is time-consuming and complicated, because extensive configurations have to be made. Besides the creation of the server structure, it is also useful not to put new ML application versions directly into production, but to observe the behavior of the ML application with respect to unknown data for quality assurance. For example, the error rate as well as the CPU and RAM consumption should be checked. The goal of this thesis is to collect requirements for a suitable server structure and an automation mechanism that generates this server structure, deploys the ML application and allows to observe the behavior of a new ML application version based on real-time user data. For this purpose, a systematic literature review is conducted to investigate how the behavior of ML applications can be analyzed under the influence of real-time user data before their productive operation. Subsequently, in the context of the requirements analysis, a target-performance analysis is carried out in the department of a management consulting company in the automotive sector. Together with the results of the literature research, a list of user stories for the automation tool is determined and prioritized. The automation tool is implemented in the form of a Python console application that enables the desired functionality by using IaC (Infrastructure as code) and the AWS (Amazon Web Services) SDK in the cloud. The automation tool is finally evaluated in the department. The ten participants independently carry out predefined usage scenarios and then evaluate the tool using a questionnaire developed on the basis of the TAM model. The results of the evaluation are predominantly positive and the constructive feedback of the participants includes numerous interesting comments on possible adaptions and extensions of the automation tool