Identification of microservices from monolithic applications through topic modelling

Dissertação de mestrado em Informatics EngineeringMicroservices emerged as one of the most popular architectural patterns in the recent years given the increased need to scale, grow and flexibilize software projects accompanied by the growth in cloud computing and DevOps. Many software applications are being submitted to a process of migration from its monolithic architecture to a more modular, scalable and flexible architecture of microservices. This process is slow and, depending on the project’s complexity, it may take months or even years to complete. This dissertation proposes a new approach on microservices identification by resorting to topic modelling in order to identify services according to domain terms. This approach in combination with clustering techniques produces a set of services based on the original software. The proposed methodology is implemented as an open-source tool for exploration of monolithic architectures and identification of microservices. An extensive quantitative analysis using the state of the art metrics on independence of functionality and modularity of services was conducted on 200 open-source projects collected from GitHub. Cohesion at message and domain level metrics showed medians of roughly 0.6. Interfaces per service exhibited a median of 1.5 with a compact interquartile range. Structural and conceptual modularity revealed medians of 0.2 and 0.4 respectively. Further analysis to understand if the methodology works better for smaller/larger projects revealed an overall stability and similar performance across metrics. Our first results are positive demonstrating beneficial identification of services due to overall metrics’ results.Os microserviços emergiram como um dos padrões arquiteturais mais populares na atualidade dado o aumento da necessidade em escalar, crescer e flexibilizar projetos de software, acompanhados da crescente da computação na cloud e DevOps. Muitas aplicações estão a ser submetidas a processos de migração de uma arquitetura monolítica para uma arquitetura mais modular, escalável e flexivel de microserviços. Este processo de migração é lento, e dependendo da complexidade do projeto, poderá levar vários meses ou mesmo anos a completar. Esta dissertação propõe uma nova abordagem na identificação de microserviços recorrendo a modelação de tópicos de forma a identificar serviços de acordo com termos de domínio de um projeto de software. Esta abordagem em combinação com técnicas de clustering produz um conjunto de serviços baseado no projeto de software original. A metodologia proposta é implementada como uma ferramenta open-source para exploração de arquiteturas monolíticas e identificação de microserviços. Uma análise quantitativa extensa recorrendo a métricas de independência de funcionalidade e modularidade de serviços foi conduzida em 200 aplicações open-source recolhidas do GitHub. Métricas de coesão ao nível da mensagem e domínio revelaram medianas em torno de 0.6. Interfaces por serviço demonstraram uma mediana de 1.5 com um intervalo interquartil compacto. Métricas de modularidade estrutural e conceptual revelaram medianas de 0.2 e 0.4 respetivamente. Uma análise mais aprofundada para tentar perceber se a metodologia funciona melhor para projetos de diferentes dimensões/características revelaram uma estabilidade geral do funcionamento do método. Os primeiros resultados são positivos demonstrando identificações de serviços benéficos tendo em conta que os valores das métricas são de uma forma global positivos e promissores

Understanding Legacy Workflows through Runtime Trace Analysis

abstract: When scientific software is written to specify processes, it takes the form of a workflow, and is often written in an ad-hoc manner in a dynamic programming language. There is a proliferation of legacy workflows implemented by non-expert programmers due to the accessibility of dynamic languages. Unfortunately, ad-hoc workflows lack a structured description as provided by specialized management systems, making ad-hoc workflow maintenance and reuse difficult, and motivating the need for analysis methods. The analysis of ad-hoc workflows using compiler techniques does not address dynamic languages - a program has so few constrains that its behavior cannot be predicted. In contrast, workflow provenance tracking has had success using run-time techniques to record data. The aim of this work is to develop a new analysis method for extracting workflow structure at run-time, thus avoiding issues with dynamics. The method captures the dataflow of an ad-hoc workflow through its execution and abstracts it with a process for simplifying repetition. An instrumentation system first processes the workflow to produce an instrumented version, capable of logging events, which is then executed on an input to produce a trace. The trace undergoes dataflow construction to produce a provenance graph. The dataflow is examined for equivalent regions, which are collected into a single unit. The workflow is thus characterized in terms of its treatment of an input. Unlike other methods, a run-time approach characterizes the workflow's actual behavior; including elements which static analysis cannot predict (for example, code dynamically evaluated based on input parameters). This also enables the characterization of dataflow through external tools. The contributions of this work are: a run-time method for recording a provenance graph from an ad-hoc Python workflow, and a method to analyze the structure of a workflow from provenance. Methods are implemented in Python and are demonstrated on real world Python workflows. These contributions enable users to derive graph structure from workflows. Empowered by a graphical view, users can better understand a legacy workflow. This makes the wealth of legacy ad-hoc workflows accessible, enabling workflow reuse instead of investing time and resources into creating a workflow.Dissertation/ThesisMasters Thesis Computer Science 201

Workflow Provenance: from Modeling to Reporting

Workflow provenance is a crucial part of a workflow system as it enables data lineage analysis, error tracking, workflow monitoring, usage pattern discovery, and so on. Integrating provenance into a workflow system or modifying a workflow system to capture or analyze different provenance information is burdensome, requiring extensive development because provenance mechanisms rely heavily on the modelling, architecture, and design of the workflow system. Various tools and technologies exist for logging events in a software system. Unfortunately, logging tools and technologies are not designed for capturing and analyzing provenance information. Workflow provenance is not only about logging, but also about retrieving workflow related information from logs. In this work, we propose a taxonomy of provenance questions and guided by these questions, we created a workflow programming model 'ProvMod' with a supporting run-time library to provide automated provenance and log analysis for any workflow system. The design and provenance mechanism of ProvMod is based on recommendations from prominent research and is easy to integrate into any workflow system. ProvMod offers Neo4j graph database support to manage semi-structured heterogeneous JSON logs. The log structure is adaptable to any NoSQL technology. For each provenance question in our taxonomy, ProvMod provides the answer with data visualization using Neo4j and the ELK Stack. Besides analyzing performance from various angles, we demonstrate the ease of integration by integrating ProvMod with Apache Taverna and evaluate ProvMod usability by engaging users. Finally, we present two Software Engineering research cases (clone detection and architecture extraction) where our proposed model ProvMod and provenance questions taxonomy can be applied to discover meaningful insights

Migration from Legacy to Reactive Applications in OutSystems

A legacy system is an information system that significantly resists evolution. Through a migration, these systems can be moved to a more modernized environment without having to be redeveloped. OutSystems is a software company with a platform to develop and maintain applications using abstraction to increase productivity. In October 2019, OutSystems launched a new paradigm to allow developers to build reactive web applications. Because of this, the applications implemented in the old web paradigm turned into legacy systems. The OutSystems’ approach to this problem was a manual migration. However, it discards a considerable part of the effort previously made on the legacy system. A well-founded case study took place and allowed us to classify the UI as the most prioritized feature, but coincidently, the major bottleneck in migrations. So, this project had the following objectives: (1) The design and implementation of an automatic migration approach capable of converting UI elements to accelerate the manual migration; (2) The integration of the developed tool in the OutSystems platform. To transform the OutSystems paradigm’s elements, model-driven transformation rules must be set to receive the source UI elements and produce the target equivalent implementation in the new paradigm (each according to their model). However, the trans formations may not be straightforward, and a set of elements may need to be migrated to a different implementation due to Reactive Web’s best practices. Via the creation and search of UI patterns, it is possible to make special transformations for such scenarios. As a result, a migration approach was developed, allowing for the migration of UI (and other) elements. To complement this objective, the developed tool was integrated into the OutSystems platform with an easy to use interaction. Performance and usability tests proved the necessity and impact the final result had on the migration problem. This dissertation’s objectives were fully met and even exceeded, accelerating the man ual migration by providing an automatic UI conversion. This provided a quality increase in the existing process and results, giving OutSystems and its users the possibility of evolving their applications with considerable less effort and investment.Um sistema legado é um sistema de informação que resiste à evolução. Através de uma migração, estes sistemas podem ser movidos para um ambiente modernizado sem necessitar de re-implementação. A OutSystems é uma empresa de software com uma plataforma para desenvolver e manter aplicações usando abstracção para aumentar a produtividade. Em Outubro de 2019, a OutSystems lançou um novo paradigma para desenvolver aplicações reactive web. Assim, as aplicações implementadas no antigo paradigma web tornaram-se sistemas legados. A abordagem da OutSystems ao problema foi uma migração manual, no entanto, esta abordagem desconsidera uma parte significativa do investimento feito no sistema legado. Uma análise permitiu classificar a UI como a característica mais priorizada, mas também como o maior obstáculo em migrações. Assim, este projecto tem como objectivos: (1) O desenho e implementação de uma migração automática capaz de converter os elementos de UI para acelerar a migração manual; (2) A integração da ferramenta desenvolvida na plataforma da OutSystems. Para transformar os elementos dos paradigmas OutSystems, transformações de modelos têm de ser definidas para receber os elementos UI e produzir a implementação equivalente no novo paradigma (de acordo com o seu modelo). No entanto, as transformações podem não ser lineares, e um conjunto de elementos pode necessitar de uma migração para uma implementação diferente devido ao Reactive Web. Com a definição e procura de padrões de UI, é possível fazer transformações especiais para esses cenários. Como resultado, a migração foi desenvolvida, permitindo a conversão de elementos de UI (e não só). Para complementar, a ferramenta desenvolvida foi integrada na plataforma da OutSystems com uma interacção de fácil uso. Testes de desempenho e usabilidade provaram a necessidade e impacto da ferramenta no contexto da migração manual. Os objectivos desta dissertação foram completados na totalidade, acelerando a migração manual com a automação da migração de UI. Isto traz um aumento da qualidade no processo existente e nos seus resultados, dando à OutSystems e aos seus utilizadores a possibilidade de evoluírem as suas aplicações com um esforço e investimento menores

Reinforcing Digital Trust for Cloud Manufacturing Through Data Provenance Using Ethereum Smart Contracts

Cloud Manufacturing(CMfg) is an advanced manufacturing model that caters to fast-paced agile requirements (Putnik, 2012). For manufacturing complex products that require extensive resources, manufacturers explore advanced manufacturing techniques like CMfg as it becomes infeasible to achieve high standards through complete ownership of manufacturing artifacts (Kuan et al., 2011). CMfg, with other names such as Manufacturing as a Service (MaaS) and Cyber Manufacturing (NSF, 2020), addresses the shortcoming of traditional manufacturing by building a virtual cyber enterprise of geographically distributed entities that manufacture custom products through collaboration. With manufacturing venturing into cyberspace, Digital Trust issues concerning product quality, data, and intellectual property security, become significant concerns (R. Li et al., 2019). This study establishes a trust mechanism through data provenance for ensuring digital trust between various stakeholders involved in CMfg. A trust model with smart contracts built on the Ethereum blockchain implements data provenance in CMfg. The study covers three data provenance models using Ethereum smart contracts for establishing digital trust in CMfg. These are Product Provenance, Order Provenance, and Operational Provenance. The models of provenance together address the most important questions regarding CMfg: What goes into the product, who manufactures the product, who transports the products, under what conditions the products are manufactured, and whether regulatory constraints/requisites are met

Technical Challenges of Microservices Migration

The microservices architecture is a recent trend in the software engineering community, with the number of research articles in the field increasing, and more companies adopting the architectural style every year. However, the migration of a monolith to the microservices architecture is an error-prone process with a lack of guidelines for its execution. Also, microservices introduce a lot of different challenges that are not faced when following a monolithic architecture. This work aims to fill some gaps in current microservices research by providing a catalogue of the currently most common challenges of adopting this architectural style, and possible solutions for them. For this reason, a systematic mapping study was executed analysing 54 different articles. Also, 30 industry professionals participated in a questionnaire regarding the topic. Furthermore, a participant observation experiment was performed to retrieve additional industry data. Moreover, one of the identified challenges – distributed transactions management – was further detailed and a solution implemented using the choreographed saga pattern. The solution is publicly available as an open-source project. Finally, multiple experts in the microservices field validated the results of the research and the distributed transactions solution and provided insights regarding the value of this work.A arquitetura de microserviços é uma tendência recente na comunidade de engenharia de software, com o número de artigos publicados sobre o tema a aumentar, assim como o número de empresas a adoptar o estilo arquitetural todos os anos. No entanto, o processo de migração de um monolito para uma arquitetura orientada a microserviços tem um alto potencial de erros, uma vez que existe falta de orientações sobre como conduzir o processo corretamente. Para além disso, os microserviços introduzem muitos desafios diferentes que não são enfrentados no desenvolvimento de um sistema monolitico. Este trabalho pretende preencher algumas destas lacunas na investigação da arquitetura de microserviços através da construção de um catalogo dos principais desafios enfrentados ao adoptar o estilo arquitetural e soluções possíveis para estes. Por este motivo, um systematic mapping study foi desenvolvido, analisando 54 artigos diferentes. Para além disso, 30 profissionais da industria responderam a questionario sobre o tema. Finalmente, para obter dados adicionais da indústria, uma experiência de migração foi realizada e observada de forma ativa. Ainda, um dos desafios identificados – gestão de transações distribuídas – foi detalhado e uma solução implementada usando o padrão de sagas coreografadas. A solução está publicamente disponível como um projecto open-source. Finalmente, vários peritos em microserviços avaliaram os resultados deste trabalho, incluindo a solução desenvolvida para gestão de transações distribuídas, e deram feedback relativamente ao valor deste trabalho