Implementation of DevOps pipeline for Serverless Applications

Serverless computing is a cloud computing execution model where server-side logic runs in the stateless compute containers that are event-triggered and usually fully managed by vendor hosts such as AWS Lambda. This approach is also called Function as a Service (FaaS). Applications that rely on this approach are called Serverless applications. Serverless usage promises infrastructure cost reduction and automatic scalability. One more important benefit of serverless is making the operations part of DevOps process simpler. It reduces the time on the management and maintenance of the servers and sometimes makes them even completely unnecessary. Despite this fact, applications using serverless computing model require a new look at DevOps automation practices since it is a new approach to software architecture design and software development workflow. The goal of this thesis is to implement DevOps pipeline for a Serverless application within a single case organization and evaluate the results of implementation. This is done through design science research, where result artifact is a release pipeline designed and implemented according to the requirements for a new project in the case organization. The result of the study is automated DevOps pipeline with implemented Continuous Integration (CI), Continuous Delivery (CD) and Monitoring practices required for the case project. The research shows that architecture of Serverless applications affects many DevOps automation practices such as test execution, deployment and monitoring of the application. It also affects the decisions about source code repositories structure, mocking libraries and Infrastructure as Code (IaC) tools

Security in DevOps: understanding the most efficient way to integrate security in the agile software development process

Modern development methodologies follow a fast and dynamic pace, which gives great attention to customers’ satisfaction in the delivery of new releases. On the other hand, the work pursued to secure a system, if not adapted to the new development trend, can risk to slow down the delivery of new software and the adaptability typical for an Agile environment. Therefore, it is paramount to think about a new way to integrate security into the development framework, in order to secure the software in the best way without slowing down the pace of the developers. Moreover, the implementation of automatic and repeatable security controls inside the development pipeline can help to catch the presence of vulnerabilities as early as possible, thus reducing costs, comparing to solving the issues at later stages. The thesis presents a series of recommendations on how to best deploy a so called DevSecOps approach and applies the theory to the use case of Awake.AI, a Finnish startup company focusing its business on the maritime industry. It is not always easy and feasible to practically apply all the suggestions presented in the literature to a real case scenario, but rather the recommendations need to be adapted and forged in a way that best suits the situation and the current target. It is undeniable that the presence of a strong and efficient secure development framework can give substantial advantage to the success of a company. In fact, not only it makes sure that the delivery of good quality code to the customers is not slowed down, but it also dramatically reduces the risk of incurring in expensive security incidents. Lastly, it is valuable to also mention that, being able to show a clean and efficient approach to security, the framework improves the reputation and trustfulness of the company under the eyes of the customers

RADON: Rational decomposition and orchestration for serverless computing

Emerging serverless computing technologies, such as function as a service (FaaS), enable developers to virtualize the internal logic of an application, simplifying the management of cloud-native services and allowing cost savings through billing and scaling at the level of individual functions. Serverless computing is therefore rapidly shifting the attention of software vendors to the challenge of developing cloud applications deployable on FaaS platforms. In this vision paper, we present the research agenda of the RADON project (http://radon-h2020.eu), which aims to develop a model-driven DevOps framework for creating and managing applications based on serverless computing. RADON applications will consist of fine-grained and independent microservices that can efficiently and optimally exploit FaaS and container technologies. Our methodology strives to tackle complexity in designing such applications, including the solution of optimal decomposition, the reuse of serverless functions as well as the abstraction and actuation of event processing chains, while avoiding cloud vendor lock-in through models

Developing Self-Adaptive Microservice Systems: Challenges and Directions

A self-adaptive system can dynamically monitor and adapt its behavior to preserve or enhance its quality attributes under uncertain operating conditions. This article identifies key challenges for the development of microservice applications as self-adaptive systems, using a cloud-based intelligent video surveillance application as a motivating example. It also suggests potential new directions for addressing most of the identified challenges by leveraging existing microservice practices and technologies.Comment: 8 pages, 1 figur

Cloud provider independence using DevOps methodologies with Infrastructure-as-Code

On choosing cloud computing infrastructure for IT needs there is a risk of becoming dependent and locked-in on a specific cloud provider from which it becomes difficult to switch should an entity decide to move all of the infrastructure resources into a different provider. There’s widespread information available on how to migrate existing infrastructure to the cloud notwithstanding common cloud solutions and providers don't have any clear path or framework for supporting their tenants to migrate off the cloud into another provider or cloud infrastructure with similar service levels should they decide to do so. Under these circumstances it becomes difficult to switch from cloud provider not just because of the technical complexity of recreating the entire infrastructure from scratch and moving related data but also because of the cost it may involve. One possible solution is to evaluate the use of Infrastructure-as-Code languages for defining infrastructure (“Infrastructure-as-Code”) combined with DevOps methodologies and technologies to create a mechanism that helps streamline the migration process between different cloud infrastructure especially if taken into account from the beginning of a project. A well-structured DevOps methodology combined with Infrastructure-as-Code may allow a more integrated control on cloud resources as those can be defined and controlled with specific languages and be submitted to automation processes. Such definitions must take into account what is currently available to support those operations under the chosen cloud infrastructure APIs, always seeking to guarantee the tenant an higher degree of control over its infrastructure and higher level of preparation of the necessary steps for the recreation or migration of such infrastructure should the need arise, somehow integrating cloud resources as part of a development model. The objective of this dissertation is to create a conceptual reference framework that can identify different forms for migration of IT infrastructure while always contemplating a higher provider independence by resorting to such mechanisms, as well as identify possible constraints or obstacles under this approach. Such a framework can be referenced from the beginning of a development project if foreseeable changes in infrastructure or provider are a possibility in the future, taking into account what the API’s provide in order to make such transitions easier.Ao optar-se por infraestruturas de computação em nuvem para soluções de TI existe um risco associado de se ficar dependente de um fornecedor de serviço específico, do qual se torna difícil mudar caso se decida posteriormente movimentar toda essa infraestrutura para um outro fornecedor. Encontra-se disponível extensa documentação sobre como migrar infraestrutura já  existente para modelos de computação em nuvem, de qualquer modo as soluções e os fornecedores de serviço não dispõem de formas ou metodologias claras que suportem os seus clientes em migrações para fora da nuvem, seja para outro fornecedor ou infraestrutura com semelhantes tipos de serviço, caso assim o desejem. Nestas circunstâncias torna-se difícil mudar de fornecedor de serviço não apenas pela complexidade técnica associada à criação de toda a infraestrutura de raiz e movimentação de todos os dados associados a esta mas também devido aos custos que envolve uma operação deste tipo. Uma possível solução é avaliar a utilização de linguagens para definição de infraestrutura como código (“Infrastructure-as-Code”) em conjunção com metodologias e tecnologias “DevOps” de forma a criar um mecanismo que permita flexibilizar um processo de migração entre diferentes infraestruturas de computação em nuvem, especialmente se for contemplado desde o início de um projecto. Uma metodologia “DevOps” devidamente estruturada quando combinada com definição de infraestrutura como código pode permitir um controlo mais integrado de recursos na nuvem uma vez que estes podem ser definidos e controlados através de linguagens específicas e submetidos a processos de automação. Tais definições terão de ter em consideração o que existe disponível para suportar as necessárias operações através das “API’s” das infraestruturas de computação em nuvem, procurando sempre garantir ao utilizador um elevado grau de controlo sobre a sua infraestrutura e um maior nível de preparação dos passos necessários para recriação ou migração da infraestrutura caso essa necessidade surja, integrando de certa forma os recursos de computação em nuvem como parte do modelo de desenvolvimento. Esta dissertação tem como objetivo a criação de um modelo de referência conceptual que identifique formas de migração de infraestruturas de computação procurando ao mesmo tempo uma maior independência do fornecedor de serviço com recurso a tais mecanismos, assim como identificar possíveis constrangimentos ou impedimentos nesta aproximação. Tal modelo poderá ser referenciado desde o início de um projecto de desenvolvimento caso seja necessário contemplar uma possível necessidade futura de alterações ao nível da infraestrutura ou de fornecedor, com base no que as “API’s” disponibilizam, de modo a facilitar essa operação.info:eu-repo/semantics/publishedVersio

Evaluating Performance of Serverless Virtualization

Abstract. The serverless computing has posed new challenges for cloud vendors that are difficult to solve with existing virtualization technologies. Maintaining security, resource isolation, backwards compatibility and scalability is extremely difficult when the platform should be able to deliver native performance. This paper contains a literature review of recently published results related to the performance of virtualization technologies such as KVM and Docker, and further reports a DESMET benchmarking evaluation against KVM and Docker, as well as Firecracker and gVisor, which are being used by Amazon Web Services and Google Cloud in their cloud services. The context for this research is coming from education, where students return their programming assignments into a source code repository system that further triggers automated tests and potentially other tasks against the submitted code. The used environment consists of several software components, such as web server, database and job executor, and thus represents a common architecture in web-based applications. The results of the research show that Docker is still the most performant virtualization technology amongst the selected ones. Additionally, Firecracker and gVisor perform better in some areas than KVM and thus are viable options for single-tenant environments. Lastly, applications that run untrusted code or have otherwise really high security requirements could potentially leverage from using either Firecracker or gVisor

LEAN DATA ENGINEERING. COMBINING STATE OF THE ART PRINCIPLES TO PROCESS DATA EFFICIENTLYS

The present work was developed during an internship, under Erasmus+ Traineeship program, in Fieldwork Robotics, a Cambridge based company that develops robots to operate in agricultural fields. They collect data from commercial greenhouses with sensors and real sense cameras, as well as with gripper cameras placed in the robotic arms. This data is recorded mainly in bag files, consisting of unstructured data, such as images and semi-structured data, such as metadata associated with both the conditions where the images were taken and information about the robot itself. Data was uploaded, extracted, cleaned and labelled manually before being used to train Artificial Intelligence (AI) algorithms to identify raspberries during the harvesting process. The amount of available data quickly escalates with every trip to the fields, which creates an ever-growing need for an automated process. This problem was addressed via the creation of a data engineering platform encom- passing a data lake, data warehouse and its needed processing capabilities. This platform was created following a series of principles entitled Lean Data Engineering Principles (LDEP), and the systems that follows them are called Lean Data Engineering Systems (LDES). These principles urge to start with the end in mind: process incoming batch or real-time data with no resource wasting, limiting the costs to the absolutely necessary for the job completion, in other words to be as lean as possible. The LDEP principles are a combination of state-of-the-art ideas stemming from several fields, such as data engineering, software engineering and DevOps, leveraging cloud technologies at its core. The proposed custom-made solution enabled the company to scale its data operations, being able to label images almost ten times faster while reducing over 99.9% of its associated costs in comparison to the previous process. In addition, the data lifecycle time has been reduced from weeks to hours while maintaining coherent data quality results, being able, for instance, to correctly identify 94% of the labels in comparison to a human counterpart.Este trabalho foi desenvolvido durante um estágio no âmbito do programa Erasmus+ Traineeship, na Fieldwork Robotics, uma empresa sediada em Cambridge que desenvolve robôs agrícolas. Estes robôs recolhem dados no terreno com sensores e câmeras real- sense, localizados na estrutura de alumínio e nos pulsos dos braços robóticos. Os dados recolhidos são ficheiros contendo dados não estruturados, tais como imagens, e dados semi- -estruturados, associados às condições em que as imagens foram recolhidas. Originalmente, o processo de tratamento dos dados recolhidos (upload, extração, limpeza e etiquetagem) era feito de forma manual, sendo depois utilizados para treinar algoritmos de Inteligência Artificial (IA) para identificar framboesas durante o processo de colheita. Como a quantidade de dados aumentava substancialmente com cada ida ao terreno, verificou-se uma necessidade crescente de um processo automatizado. Este problema foi endereçado com a criação de uma plataforma de engenharia de dados, composta por um data lake, uma data warehouse e o respetivo processamento, para movimentar os dados nas diferentes etapas do processo. Esta plataforma foi criada seguindo uma série de princípios intitulados Lean Data Engineering Principles (LDEP), sendo os sistemas que os seguem intitulados de Lean Data Engineering Systems (LDES). Estes princípios incitam a começar com o fim em mente: processar dados em batch ou em tempo real, sem desperdício de recursos, limitando os custos ao absolutamente necessário para a concluir o trabalho, ou seja, tornando-os o mais lean possível. Os LDEP combinam vertentes do estado da arte em diversas áreas, tais como engenharia de dados, engenharia de software, DevOps, tendo no seu cerne as tecnologias na cloud. O novo processo permitiu à empresa escalar as suas operações de dados, tornando-se capaz de etiquetar imagens quase 10× mais rápido e reduzindo em mais de 99,9% os custos associados, quando comparado com o processo anterior. Adicionalmente, o ciclo de vida dos dados foi reduzido de semanas para horas, mantendo uma qualidade equiparável, ao ser capaz de identificar corretamente 94% das etiquetas em comparação com um homólogo humano

Adaptive Big Data Pipeline

Over the past three decades, data has exponentially evolved from being a simple software by-product to one of the most important companies’ assets used to understand their customers and foresee trends. Deep learning has demonstrated that big volumes of clean data generally provide more flexibility and accuracy when modeling a phenomenon. However, handling ever-increasing data volumes entail new challenges: the lack of expertise to select the appropriate big data tools for the processing pipelines, as well as the speed at which engineers can take such pipelines into production reliably, leveraging the cloud. We introduce a system called Adaptive Big Data Pipelines: a platform to automate data pipelines creation. It provides an interface to capture the data sources, transformations, destinations and execution schedule. The system builds up the cloud infrastructure, schedules and fine-tunes the transformations, and creates the data lineage graph. This system has been tested on data sets of 50 gigabytes, processing them in just a few minutes without user intervention.ITESO, A. C