The Making of Cloud Applications An Empirical Study on Software Development for the Cloud

Cloud computing is gaining more and more traction as a deployment and provisioning model for software. While a large body of research already covers how to optimally operate a cloud system, we still lack insights into how professional software engineers actually use clouds, and how the cloud impacts development practices. This paper reports on the first systematic study on how software developers build applications in the cloud. We conducted a mixed-method study, consisting of qualitative interviews of 25 professional developers and a quantitative survey with 294 responses. Our results show that adopting the cloud has a profound impact throughout the software development process, as well as on how developers utilize tools and data in their daily work. Among other things, we found that (1) developers need better means to anticipate runtime problems and rigorously define metrics for improved fault localization and (2) the cloud offers an abundance of operational data, however, developers still often rely on their experience and intuition rather than utilizing metrics. From our findings, we extracted a set of guidelines for cloud development and identified challenges for researchers and tool vendors

DevOps and information technology service management: A problem management case study

The use of DevOps is a predominant attribute of businesses engaged in the development and maintenance of Information Technology systems. Although literature exploring DevOps practices has expanded, there is still much unexplored territory on its operational ramifications. This is particularly observed when considering their potential impact on ITSM frameworks such as ITIL, which governs Operations. This research aims to establish how DevOps principles and practices can be applied to Problem Management, a core Service Management process. Specifically, it explores which DevOps practices may be used throughout the Problem lifecycle, as well as benefits which may result from them. An exploratory case study was carried out with the participation of Problem Managers operating in a DevOps environment. Three data collection methods were applied: Semi structured interviews, in which participants described their experience and insight in relation to DevOps and Problem Management; documental analysis and observation, where processes and workflows were examined; and a focus group exercise in which study outcomes were discussed and systematized. This research indicates that DevOps practices have varying degrees of significance for a Problem Management process. Practices associated with continuous planning and collaboration are prone to having greater significance in a Problem lifecycle, with the potential of enabling benefits such as quicker Problem identification, higher quality Root Cause Analysis, and improved resolution times. The novelty of insight gathered in this study benefits both academics, through its contribution to an expanding body of knowledge, and professionals, considering the practical and applicable nature of findings. Future work is also presented.A utilização de metodologias DevOps é hoje uma característica predominante de organizações envolvidas no desenvolvimento e manutenção de sistemas de Tecnologia e Informação. Apesar da crescente produção de literatura a examinar práticas DevOps, existe muito território por explorar referente às suas ramificações a nível operacional. Isto é particularmente notável quando se consideram potenciais interações com frameworks de ITSM como o ITIL, que governam Operações. Esta pesquisa tem como objetivo estabelecer quais princípios e práticas DevOps podem ser aplicadas na Gestão de Problemas, um processo central para a Gestão de Serviços. Especificamente, exploramos quais práticas DevOps podem ser utilizadas ao longo do ciclo de vida de um Problema, tal como que benefícios poderão resultar da sua aplicação. Um caso de estudo exploratório foi realizado com a participação de Gestores de Problemas a operar num ambiente DevOps. Três métodos de recolha de dados foram aplicados: Entrevistas semiestruturadas, onde participantes descreveram a sua experiência e conhecimento em relação a DevOps e Gestão de Problemas; análise documental e observação, onde processos operacionais foram examinados; e uma discussão em grupo onde resultados do estudo foram discutidos e sistematizados. Esta investigação indica que práticas DevOps tem variados níveis de significância para um processo de Gestão de Problemas. Práticas associadas ao planeamento contínuo e colaboração tendem a ter maior significância no ciclo de vida de um Problema, com potencial para gerar benefícios como a mais rápida identificação de Problemas, maior qualidade na análise de causa, e melhorias nos tempos de resolução. As conclusões apresentadas neste estudo trazem benefícios tanto para académicos, expandindo o corpo de conhecimento disponível sobre o tema, como para profissionais, considerando a sua natureza prática e aplicável. Direções para trabalho futuro são também apresentadas

An empirical study of architecting for continuous delivery and deployment

Recently, many software organizations have been adopting Continuous Delivery and Continuous Deployment (CD) practices to develop and deliver quality software more frequently and reliably. Whilst an increasing amount of the literature covers different aspects of CD, little is known about the role of software architecture in CD and how an application should be (re-) architected to enable and support CD. We have conducted a mixed-methods empirical study that collected data through in-depth, semi-structured interviews with 21 industrial practitioners from 19 organizations, and a survey of 91 professional software practitioners. Based on a systematic and rigorous analysis of the gathered qualitative and quantitative data, we present a conceptual framework to support the process of (re-) architecting for CD. We provide evidence-based insights about practicing CD within monolithic systems and characterize the principle of "small and independent deployment units" as an alternative to the monoliths. Our framework supplements the architecting process in a CD context through introducing the quality attributes (e.g., resilience) that require more attention and demonstrating the strategies (e.g., prioritizing operations concerns) to design operations-friendly architectures. We discuss the key insights (e.g., monoliths and CD are not intrinsically oxymoronic) gained from our study and draw implications for research and practice.Comment: To appear in Empirical Software Engineerin

A DevOps approach to integration of software components in an EU research project

We present a description of the development and deployment infrastructure being created to support the integration effort of HARNESS, an EU FP7 project. HARNESS is a multi-partner research project intended to bring the power of heterogeneous resources to the cloud. It consists of a number of different services and technologies that interact with the OpenStack cloud computing platform at various levels. Many of these components are being developed independently by different teams at different locations across Europe, and keeping the work fully integrated is a challenge. We use a combination of Vagrant based virtual machines, Docker containers, and Ansible playbooks to provide a consistent and up-to-date environment to each developer. The same playbooks used to configure local virtual machines are also used to manage a static testbed with heterogeneous compute and storage devices, and to automate ephemeral larger-scale deployments to Grid5000. Access to internal projects is managed by GitLab, and automated testing of services within Docker-based environments and integrated deployments within virtual-machines is provided by Buildbot

Explorar kubernetes e devOps num contexto de IoT

Containerized solutions and container orchestration technologies have recently been of great interest to organizations as a way of accelerating both software development and delivery processes. However, adopting these is a rather complex shift that may impact an organization and teams that were already established. This is where development cultures such as DevOps emerge to ease such shift amongst teams, promoting collaboration and automation of development and deployment processes throughout. The purpose of the current dissertation is to illustrate the path that led to the use of DevOps and containerization as means to support the development and deployment of a proof of concept system, Firefighter Sync – an Internet of Things based solution applied to a firefighting monitoring scenario. The goal, besides implementing Firefighter Sync, was to propose and deploy a development and operations ecosystem based on DevOps practices to achieve a full automation pipeline for both the development and operations processes. Firefighter Sync enabled the exploration of such state-of-the-art solutions such as Kubernetes to support container-based deployment and Jenkins for a fully automated CI/CD pipeline. Firefighter Sync clearly illustrates that addressing the development of a system from a DevOps perspective from the very beginning, although it requires an accentuated learning curve due to the large range of concepts and technologies addressed throughout, has illustrated to effectively impact the development process as well as ease the solution for future evolution. A good example is the automation process pipeline, that whilst allowing an easy integration of new features within a DevOps process – implies addressing the development and operations as a whole – it abstracts specific technological concerns turning these transversals to the traditional stages from development to deployment.Soluções de contentores e orquestração de contentores têm vindo a tornar-se de grande interesse para as organizações como uma forma de acelerar os processos de desenvolvimento e entrega de software. No entanto, adotá-las é uma mudança bastante complexa que pode impactar uma organização e equipas já estabelecidas. É aqui que surgem culturas como o DevOps para facilitar essa mudança, promovendo a colaboração e a automação dos processos de desenvolvimento e deployment entre equipas. O objetivo desta dissertação é ilustrar o caminho que levou ao uso de DevOps e à conteinerização de modo a apoiar o desenvolvimento e o deployment de um sistema como prova de conceito, o Firefighter Sync – uma solução baseada na Internet das Coisas aplicada a um cenário de monitorização de combate a incêndios. Além de implementar o Firefighter Sync, o objetivo era também propor e implementar um ecossistema de desenvolvimento e operações com base nas práticas de DevOps para alcançar uma pipeline de automação completa para os processos de desenvolvimento e operações. O Firefighter Sync permitiu explorar soluções que constituem o estado da arte neste contexto, como o Kubernetes para apoiar o deployment baseado em contentores e o Jenkins para suportar a pipeline de CI/CD totalmente automatizada. O Firefighter Sync ilustra claramente que abordar o desenvolvimento de um sistema a partir da perspectiva de DevOps, embora exija uma curva de aprendizagem acentuada devido à grande variedade de conceitos e tecnologias inerentes ao longo do processo, demonstrou tornar mais eficiente o processo de desenvolvimento, bem como facilitar evolução futura. Um exemplo é a pipeline de automação, que permite uma fácil integração de novos recursos dentro de um processo de DevOps – que implica abordar o desenvolvimento e as operações como um todo – abstraindo assim preocupações tecnológicas específicas, transformando essas transversais nas fases tradicionais do desenvolvimento ao deployment.Mestrado em Engenharia Informátic

Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

This report documents the program and the outcomes of GI-Dagstuhl Seminar 16394 "Software Performance Engineering in the DevOps World". The seminar addressed the problem of performance-aware DevOps. Both, DevOps and performance engineering have been growing trends over the past one to two years, in no small part due to the rise in importance of identifying performance anomalies in the operations (Ops) of cloud and big data systems and feeding these back to the development (Dev). However, so far, the research community has treated software engineering, performance engineering, and cloud computing mostly as individual research areas. We aimed to identify cross-community collaboration, and to set the path for long-lasting collaborations towards performance-aware DevOps. The main goal of the seminar was to bring together young researchers (PhD students in a later stage of their PhD, as well as PostDocs or Junior Professors) in the areas of (i) software engineering, (ii) performance engineering, and (iii) cloud computing and big data to present their current research projects, to exchange experience and expertise, to discuss research challenges, and to develop ideas for future collaborations

DevSecOps for web applications: a case study

O paradigma DevOps permite agilizar o processo de entrega de software. Visa reduzir as barreiras existentes entre as equipas responsáveis pelo desenvolvimento e as equipas de operação. Com recurso a estruturas de pipelines o processo de desenvolvimento de software é conduzido através de diversas etapas até à sua entrega. Estas estruturas permitem automatizar várias tarefas de forma a evitar erros humanos, liberta os intervenientes de tarefas morosas e repetitivas. Mais previsível e com maior exatidão o tempo necessário para as entregas de software é encurtado e mais frequente. Dadas estas vantagens o paradigma tem muita adoção por parte da indústria de desenvolvimento, no entanto, o aumento do volume das entregas acarreta desafios, nomeadamente no que diz respeito à segurança das soluções desenvolvidas. Negligenciar os fatores de segurança pode levar a organização a acarretar com custos financeiros e denegrir a sua reputação. A integração entre o paradigma DevOps e segurança originou o paradigma designado por DevSecOps. Este visa a adoção pelo processo de desenvolvimento de ações de segurança, que após inseridas nas diversas fases de entrega, permitirão analisar e validar a solução, de forma a assegurar a sua consistência. A arquitetura das aplicações web é por sua natureza acessível, o que resulta à sua maior exposição. Este projeto apresenta uma lista de problemas de segurança encontrados durante a pesquisa efetuada no domínio das aplicações web, analisa quais as ferramentas para a deteção e resolução destes problemas, quais as suas implicações no tempo de entrega de software e a sua eficiência na deteção de falhas. Concluí com uma implementação de um fluxo de execução utilizando o paradigma DevSecOps, para compreender a sua contribuição no melhoramento da qualidade do software.The DevOps paradigm streamlines the software delivery process, reducing the barriers between the teams involved in development and operations. It relies on pipelines to structure the development process until delivered. These structures enable the automation of many tasks, avoiding human error and freeing the team elements from doing slow and repeated tasks. More predictable and accurate development allows teams to reduce the time required for software deliveries and make them more frequent. Despite the wide adoption of the paradigm, the increase in deliveries cannot compromise the security aspects of the developed solutions. Companies may incur financial costs and tarnish their reputations by neglecting security factors. Joining security and DevOps originate a new paradigm, DevSecOps. It aims to bring more quality compliance and avoid risk by adding security considerations to discover all potential security defects before delivery. Web applications architecture, by their accessibility intent, has a vast exposed area. This project presents a list of common security issues found during the research performed in the web application security domain analyses, what tools are used to detect and solve these problems, which time implications they cause in the overall software delivery and their effectiveness in defect detection. It concludes with implementing a pipeline using the DevSecOps paradigm to establish its viability in improving software quality

A Framework of DevSecOps for Software Development Teams

This master's thesis explores a broad evaluation of automated security testing in the context of DevOps practices. The primary objective of this study is to propose a framework that facilitates the seamless integration of security scanning tools within DevOps practices. The thesis will focus on examining the existing set of tools and their effective integration into fully automated DevOps CI/CD pipelines. The thesis starts by examining the theoretical concepts of DevOps and provides guidelines for integrating security within DevOps methodologies. Furthermore, it assesses the current state of security by analysing the OWASP Web API top 10 security vulnerability list and evaluating existing security automation tools. Additionally, the research investigates the performance and efficacy of these tools across various stages of the SDLC and investigates ongoing research and development activities. A fully automated DevOps CI/CD pipeline is implemented to integrate security scanning tools, enforcing complete security checks throughout the SDLC. Azure DevOps build and release pipelines, along with Snyk, were used to create a comprehensive automated security scanning framework. The study considerably investigates the integration of these security scanning tools and assesses their influence on the overall security posture of the developed applications. The finding of the study reveals that security scanning tools can be efficiently integrated into fully automated DevOps practices. Based on the results, recommendations are provided for the selection of suitable tools and techniques to achieve a DevSecOps practice. In conclusion, this thesis provides valuable insights into security integration in DevOps practices, highlighting the effectiveness of security automation tools. The research also recommends areas for further improvements to meet the industry's evolving requirements