Jatkuvaan ohjelmistokehitykseen siirtyminen vakiintuneessa organisaatiossa

Software development today has rapidly developed into a significant part of business and its value creation chain. Increasingly more stakeholders within an organization are tied in with more frequent software releases. This has driven organizations to adapt to more flexible and continuous software development methods. This Master’s Thesis addresses the challenges, advantages and disadvantages associated in shifting an organization’s software development culture towards that of continuous development. The specific type of continuous development within this research considers the new software development culture of DevOps. DevOps is seen as a fundamental change in the IT world today for the transition towards continuous software development, where dedication is given to the successful collaboration between development and operations. The aim of this research is to discover the vastness of attempting to change an organizational culture for an improved and modern software development process for all stakeholders involved. Furthermore, this research attempts to provide the organization at hand with information on how and where to begin initiating the required changes. New cloud computing technologies have enabled development teams to become less dependent on companies’ traditional IT departments. The research is conducted via literature review and the data collected through interviews with employees of the organization attempting to shift towards continuous development. Further information is gathered through three case studies of other companies that have successfully undergone a transition towards continuous development and DevOps.Tänä päivänä ohjelmistokehityksestä on nopeasti muodostunut merkittävä osa liiketoimintaa ja sen lisäarvon tuottamista. Yhtiöiden sisällä yhä useampi sidosryhmä on osallisena yhä useammin toistuvissa ohjelmistojulkaisuissa. Tästä johtuen yritykset ovat joutuneet sopeutumaan joustaviin ja jatkuviin tapoihin kehittää ohjelmistoa. Tämä diplomityö tutkii yhtiön jatkuvaan ohjelmistokehitykseen siirtymisen haasteita, hyötyjä ja haittoja. Jatkuvan ohjelmistokehityksen tyyppi, jota tässä työssä tutkitaan, on nimeltään DevOps. DevOps:ia pidetään keskeisenä muutoksena nykypäivän IT-alalla jatkuvaan kehitykseen siirtymisessä. Sen pääpiirteeksi koetaan sulava yhteistyö kehityksen ja ylläpidon välillä. Tämän tutkimuksen tavoite on selvittää yrityksen ohjelmistokehityksen muuttamisen laajuuden ottamalla samalla sen kaikki sidosryhmät huomioon. Lisäksi tämä tutkimus pyrkii tuottamaan yritykselle, jolle tutkimus tehdään, lisätietoa siitä miten tarvittavat muutokset voidaan käynnistää ja toteuttaa. Uudet pilvipalvelut ovat lisänneet kehityksen autonomisia työskentelytapoja ja vähentäneet heidän riippuvuuksia perinteisen IT-osaston toiminnollisuuksista. Tutkimus toteutetaan kirjallisuuskatsauksen sekä yrityksen eri sidosryhmien haastattelujen kautta. Lisätietoa kerätään esimerkkien avulla, joissa tutkitaan kolmen eri yrityksen menestyksekkäitä siirtymisiä jatkuvan ohjelmistokehityksen pariin

LEAN DATA ENGINEERING. COMBINING STATE OF THE ART PRINCIPLES TO PROCESS DATA EFFICIENTLYS

The present work was developed during an internship, under Erasmus+ Traineeship program, in Fieldwork Robotics, a Cambridge based company that develops robots to operate in agricultural fields. They collect data from commercial greenhouses with sensors and real sense cameras, as well as with gripper cameras placed in the robotic arms. This data is recorded mainly in bag files, consisting of unstructured data, such as images and semi-structured data, such as metadata associated with both the conditions where the images were taken and information about the robot itself. Data was uploaded, extracted, cleaned and labelled manually before being used to train Artificial Intelligence (AI) algorithms to identify raspberries during the harvesting process. The amount of available data quickly escalates with every trip to the fields, which creates an ever-growing need for an automated process. This problem was addressed via the creation of a data engineering platform encom- passing a data lake, data warehouse and its needed processing capabilities. This platform was created following a series of principles entitled Lean Data Engineering Principles (LDEP), and the systems that follows them are called Lean Data Engineering Systems (LDES). These principles urge to start with the end in mind: process incoming batch or real-time data with no resource wasting, limiting the costs to the absolutely necessary for the job completion, in other words to be as lean as possible. The LDEP principles are a combination of state-of-the-art ideas stemming from several fields, such as data engineering, software engineering and DevOps, leveraging cloud technologies at its core. The proposed custom-made solution enabled the company to scale its data operations, being able to label images almost ten times faster while reducing over 99.9% of its associated costs in comparison to the previous process. In addition, the data lifecycle time has been reduced from weeks to hours while maintaining coherent data quality results, being able, for instance, to correctly identify 94% of the labels in comparison to a human counterpart.Este trabalho foi desenvolvido durante um estágio no âmbito do programa Erasmus+ Traineeship, na Fieldwork Robotics, uma empresa sediada em Cambridge que desenvolve robôs agrícolas. Estes robôs recolhem dados no terreno com sensores e câmeras real- sense, localizados na estrutura de alumínio e nos pulsos dos braços robóticos. Os dados recolhidos são ficheiros contendo dados não estruturados, tais como imagens, e dados semi- -estruturados, associados às condições em que as imagens foram recolhidas. Originalmente, o processo de tratamento dos dados recolhidos (upload, extração, limpeza e etiquetagem) era feito de forma manual, sendo depois utilizados para treinar algoritmos de Inteligência Artificial (IA) para identificar framboesas durante o processo de colheita. Como a quantidade de dados aumentava substancialmente com cada ida ao terreno, verificou-se uma necessidade crescente de um processo automatizado. Este problema foi endereçado com a criação de uma plataforma de engenharia de dados, composta por um data lake, uma data warehouse e o respetivo processamento, para movimentar os dados nas diferentes etapas do processo. Esta plataforma foi criada seguindo uma série de princípios intitulados Lean Data Engineering Principles (LDEP), sendo os sistemas que os seguem intitulados de Lean Data Engineering Systems (LDES). Estes princípios incitam a começar com o fim em mente: processar dados em batch ou em tempo real, sem desperdício de recursos, limitando os custos ao absolutamente necessário para a concluir o trabalho, ou seja, tornando-os o mais lean possível. Os LDEP combinam vertentes do estado da arte em diversas áreas, tais como engenharia de dados, engenharia de software, DevOps, tendo no seu cerne as tecnologias na cloud. O novo processo permitiu à empresa escalar as suas operações de dados, tornando-se capaz de etiquetar imagens quase 10× mais rápido e reduzindo em mais de 99,9% os custos associados, quando comparado com o processo anterior. Adicionalmente, o ciclo de vida dos dados foi reduzido de semanas para horas, mantendo uma qualidade equiparável, ao ser capaz de identificar corretamente 94% das etiquetas em comparação com um homólogo humano

An evaluation of galaxy and ruffus-scripting workflows system for DNA-seq analysis

>Magister Scientiae - MScFunctional genomics determines the biological functions of genes on a global scale by using large volumes of data obtained through techniques including next-generation sequencing (NGS). The application of NGS in biomedical research is gaining in momentum, and with its adoption becoming more widespread, there is an increasing need for access to customizable computational workflows that can simplify, and offer access to, computer intensive analyses of genomic data. In this study, the Galaxy and Ruffus frameworks were designed and implemented with a view to address the challenges faced in biomedical research. Galaxy, a graphical web-based framework, allows researchers to build a graphical NGS data analysis pipeline for accessible, reproducible, and collaborative data-sharing. Ruffus, a UNIX command-line framework used by bioinformaticians as Python library to write scripts in object-oriented style, allows for building a workflow in terms of task dependencies and execution logic. In this study, a dual data analysis technique was explored which focuses on a comparative evaluation of Galaxy and Ruffus frameworks that are used in composing analysis pipelines. To this end, we developed an analysis pipeline in Galaxy, and Ruffus, for the analysis of Mycobacterium tuberculosis sequence data. Furthermore, this study aimed to compare the Galaxy framework to Ruffus with preliminary analysis revealing that the analysis pipeline in Galaxy displayed a higher percentage of load and store instructions. In comparison, pipelines in Ruffus tended to be CPU bound and memory intensive. The CPU usage, memory utilization, and runtime execution are graphically represented in this study. Our evaluation suggests that workflow frameworks have distinctly different features from ease of use, flexibility, and portability, to architectural designs

Internal Framework Refactor/Improvement

The FCN (Fusion CoNsole) is a framework created with the purpose of providing developers with the rapid development of internal management applications that bundles solutions for common requisites in medium to large companies, reducing the setup boilerplate for the teams. The developed applications are available under a single hostname to provide its users with a single point of access. However, the current version of the FCN has several problems and bottlenecks that resulted from the lack of opportunity to solve them and due to the technology stack used becoming outdated. Such problems sometimes lead to time expensive processes and dependencies that, otherwise, could be avoided. From these needs, IAP – Internal Applications Platform – is born. IAP goals are to be a new version of the FCN where the raised problems are solved. The purpose of this thesis is to raise, understand, and solve the current problems of the framework. Such was achieved through a deep analysis of the current architecture, technologies that could be used to solve them or enhance the current version and to study and design solutions that allow to overcome them. It is hoped that the presented solutions not only solve the raised problems but improve the overall usability for developers and applications’ users. In the end, the presented solution not only solves the raised problems as it improves IAP’s usability for software developers and users of the applications built on top of it.O FCN (Fusion CoNsole) é uma framework criada com o propósito de fornecer aos desenvolvedores de software um rápido desenvolvimento de aplicações de gestão interna que contém soluções para requisitos comuns em médias e grandes empresas, reduzindo todo o trabalho inicial de configuração de uma nova aplicação. As aplicações desenvolvidas com o FCN estão disponíveis através de um único hostname de forma a garantir um ponto de acesso único. Apesar de poder ser visto como apenas uma coleção de componentes e ferramentas para desenvolvimento frontend, o FCN tenta, não só, promover o padrão da utilização e criação de componentes web reutilizáveis, bem como tenta facilitar o desenvolvimento de tais componentes utilizando um único tema definido pela empresa. O FCN é, ainda, implantado sobre uma plataforma que apresenta uma arquitetura de micro-serviços desenvolvida internamente. Tal permite quer a adição quer a utilização de serviços disponíveis para compor aplicações que tenham como objetivo vir a ser usadas por toda a empresa. Uma aplicação desenvolvida com o FCN, terá uma aparência e experiência de utilização semelhante a todas as aplicações desenvolvidas dentro da empresa, o que torna a experiência mais suave e ameniza a mudança de contexto entre as aplicações. Contudo, a versão atual do FCN apresenta vários problemas e limitações que resultam da falta de oportunidade para a resolução dos mesmos e devido à utilização de tecnologias desatualizadas. Problemas esses que levam, muitas vezes, a processos mais lentos e a dependências indesejadas que, caso contrário, poderiam ser evitadas. Nasce desta necessidade o IAP – Internal Applications Platform. O IAP tem como objetivo ser uma nova versão do FCN em que os problemas identificados estão resolvidos e/ou minimizados. Esta tese tem como objetivo o levantamento, a compreensão e a resolução dos problemas da atual framework. Tal foi alcançado através de uma profunda análise da arquitetura atual, das tecnologias que podiam ser utilizadas para ajudar a resolver e melhorar tais problemas e do estudo e design de soluções que possibilitem a sua resolução. No final, a solução apresentada neste documento não só resolve os problemas levantados como também melhora a usabilidade para os desenvolvedores de software e utilizadores das respetivas aplicações

Best practices in cloud-based Penetration Testing

This thesis addresses and defines best practices in cloud-based penetration testing. The aim of this thesis is to give guidance for penetration testers how cloud-based penetration testing differs from traditional penetration testing and how certain aspects are limited compared to traditional penetration testing. In addition, this thesis gives adequate level of knowledge to reader what are the most important topics to consider when organisation is ordering a penetration test of their cloud-based systems or applications. The focus on this thesis is the three major cloud service providers (Microsoft Azure, Amazon AWS, and Google Cloud Platform). The purpose of this research is to fill the gap in scientific literature about guidance for cloud-based penetration testing for testers and organisations ordering penetration testing. This thesis contains both theoretical and empirical methods. The result of this thesis is focused collection of best practices for penetration tester, who is conducting penetration testing for cloud-based systems. The lists consist of topics focused on planning and execution of penetration testing activities

Software development using DevOps tools and CD pipelines : a case study

The objective of this study is to get insight on the usage of automated build and deployment pipelines by software development teams at Avaintec Oy in the context of DevOps methodologies. Automated deployments are used widely in today’s software development and they allow for complex installation operations and testing to be done in a time-saving manner as less manual work is required. While there is a lot of research in the area of DevOps and automation pipelines, standardised ways of working and best practices are still vague in many ways. Studying the effects of DevOps and the use of the automation pipelines in the scope of the development team is important to establish practices and ways of working that can best support the work of the developers and also be cost effective for the companies. The research methods of this study are a literature review of the most recent literature on using automation in software development, and a case study of the process of taking DevOps methodologies and a Continuous Deployment pipeline into use at Avaintec Oy. The literature is analysed with the help of a theme matrix. The case study was conducted by interviewing selected members of development teams at Avaintec. The interviews were recorded and transcribed, and then analysed with a theme matrix. The output from the literature review is reflected to the output from the interviews in the case study section. The conclusions from this study indicate that development teams find DevOps and DevOps pipelines useful, despite the learning curve in the new tools and methodologies and the amount of initial setup work. The evolution of an automated build and deployment pipeline should be continuous and pipelines should be built incrementally, both of which were true for the studied case. The engagement of developers to the Operations side is a challenge present in both literature and the studied case, as developers at times are not that familiar with the pipeline. Apart from the pipeline, products should also be developed to be suitable for DevOps and automatic deployments, microservices seem to provide advantage in this

Verkkosovellusten jatkuva kehitys ja julkaisuautomaatio

Software development methods have evolved towards more agile practices for getting changes implemented more quickly. While the new features are finished faster, the release processes are also made more automated to get the changes in production environment reliably, repeatably and rapid. This thesis examines the practices used for continuous software development and web application release automation. The main objective is to find out and implement a way for making changes agilely and getting them tested and released in several environments effortlessly. After the research, different tools are sought for, compared and suitable tools are selected. Lean software development is chosen as the working practice for the development. GitHub enterprise is used for version control, JetBrains TeamCity for continuous integration and Octopus Deploy for deployment automation. SonarQube is used for static code analysis and UseTrace for automated functionality testing. The lean development practice is found well fit for real world use. The deployment pipeline is also well operational, founding issues early and deployments are enabled steady, effortless and fast. Some issues with code analysis are found due to the decisions in the application implementation. UseTrace tests have occasionally some false positives in the failing test results but overall they worked as expected.Sovelluskehitysmenetelmät ovat kehittyneet ketterämmiksi, jotta muutokset saataisiin toteutettua nopeammin. Kun uudet ominaisuudet valmistuvat nopeammin, myös julkaisuprosesseista tehdään automatisoidumpia, jotta muutokset saadaan tuotantoon luotettavasti, toistettavasti ja nopeasti. Tässä työssä tutkitaan menetelmiä, joita käytetään jatkuvaan sovelluskehitykseen ja verkkosovellusten julkaisuautomaatioon. Pääasiallisena tavoitteena on selvittää ja toteuttaa tapa, jolla muutoksia voidaan tehdä ketterästi ja julkaista testattuina moniin ympäristöihin pienellä vaivalla. Tutkimuksen jälkeen etsitään erilaisia työkaluja, joita vertaillaan keskenään ja soveltuvat työkalut valitaan. Lean-malli valitaan sovelluskehityksessä käytettäväksi tavaksi. GitHub enterprise:a käytetään versionhallintaan, JetBrains TeamCity:ä jatkuvaan integraatioon ja Octopus Deploy:ta jatkuvaan toimittamiseen. SonarQube:a käytetään staattiseen koodianalyysiin ja UseTrace:a automaattiseen funktionaalisuustestaamiseen. Lean-sovelluskehitysmalli todetaan hyvin toimivaksi todellisessa käytössä. Julkaisuputki on myös hyvin toimiva, löytäen ongelmat ajoissa ja mahdollistaen julkaisut luotettavasti, vaivattomasti ja nopeasti. Koodianalyysin osalta joitain ongelmia ilmenee sovellustoteutukseen liittyvistä päätöksistä johtuen. UseTrace-automaattitestit tuottavat satunnaisesti virheellisiä ongelmia testituloksissa, mutta yleisesti ottaen ne toimivat odotetusti

Automated, Systematic and Parallel Approaches to Software Testing in Bioinformatics

Software quality assurance becomes especially critical if bioinformatics tools are to be used in a translational medical setting, such as analysis and interpretation of biological data. We must ensure that only validated algorithms are used, and that they are implemented correctly in the analysis pipeline – and not disrupted by hardware or software failure. In this thesis, I review common quality assurance practice and guidelines for bioinformatics software testing. Furthermore, I present a novel cloud-based framework to enable automated testing of genetic sequence alignment programs. This framework performs testing based on gold standard simulation data sets, and metamorphic testing. I demonstrate the effectiveness of this cloudbased framework using two widely used sequence alignment programs, BWA and Bowtie, and some fault-seeded ‘mutant’ versions of BWA and Bowtie. This preliminary study demonstrates that this type of cloud-based software testing framework is an effective and promising way to implement quality assurance in bioinformatics software that is used in genomic medicine