A Systematic Mapping Study of Empirical Studies on Software Cloud Testing Methods

Context: Software has become more complicated, dynamic, and asynchronous than ever, making testing more challenging. With the increasing interest in the development of cloud computing, and increasing demand for cloud-based services, it has become essential to systematically review the research in the area of software testing in the context of cloud environments. Objective: The purpose of this systematic mapping study is to provide an overview of the empirical research in the area of software cloud-based testing, in order to build a classification scheme. We investigate functional and non-functional testing methods, the application of these methods, and the purpose of testing using these methods. Method: We searched for electronically available papers in order to find relevant literature and to extract and analyze data about the methods used. Result: We identified 69 primary studies reported in 75 research papers published in academic journals, conferences, and edited books. Conclusion: We found that only a minority of the studies combine rigorous statistical analysis with quantitative results. The majority of the considered studies present early results, using a single experiment to evaluate their proposed solution

On the Fly Orchestration of Unikernels: Tuning and Performance Evaluation of Virtual Infrastructure Managers

Network operators are facing significant challenges meeting the demand for more bandwidth, agile infrastructures, innovative services, while keeping costs low. Network Functions Virtualization (NFV) and Cloud Computing are emerging as key trends of 5G network architectures, providing flexibility, fast instantiation times, support of Commercial Off The Shelf hardware and significant cost savings. NFV leverages Cloud Computing principles to move the data-plane network functions from expensive, closed and proprietary hardware to the so-called Virtual Network Functions (VNFs). In this paper we deal with the management of virtual computing resources (Unikernels) for the execution of VNFs. This functionality is performed by the Virtual Infrastructure Manager (VIM) in the NFV MANagement and Orchestration (MANO) reference architecture. We discuss the instantiation process of virtual resources and propose a generic reference model, starting from the analysis of three open source VIMs, namely OpenStack, Nomad and OpenVIM. We improve the aforementioned VIMs introducing the support for special-purpose Unikernels and aiming at reducing the duration of the instantiation process. We evaluate some performance aspects of the VIMs, considering both stock and tuned versions. The VIM extensions and performance evaluation tools are available under a liberal open source licence

Pakettiprosessointijärjestelmien Suorituskykyanalyysi

This thesis investigates the use of measurement, simulation, and modeling methods for the performance analysis of packet processing systems, and more precisely hardware accelerated multiprocessor system-on-chip (MPSoC) devices running task-parallel applications. To guarantee the tight latency and throughput requirements, the devices often incorporate complex hardware accelerated packet scheduling mechanisms. At the same time, due to the complexity of these systems, different software abstractions, such as task-based programming models, are used to develop packet processing applications. These challenges, together with dynamic characteristics of the packet streams makes the performance analysis of packet processing systems non-trivial. We demonstrate that, with extended queue disciplines and support for modeling parallelism, resource network methodology is a viable approach for modeling complex MPSoC based systems running task-based parallel applications on dynamic workloads. The main contributions of our work are three-fold. First, we have extended the toolset of an existing in-house modeling and simulation software, Performance Simulation Environment. The extensions enable modeling of user-definable queue disciplines, which further enable flexible modeling of complex hardware interactions of MPSoCs and the parallelism of task-based programming models. Secondly, we have studied, instrumented, and measured the characteristics of a packet processing system. Finally we have modeled a multi-blade packet processing system with customizable workload and task-parallel application models, and run simulation experiments. In both experiments, the model acts as expected. According to the experiment results, the resource network concept seems to be a viable tool for the performance analysis of packet processing systems. The chosen abstraction level provides desired balance between the functionality, ease of use, and simulation performance.Tässä työssä tutkitaan mittaus-, mallinnus-, ja simulaatiometodien käyttöä pakettiprosessisysteemien, tarkemmin ottaen tehtävärinnakkaisia sovelluksia ajavien laitteistokiihdytettyjen moniydinjärjestelmien, suorityskykyanalyysiin. Tiukoista viive- ja läpivirtausvaatimuksista johtue pakettiprosessointilaitteistot sisältävät usein monimutkaisia laitteistokiihdytettyjä pakettiajoitusmekanismeja. Laittestojen monimutkaisuudesta johtuen pakettiprosessointisovellusten kehittämiseen käytetään usein erilaisia ohjelmointiabstraktioita, kuten tehtävärinnakkaisia ohjelmointimalleja. Laitteston ja ohjelmiston asettamat haasteet yhdessä pakettivirtojen dynaamisen luonteen kanssa tekevät pakettiprosessointijärjestelmien suorituskykyanalyysista epätriviaalia. Työssä havainnollistamme, että laajennettujen jonokurien ja rinnakkaismallinnustuen avulla resurssiverkkometodologia on toimiva lähestymistapa tehtävärinnakkaisia rinnakkaisohjelmointisovelluksia ajavien monimutkaisten laitteistokiihdytettyjen moniydinjärjestelmien suorituskykyanalyysiin dynaamisilla työkuormilla. Työmme päätulokset ovat kolmiosaiset. Ensinnäkin, olemme laajentaneet olemassaolevan mallinnus- ja simulaatioohjelmiston, Performance Simulation Environmentin, ohjelmointityökaluja. Laajennukset mahdollistavat käyttäjän määriteltävien jonokurien mallintamisen, mikä edelleen mahdollistaa tehtävärinnakkaisia sovelluksia ajavien laittestokiihdytettyjen moniydinjärjestelmien laittestovuorovaikutusten joustavan mallinnuksen. Toiseksi, olemme tutkineet ja mitanneet erään pakettiprosessointijärjestelmän ominaisuuksia. Viimeiseksi, olemme mallintaneet pakettiprosessointijärjestelmän muunnettavilla työkuormilla ja tehtävärinnakkaisilla sovellusmalleilla, sekä suorittaneet näitä simulaatiokokein. Molempien kokeiden mallit käyttäytyvät odotetulla tavalla. Koetulosten perusteella resurssiverkkokonsepti vaikuttaa toimivalta työkalulta kompleksien pakettiprosessointijärjestelmien suorituskykyanalyysiin. Valittu abstraktiotaso tarjoaa toivotun tasapainon simulaation suorituskyvyn, toiminnallisuuden ja helppokäyttöisyyden välillä

Thin Hypervisor-Based Security Architectures for Embedded Platforms

Virtualization has grown increasingly popular, thanks to its benefits of isolation, management, and utilization, supported by hardware advances. It is also receiving attention for its potential to support security, through hypervisor-based services and advanced protections supplied to guests. Today, virtualization is even making inroads in the embedded space, and embedded systems, with their security needs, have already started to benefit from virtualization’s security potential. In this thesis, we investigate the possibilities for thin hypervisor-based security on embedded platforms. In addition to significant background study, we present implementation of a low-footprint, thin hypervisor capable of providing security protections to a single FreeRTOS guest kernel on ARM. Backed by performance test results, our hypervisor provides security to a formerly unsecured kernel with minimal performance overhead, and represents a first step in a greater research effort into the security advantages and possibilities of embedded thin hypervisors. Our results show that thin hypervisors are both possible and beneficial even on limited embedded systems, and sets the stage for more advanced investigations, implementations, and security applications in the future

MECInOT: a multi-access edge computing and industrial internet of things emulator for the modelling and study of cybersecurity threats

In recent years, the Industrial Internet of Things (IIoT) has grown rapidly, a fact that has led to an increase in the number of cyberattacks that target this environment and the technologies that it brings together. Unfortunately, when it comes to using tools for stopping such attacks, it can be noticed that there are inherent weaknesses in this paradigm, such as limitations in computational capacity, memory and network bandwidth. Under these circumstances, the solutions used until now in conventional scenarios cannot be directly adopted by the IIoT, and so it is necessary to develop and design new ones that can effectively tackle this problem. Furthermore, these new solutions must be tested in order to verify their performance and viability, which requires testing architectures that are compatible with newly introduced IIoT topologies. With the aim of addressing these issues, this work proposes MECInOT, which is an architecture based on openLEON and capable of generating test scenarios for the IIoT environment. The performance of this architecture is validated by creating an intelligent threat detector based on tree-based algorithms, such as decision tree, random forest and other machine learning techniques. Which allows us to generate an intelligent and to demonstrate, we could generate an intelligent threat detector and demonstrate the suitability of our architecture for testing solutions in IIoT environments. In addition, by using MECInOT, we compare the performance of the different machine learning algorithms in an IIoT network. Firstly, we present the benefits of our proposal, and secondly, we describe the emulation of an IIoT environment while ensuring the repeatability of the experiments

Advancing Operating Systems via Aspect-Oriented Programming

Operating system kernels are among the most complex pieces of software in existence to- day. Maintaining the kernel code and developing new functionality is increasingly compli- cated, since the amount of required features has risen significantly, leading to side ef fects that can be introduced inadvertedly by changing a piece of code that belongs to a completely dif ferent context. Software developers try to modularize their code base into separate functional units. Some of the functionality or “concerns” required in a kernel, however, does not fit into the given modularization structure; this code may then be spread over the code base and its implementation tangled with code implementing dif ferent concerns. These so-called “crosscutting concerns” are especially dif ficult to handle since a change in a crosscutting concern implies that all relevant locations spread throughout the code base have to be modified. Aspect-Oriented Software Development (AOSD) is an approach to handle crosscutting concerns by factoring them out into separate modules. The “advice” code contained in these modules is woven into the original code base according to a pointcut description, a set of interaction points (joinpoints) with the code base. To be used in operating systems, AOSD requires tool support for the prevalent procedu- ral programming style as well as support for weaving aspects. Many interactions in kernel code are dynamic, so in order to implement non-static behavior and improve performance, a dynamic weaver that deploys and undeploys aspects at system runtime is required. This thesis presents an extension of the “C” programming language to support AOSD. Based on this, two dynamic weaving toolkits – TOSKANA and TOSKANA-VM – are presented to permit dynamic aspect weaving in the monolithic NetBSD kernel as well as in a virtual- machine and microkernel-based Linux kernel running on top of L4. Based on TOSKANA, applications for this dynamic aspect technology are discussed and evaluated. The thesis closes with a view on an aspect-oriented kernel structure that maintains coherency and handles crosscutting concerns using dynamic aspects while enhancing de- velopment methods through the use of domain-specific programming languages

Dynamic Isolated Domains

Operating System-level Virtualization is virtualization technology based on running multiple isolated userspace instances, commonly referred to as containers, on top of a single operating system kernel. The fundamental difference compared to traditional virtualization is that the targets of virtualization in OS-level virtualization are kernel resources, not hardware. OS-level virtualization is used to implement Bring Your Own Device (BYOD) policies on contemporary mobile platforms. Current commercial BYOD solutions, however, don't allow for applications to be containerized dynamically upon user request. The ability to do so would greatly improve the flexibility and usability of such schemes. In this work we study if existing OS-level virtualization features in the Linux kernel can meet the needs of use cases reliant on such dynamic isolation. We present the design and implementation of a prototype which allows applications in dynamic isolated domains to be migrated from one device to another. Our design fits together with security features in the Linux kernel, allowing the security policy influenced by user decisions to be migrated along with the application. The deployability of the design is improved by basing the solution on functionality already available in the mainline Linux kernel. Our evaluation shows that the OS-level virtualization features in the Linux kernel indeed allow applications to be isolated in a dynamic fashion, although known gaps in the compartmentalization of kernel resources require trade-offs between the security and interoperability to be made in the design of such containers

Testbed para aplicações e serviços 5G

With 5G technologies growing, there is the need to adopt the NFV paradigm to align with the demands of a 5G environment. This entails the decoupling of hardware from the functions it provides, expediting service development. However, the adoption of NFV raises many challenges, one of which is the validation of Virtualized Network Functions. 5GASP is a European project that aims to shorten the idea-to-market process by creating a fully automated and self-service 5G testbed. In alignment with 5GASP’s objectives, this dissertation seeks to design tests for the validation of Network Applications used within a 5G environment. Furthermore, the Service-Based Architecture offers the NEF, which enables external application access to 5G core functions. With these concepts in mind, the solution provided by this dissertation uses NEF to verify whether a Network Application can correctly interact with a 5G Network. This is achieved by integrating NEF into the validation pipeline of 5GASP and developing tests to assess the behavior of Network Applications. In summary, this document presents the designed architecture and the implementation of these mechanisms, along with their respective results and outcomes.Com o crescimento das tecnologias 5G, há a necessidade de adotar o paradigma NFV para responder às necessidades de um ambiente 5G, o que leva à separação do hardware das funções que ele fornece, acelerando o desenvolvimento de serviços. No entanto, a adoção de NFV traz muitos desafios, um dos quais é a validação das VNF. O 5GASP é um projeto europeu que visa a encurtar o processo de ideia-ao-mercado, criando um ambiente de teste 5G totalmente automatizado. Em alinhamento com os objetivos do 5GASP, esta dissertação procura criar testes para a validação de Network Applications usadas em ambiente 5G. Além disso, a Arquitetura Baseada em Serviços oferece o NEF, que permite o acesso de aplicações externas às funções da rede 5G. Com estes conceitos em mente, a solução apresentada por esta dissertação utiliza o NEF para verificar se uma Network Application consegue interagir corretamente com uma rede 5G. Isto é alcançado integrando o NEF no pipeline de validação do 5GASP e desenvolvendo testes para avaliar o comportamento das Network Applications. No geral, este documento apresenta a arquitetura definida e a implementação desses mecanismos, juntamente com seus respectivos resultados.Mestrado em Engenharia de Computadores e Telemátic

Leveraging Virtualization for Performance Driven Development

This thesis contains the research component of a software engineering study to create a .NET application performance testing lab, and several guided learning activities intended to teach the fundamentals of how to use it. In arriving upon the research which serves as the groundwork for this project, an introduction to the concepts of software performance, the risks associated with performances, and an approach to mitigating this risks called performance driven development is presented. This introduction is expanded by an overview of how performance is affected from application, network, database and presentation aspects. To address problems associated with performance in .NET web applications, a virtual test lab has been created on the software engineering lab server at Regis University\u27s Academic Research Network (ARNe), and this paper documents the architecture of that test lab. In order to demonstrate how it can be used students, developers or others previously unfamiliar with performance testing, a series of presentations has been composed, and this paper represents the research conducted in composing them. This research includes a basic level understanding of Visual Studio Team System 2008\u27s test tools, and virtualization with VMWare