A Systematic Mapping Study of Empirical Studies on Software Cloud Testing Methods

Context: Software has become more complicated, dynamic, and asynchronous than ever, making testing more challenging. With the increasing interest in the development of cloud computing, and increasing demand for cloud-based services, it has become essential to systematically review the research in the area of software testing in the context of cloud environments. Objective: The purpose of this systematic mapping study is to provide an overview of the empirical research in the area of software cloud-based testing, in order to build a classification scheme. We investigate functional and non-functional testing methods, the application of these methods, and the purpose of testing using these methods. Method: We searched for electronically available papers in order to find relevant literature and to extract and analyze data about the methods used. Result: We identified 69 primary studies reported in 75 research papers published in academic journals, conferences, and edited books. Conclusion: We found that only a minority of the studies combine rigorous statistical analysis with quantitative results. The majority of the considered studies present early results, using a single experiment to evaluate their proposed solution

Modelo de Calidad para Servicios Cloud

[EN] Modelo de Calidad para Servicios Cloud 4 Abstract Context: Cloud computing is a model of provision and consumption of services that offers many advantages to companies (high availability, flexibility, maximum utilization of resources, etc.) that result in quality requirements that must be met by the servi ce. In recent years there have been proposed numerous quality attributes and metrics for cloud services, but there is no study to collect this information and classified it with respect to internal and external characteristics of service (Quality of Servic e - QoS) and characteristics in use of the service (Quality of Experience - QoE). Objective: The objective of this final master ’s work is to define a model specific quality cloud services aligned with the ISO / IEC 25010, which integrate the quality feature s, attributes and metrics proposed in the literature and which allow to assess the quality cloud of artifacts in various stages of the life cycle. Method: We performed a systematic review of the literature in order to identify and analyze the attributes an d quality metrics proposed to assess the quality of cloud services. This method has been widely used in the field of Software Engineering and has proven useful to collect and analyze existing information on a particular research topic. Results: The result is a quality model for cloud services which has been built from 178 attributes and 364 metric obtained as a result of the systematic review. In particular, the results of the review indicate that 48% of proposals are metrics to measure performance efficien cy, reliability metrics following him with 23%. With respect to the phase of the life cycle, 55% of these metrics are used in the operation phase and 32% in the acquisition phase. Regarding the point of view of stakeholders, 39% of the metrics are oriented to the service provider, 33% consumer, 7% to the facilitator (broker) and only 5% service developer. With respect to cloud evaluated artifacts, most metrics (97%) are applied to the cloud service being tested or deployed in the cloud; only 2% of the metri cs are applied to the service architecture and 1% on the service specification. With regard to the validation, the results show that 99% of metric proposals lack any type of validation, although 44% presents a proof of concept illustrating how the metrics can be used. Additionally, we identified 27 attributes for cloud services, the elasticity was the most named, with 14%. Conclusions: The results of this work have provided relevant information on the current status and gaps that exist in the field of quali ty assessment of cloud services. They have also allowed us to define a quality model to meet some identified shortcomings. As future work, we intend to refine the proposed model, propose new metrics and adapt some existing architectures for evaluating clou d and empirical studies to provide evidence about theusefulness of a set of metrics[ES] El trabajo consiste en definir un modelo de calidad que determine las características de los servicios cloud y proporcione los mecanismos necesarios para evaluar su calidad. Se realizará una revisión sistemática de la literatura con el objetivo de identificar un conjunto de atributos de calidad, métricas e indicadores que permitirán medir las características identificadas. Como resultado se obtendrá un modelo de calidad alineado con la ISO/IEC 25010 que integrará las características de calidad, atributos, métricas e indicadores que dan soporte a su evaluación.[CA] Context: La computacio en el nuvol es un model de prestacio i consum de servicis que oferix moltes ventages a les empreses (alta disponibilitat, elasticitat, maxim aprofitament de recursos, etc.) que se traduixen en requisits de calitat que deuen ser complits pel servici. En els ultims anys s'han propost numerosos atributs de calitat i metriques per a servicis cloud, pero no existix un estudi que recoja esta informacio i la classifique en respecte a les caracteristiques internes i externes del servici (Quality of Service – QoS), aixina com caracteristiques en us del servici (Quality of Experience – QoE). Objectiu: L'objectiu d'este treball de fi de máster es definir un model de calitat especifica per a servicis cloud, enringlerat en l'ISO/IEC 25010, que integre les caracteristiques de calitat, atributs i metriques proposts en la lliteratura i que permeten evaluar la calitat dels artefactes cloud en distintes fases del cicle de vida. Metodo: S'ha realisat una revisio sistematica de la lliteratura en l'objectiu d'identificar i analisar els atributs i metriques de calitat propostes per a evaluar la calitat dels servicis cloud. Este metodo ha segut utilisat extensament en l'ambit de l'Ingenieria del Software i ha demostrat ser util per a recopilar i analisar l'informacio existent relativa a un determinat tema d'investigacio. Resultats: El resultat es un model de calitat per a servicis cloud que ha segut construit a partir dels 178 atributs i 364 metriques obtingudes com resultat de la revisio sistematica. En particular, els resultats de la revisio indiquen que el 48% de les metriques propostes son per a mesurar Eficiencia de desempenyorament, seguint-li les metriques de fiabilidad en en un 23%. En respecte a la fase del cicle de vida, un 55% d'estes metriques s'utilisen en la fase d'Operacio i un 32% en la fase d'Adquisicio. En respecte al punt de vista dels stakeholders, el 39% de les metriques estan orientades al proveïdor del servici, el 33% al consumidor, el 7% al facilitador (bróker) i nomes un 5% al desarrollador del servici. En respecte als artefactes cloud valorats, la majoria de les metriques (97%) s'apliquen sobre el servici cloud en fase de proves o desplegat en el cloud; nomes un 2% de les metriques s'apliquen sobre l'arquitectura del servici i un 1% sobre l'especificacio del servici. En respecte a la validacio, els resultats mostren que el 99% de les metriques propostes carixen de qualsevol tipo de validacio, encara que el 44% presenta una prova de concepte que ilustra com se pot utilisar les metriques. Adicionalment identifiquem 27 atributs propis dels servicis cloud, sent l'elasticitat el mes nomenat, en 14%. Conclusions: Els resultats del treball han proporcionat informacio rellevant sobre l'estat actual i les carencies que existixen en l'ambit de l'evaluacio de la calitat dels servicis cloud. Tambe mos han permes definir un model de calitat per a suplir algunes carencies identificades. Com trabajos futurs, pretenem refinar el model propost, propondre noves metriques i adaptar algunes existents per a l'evaluacio d'arquitectures cloud, aixina com realisar estudios empirics per a proporcionar evidencia al voltant de l'utilitat d'un conjunt de metriquesNavas Rosales, RM. (2016). Modelo de Calidad para Servicios Cloud. http://hdl.handle.net/10251/77847TFG

Scalability performance measurement and testing of cloud-based software services

Cloud-based software services have become more popular and dependable and are ideal for businesses with growing or changing workload demands. These services are increasing rapidly due to the reduced hosting costs and the increased availability and efficiency of computing resources. The delivery of cloud-based software services is based on the underlying cloud infrastructure supported by cloud providers, which delivers the potential for scalability that follows the pay-as-you-go model. Performance and scalability testing and measurements of those services are necessary for future optimisations and growth of cloud computing to support the Service Level Agreement (SLA) compliant quality of cloud services, especially in the context of rapidly expanding quantity of service delivery. This thesis addresses an important issue, understanding the scalability of cloud-based software services from a technical perspective, which is very important as more software solutions are migrated to the cloud. A novel testing and quantifying approach for the scalability performance of cloud-based software services is described. Two technical scalability metrics for software services that have been deployed and distributed in cloud environments, have been formulated: volume and quality scalability metrics based on the number of software instances and the average response time. The experimental analysis comprises three stages. The first stage involves demonstrating the approach and the metrics using real-world could-based software service running on Amazon EC2 cloud using three demand scenarios. The second stage aims to extend the practicality of the metrics with experiments on two public cloud environments (Amazon EC2 and Microsoft Azure) with two cloud-based software serices to demonstrate the use of these metrics. The experimental analysis considers three sets of comparisons to provide the platform to construct the metrics as a basis that can be used effectively to compare the scalability of software on cloud environments, consequently supporting deployment decisions with technical arguments. Moreover, the work integrates the technical scalability metrics with an earlier utility-oriented scalability metric. The third stage is a case study of application-level fault inection using real-world cloud-based software services running on Amazon EC2 cloud to demonstrate the effect of fault scenarios on the scalability behaviour. The results show that the technical metrics quantify explicitly the technical scalability performance of the cloud-based software services, and that they allow clear assessment of the impact of demand scenarios, cloud platform and fault injection on the software services’ scalability behaviour. The studies undertaken in this thesis have provided a valuable insight into the scalability of cloud-based software services delivery