A Taxonomy of Quality Metrics for Cloud Services

[EN] A large number of metrics with which to assess the quality of cloud services have been proposed over the last years. However, this knowledge is still dispersed, and stakeholders have little or no guidance when choosing metrics that will be suitable to evaluate their cloud services. The objective of this paper is, therefore, to systematically identify, taxonomically classify, and compare existing quality of service (QoS) metrics in the cloud computing domain. We conducted a systematic literature review of 84 studies selected from a set of 4333 studies that were published from 2006 to November 2018. We specifically identified 470 metric operationalizations that were then classified using a taxonomy, which is also introduced in this paper. The data extracted from the metrics were subsequently analyzed using thematic analysis. The findings indicated that most metrics evaluate quality attributes related to performance efficiency (64%) and that there is a need for metrics that evaluate other characteristics, such as security and compatibility. The majority of the metrics are used during the Operation phase of the cloud services and are applied to the running service. Our results also revealed that metrics for cloud services are still in the early stages of maturity only 10% of the metrics had been empirically validated. The proposed taxonomy can be used by practitioners as a guideline when specifying service level objectives or deciding which metric is best suited to the evaluation of their cloud services, and by researchers as a comprehensive quality framework in which to evaluate their approaches.This work was supported by the Spanish Ministry of Science, Innovation and Universities through the Adapt@Cloud Project under Grant TIN2017-84550-R. The work of Ximena Guerron was supported in part by the Universidad Central del Ecuador (UCE), and in part by the Banco Central del Ecuador.Guerron, X.; Abrahao Gonzales, SM.; Insfran, E.; Fernández-Diego, M.; González-Ladrón-De-Guevara, F. (2020). A Taxonomy of Quality Metrics for Cloud Services. IEEE Access. 8:131461-131498. https://doi.org/10.1109/ACCESS.2020.3009079S131461131498

A Review on Software Quality Forensics: Techniques, Challenges, and Limitations

Software quality forensics plays a vibrant role related to software quality, security, and integrity. The paper aims to derive a software quality forensics model through existing software quality models and their factors. The papers explore quality models, factors, approaches, tools, techniques, and standards regarding software quality investigation and confine the research area for software quality integrity breach forensics. The explore the deviations of quality attributes, standards, factors, and artifacts, it leads to further investigation of root-cause followed by digital evidence procedure for alleged software quality issues. Therefore, there is a need for a software quality forensics model and dedicated standards to fulfill the digital evidence procedure validation, satisfiable, and prosecution in the court of law in the context of alleged or illegal activity investigation quality of software. The paper has  derived the techniques, challenges, and limitations of software quality forensics based on the review of research questions

Quality Properties of Execution Tracing, an Empirical Study

The authors are grateful to all the professionals who participated in the focus groups; moreover, they also express special thanks to the management of the companies involved for making the organisation of the focus groups possible.Data are made available in the appendix including the results of the data coding process.The quality of execution tracing impacts the time to a great extent to locate errors in software components; moreover, execution tracing is the most suitable tool, in the majority of the cases, for doing postmortem analysis of failures in the field. Nevertheless, software product quality models do not adequately consider execution tracing quality at present neither do they define the quality properties of this important entity in an acceptable manner. Defining these quality properties would be the first step towards creating a quality model for execution tracing. The current research fills this gap by identifying and defining the variables, i.e., the quality properties, on the basis of which the quality of execution tracing can be judged. The present study analyses the experiences of software professionals in focus groups at multinational companies, and also scrutinises the literature to elicit the mentioned quality properties. Moreover, the present study also contributes to knowledge with the combination of methods while computing the saturation point for determining the number of the necessary focus groups. Furthermore, to pay special attention to validity, in addition to the the indicators of qualitative research: credibility, transferability, dependability, and confirmability, the authors also considered content, construct, internal and external validity

A process model for quality in use evaluation on clinical decision support systems

Developing or purchasing software is an expensive investment and needs to be justified. Furthermore, the software must be useful in its purpose, reliable, efficient and, among other characteristics, meet the expectations of users [1, 2]. It would be no different in the case of a clinical Decision Support System - CDSS. CDSS are systems developed to support clinicians and other health professionals in a medical decision making [3]. They are developed within a clinical context, following medical guidelines, with varied purposes such as diagnoses [4, 5, 6] patient monitoring [7, 8, 9], prevention [10] and disease treatment [11, 12]. Conversely, even with all the benefits offered by a CDSS, its acceptance in the medical field is still a matter of debate [13, 14]. The CDSS acceptance is linked to the perception of the end user, such as 1) the system’s ease of use and utility, 2) the quality of its results and its reliability [14], 3) the contextual accessibility of the system, sometimes not included in the health professional’s routine and workflow, and 4) the fact that numerous CDSSs are not integrated with existing systems [15]. One manner to extend the use and disseminate positive contributions of CDSSs to the medical world is to develop them in a reliable and useful way. For this, one must follow the best practices of software engineering (SE, acronym in English) [16] and be concerned with its quality, both in the design and development process and in its effective use. Evaluating the quality of the software is to measure its characteristics and sub-characteristics of quality. In order to better structure the assessment, a series of international standards, with models and frameworks, were developed for assisting software developers in assessing the quality of software products. The latest series is the ISO/IEC 25000 - System and Software Quality Requirements and Evaluation (SQuaRE) [17]. Two of the SQuaRE divisions are addressed in this thesis: 1) Division of quality models standard (ISO/IEC 25010) [18], and 2) Quality measurement division standard (ISO/IEC 25022) [19]. The ISO/IEC 25010 are divided in product quality model and the quality model in use. Quality in use (QiU), a model of ISO/IEC 25010, is the focus of this study, through its evaluation in the context of a CDSS. The quality in use model refers to the quality of the software when executed, mentioning the result of the interaction between users and the software system/product in a specific context. This model consists of five quality characteristics: • Effectiveness - means the level of precision and completeness with which users achieve their specific goals when using the system; • Efficiency - refers to the resources spent to achieve the goals and its measure is related to the level of effectiveness achieved with the consumed resources; • Satisfaction - refers to whether user requirements are satisfied in a particular context of system use; • Freedom from risk - refers to the degree to which the quality of a system reduces or avoids potential risks to human life, the economic situation, and health of the environment; • Context coverage - deals with the use of the system in all specific contexts and/or in contexts that extend beyond the initially identified contexts. Context completeness and flexibility are the sub-characteristics that represent context coverage. Thus, when measuring the quality of a CDSS, we must consider both the context of use and the choice of the characteristic and sub-characteristic that best suits the purpose of the measurement [20]. The QiU model provides a powerful contribution to the practice of evaluating a system and determining its quality. According to Harrison et al. [21], Effectiveness, Efficiency and Satisfaction are considered the key criteria to reflect the quality of use. Therefore, these QiU characteristics meet the needs and expectations of the users of the systems, in our case of CDSSs, as they consider the user experience. As a contribution, we proposed a process model to evaluate two QiU characteristics in a CDSS: satisfaction and efficiency. We believe these characteristics are important in the evaluation of a CDSS because, due to its links with the user experience and the usability of the system, when measured, can corroborate the quality of the CDSS and mitigate the non-use and non-acceptance of this type of software. Other contributions from our work are 1) in the academic context, a significant study in the area of software quality, focusing on its characteristics, especially on the quality in use. A guideline for collecting and measuring these characteristics was built into our process model; 2) in the area of software development, professionals can make use of a simple and adaptable process, applicable to other types of systems, to measure the quality-in-use characteristics of their products.Desenvolver ou adquirir software é um investimento caro e precisa ser justificado. Além de útil, o sistema deve ser confiável, eficiente e, entre outras características, atender às expectativas dos usuários [1, 2]. Não seria diferente no caso de um sistema de apoio à decisão clínica (CDSS, acrônimo em inglês), sistemas desenvolvidos para apoiar médicos e outros profissionais de saúde na tomada de uma decisão médica [3]. CDSSs são elaborados dentro de um contexto clínico, seguindo guidelines com propósitos variados, sejam para diagnósticos [4, 5, 6], acompanhamento do paciente [7, 8, 9], na prevenção [10] e tratamento de doenças [11, 12]. No entanto, apesar de todo os benefícios oferecidos por um CDSS, sua aceitação na área médica ainda é motivo de debate [13, 14]. Essa aceitação está ligada à percepção do usuário final, como 1) a facilidade de uso e utilidade do sistema; 2) a qualidade dos resultados produzidos e sua confiabilidade [14]; 3) a acessibilidade contextual do sistema, muitas vezes não incluída na rotina e no fluxo de trabalho do profissional de saúde, e 4) o fato de muitos CDSSs não estarem integrados aos sistemas existentes [15]. Uma forma de estender o uso de CDSSs e disseminar suas contribuições positivas entre os profissionais de saúde é garantir a confiabilidade de seus resultados e a satisfação do usuáriofinal. Para tal deve-se seguir as melhores práticas da engenharia de software (SE, acrônimo em inglês) em sua concepção [16]. Isso implica em preocupar-se com a qualidade do sistema tanto no processo do projeto e desenvolvimento quanto em sua efetiva utilização. Uma forma de certificar se um software obedece a essa premissa é realizando avaliações de qualidade. Avaliar a qualidade do software é medir suas características e subcaracterísticas de qualidade. Para uma melhor estruturação desta medição foram desenvolvidos séries de padrões internacionais como guidelines de avaliação de qualidade de produtos de software. A série mais recente trata-se da ISO/IEC 25000 System and Software Quality Requirements and Evaluation (SQUARE) [17]. Dois padrões desta série foram abordadas nesta tese, sendo 1) o modelos de qualidade de software e sistemas (ISO/IEC 25010) [18], no qual trabalhamos especificamente com o modelo de qualidade em uso, e 2) o padrão de medição da qualidade em uso (ISO/IEC 25022) [19]. Qualidade em uso é o foco desta tese, através de sua avaliação no contexto de utilização de um CDSS. O Modelo de qualidade em uso trata da qualidade do software quando em execução, referindose ao resultado da interação dos usuários e o software em um cenário específico. Este modelo é composto de cinco características de qualidade: • Eficácia (ou efetividade) - esta característica representa o nível de precisão e completude com que os usuários alcançam os objetivos específicos, durante a utilização do sistema ou produto de software; • Eficiência - sua medição representa o nível de eficácia alcançada em relação aos recursos consumidos para o alcance das metas; • Satisfação - trata do quanto as necessidades do usuário são satisfeitas dentro de um determinado contexto de uso do sistema ou produto de software. Esta característica é composta pelas subcaracterísticas Utilidade, Confiança, Prazer e Conforto do usuário em relação ao sistema; • Livre de risco - trata do grau em que a qualidade de um sistema ou produto permite mitigar ou evitar riscos potenciais à vida humana, à situação econômica, à saúde ou ao meio ambiente, sendo estas suas três subcaracterísticas; • Cobertura de contexto - trata do uso do sistema em todos os contextos específicos e/ou em contextos além dos inicialmente identificados, sendo composta pelas subcaracterísticas completude de contexto e flexibilidade do sistema. Assim, para se medir a qualidade de um CDSS deve-se considerar tanto o contexto de utilização quanto a escolha da característica e subcaracterística que melhor condizem ao propósito da avaliação [20]. De acordo com Harrison et al. [21], Eficácia, Eficiência e Satisfação são considerados os principais critérios a serem avaliados para refletir a qualidade de uso. Tais características de qualidades em uso refletem o atendimento das necessidades e expectativas dos usuários dos sistemas, em especial ao usuário primário ou final, uma vez que estão diretamente relacionadas com a experiência do usuário. O modelo de qualidade em uso fornece uma contribuição poderosa para a prática de avaliar um sistema e determinar sua qualidade. Como contribuição, propusemos um modelo de processo para avaliação de qualidade em uso de um CDSS através da medição, a priori, de duas características de qualidade - satisfação e eficiência. Acreditamos que tais características são importantes na avaliação de um CDSS devido estreita relação destas com a experiência do usuário-final e a usabilidade do sistema. Assim, quando mensuradas, tais características podem corroborar com a qualidade do CDSS e mitigar a não utilização e não aceitação desse tipo de software. Nosso modelo proposto é definido por cinco (5) fases, a saber: 1) Identificação de cenário e contexto de uso do sistema, 2) seleção das medidas, métricas e métodos para mensurar as características, 3) a medição da qualidade, 4) a análise dos valores encontrados na medição e 5) a apresentação dos resultados obtidos. O resultado da aplicação do modelo de processo traduz-se em um conjunto de informações que nortearão um melhoramento do software, caso a medição das características fique abaixo de um padrão pré-definido pelos atores envolvidos no processo de medição do sistema. Por outro lado, se a medição for positiva, isso vem ratificar a qualidade do sistema e ações poderão ser tomadas para disseminar esse bom resultado, buscando a adesão de mais utilizadores. Como forma de validação do modelo proposto, após sua utilização para identificação de cenários e contexto-de-uso possíveis de serem mensurados, foi apresentado um CDSS da área oncológica a profissionais de saúde, estudantes de medicina e profissionais da área de qualidade de software que, ao final de sua utilização, responderam a um inquérito com o objetivo de avaliar o sistema. A aplicação se deu de forma online, dado a necessidade de mantermos o distanciamento social e o de cumprirmos as orientações sanitárias. As respostas serviram como fonte de dados para a medição das características de qualidadeem- uso do sistema. Os resultados da aplicação revelou que nosso modelo de processo de avaliação é válido, relevante e de fácil utilização para identificar as características importantes em um sistema, bem como suas medições por meio das funções matemáticas do modelo ISO/IEC 25022. Outras contribuições do nosso trabalho, temos 1) no âmbito acadêmico, um estudo significativo na área de qualidade de software, com foco em suas características, especialmente na qualidade em uso. Uma guideline para a coleta e mensuração dessas características foi construída em nosso modelo de processo; 2) na área de desenvolvimento de software, os profissionais podem contar com um processo simples e adaptável, aplicável a outros tipos de sistema, para mensuração da qualidade em uso de seus produtos.The research has been partially funded by the FCT/MCTES through national funds, and when applicable, co-funded EU funds under the project UIDB/EEA/50008/2020 and Operação Centro 01-0145-FEDER-000019 – C4 – Centro de Competências em Cloud Computing, co-financed by the Programa Operacional Regional do Centro (CENTRO 2020), through the Sistema de Apoio à Investigação Científica e Tecnológica – Programas Integrados de IC&DT. I would also like to acknowledge the contribution of the COST Action IC1303: AAPELE—Archi- tectures, Algorithms and Protocols for Enhanced Living Environments and COST Action CA16226; SHELD-ON—Indoor living space improvement: Smart Habitat for the Elderly, supported by COST (European Cooperation in Science and Technology)

Monitoring the quality of service to support the service based system lifecycle

Service Oriented Computing (SOC) has been established in the last recent years as a successful paradigm in Software Engineering. The systems built under this paradigm, known as Service Based System (SBS), are composed of several services, which are usually third-party software run by external service providers. SBS rely on these service providers to ensure that their services comply with the agreed Quality of Service (QoS). In contrast to other systems, the dynamic behaviour of SBS requires up-to-date QoS information for its proper management in the different stages of its lifecycle, from their initial construction until their decommission. Providing such QoS information has resulted in different technological solutions built around a monitor. Nonetheless, several research challenges in the field remain still open, ranging from theoretical aspects of quality assurance to architectonical challenges in decentralized monitoring. Based on the current research challenges for service monitoring, the research gaps in which we aim to contribute are twofold: - To investigate on the definition and structure of the different quality factors of services, and provide a framework of common understanding for the definition of what to monitor. - To investigate on the different features required to support the activities of the whole SBS lifecycle (i.e. how to monitor), and develop a monitoring framework that accomplishes such features. As a result of this thesis, we provide: What to monitor - A distribution of the quality models along the time dimension and the identification of their relationships. - An analysis of the size and definition coverage of the proposed quality models. - A quantified coverage of the different ISO/IEC 25010 quality factors given by the proposals. - The identification of the most used quality factors, and provided the most consolidated definitions for them. How to monitor - The elicitation of the requirements of the different activities in the SBS lifecycle. - The definition of the set of features that supports the elicited requirements. - A modular service-oriented monitoring framework, named SALMon, implementing the defined features. SALMon has been validated by including it in several frameworks supporting the different activities of the SBS lifecycle. Finally, we have conducted a performance evaluation of SALMon over real web services.La Computació Orientada a Serveis (SOC) ha esdevingut en els darrers anys un paradigma exitós en el camp de l'Enginyeria del Software. Els sistemes construïts sota aquest paradigma, coneguts com Sistemes Basats en Serveis (SBS), estan composats de diversos serveis, que són, usualment, programari de tercers executats per proveïdors de serveis externs. Els SBS depenen dels proveïdors dels serveis per garantir que els serveis compleixen amb la Qualitat del Servei (QoS) acordada. En contrast amb altres sistemes, el comportament dinàmic dels SBS requereix d'informació actualitzada del QoS per a la correcta administració de les diferents etapes del cicle de vida dels SBS: des de la seva construcció inicial fins a la seva clausura. Proveir d'aquesta informació de QoS ha resultat en diferents solucions tecnològiques construïdes al voltant d'un monitor. Malgrat això, diversos reptes de recerca en el camp encara romanen obertes, des d'aspectes teòrics de l'assegurança de qualitat, a reptes arquitectònics en la monitorització descentralitzada. Basat en els reptes de recerca actuals per a la monitorització de serveis, els forats de recerca en els que pretenem contribuir són dobles: - Investigar en la definició i estructura dels diferents factors de qualitat dels serveis, i proveir un marc de treball d'entesa comuna per a la definició de què monitoritzar. - Investigar en les diferents característiques requerides per donar suport a les activitats de tot el cicle de vida dels SBS (i.e. com monitoritzar), i desenvolupar una plataforma de monitorització que acompleixi aquestes característiques. Com a resultats de la tesis, proveïm: Què monitoritzar - Una distribució dels models de qualitat al llarg de la dimensió temporal i la identificació de les seves interrelacions. - Un anàlisi de la mida i definició de la cobertura dels models de qualitat proposats. - Una cobertura quantificada dels diferents factors de qualitat ISO/IEC 25010 donat en les diferents propostes. - La identificació dels factors de qualitat més utilitzats, i la definició dels termes més consolidats. Com monitoritzar - L'elicitació dels requeriments per a les diferents activitats en el cicle de vida dels SBS. - La definició del conjunt de característiques que donen suport als requeriments elicitats. - Una platforma modular orientada a serveis, anomenat SALMon, que implementa les característiques definides. SALMon ha estatvalidat incloent la plataforma en diversos marcs de treball donant suport a les diferents activitat

Towards a monitoring middleware for cloud services

© 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Cloud Computing represents a new trend in the development and use of software. Many organizations are currently adopting the use of services that are hosted in the cloud by employing the Software as a Service (SaaS) model. Services are typically accompanied by a Service Level Agreement (SLA), which defines the quality terms that a provider offers to its customers. Many monitoring tools have been proposed to report compliance with the SLA. However, they have some limitations when changes to monitoring requirements must be made and because of the complexity involved in capturing low-level raw data from services at runtime. In this paper, we propose the design of a platform-independent monitoring middleware for cloud services, which supports the monitoring of SLA compliance and provides a report containing SLA violations that may help stakeholders to make decisions regarding how to improve the quality of cloud services. Moreover, our middleware definition is based on the use of [email protected], which allows the dynamic change of quality requirements and/or the dynamic selection of different metric operationalizations (i.e., calculation formulas) with which to measure the quality of services. In order to demonstrate the feasibility of our approach, we show the instantiation of the proposed middleware that can be used to monitor services when deployed on the Microsoft Azure© platform.This research is supported by the Value@Cloud project (TIN2013-46300-R); the Scholarship Program Senescyt - Ecuador; University of Cuenca – Ecuador; and the Microsoft Azure for Research Award ProgramCedillo Orellana, IP.; Jiménez Gómez, J.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2015). Towards a monitoring middleware for cloud services. IEEE. https://doi.org/10.1109/SCC.2015.68