DQSOps: Data Quality Scoring Operations Framework for Data-Driven Applications

Data quality assessment has become a prominent component in the successful execution of complex data-driven artificial intelligence (AI) software systems. In practice, real-world applications generate huge volumes of data at speeds. These data streams require analysis and preprocessing before being permanently stored or used in a learning task. Therefore, significant attention has been paid to the systematic management and construction of high-quality datasets. Nevertheless, managing voluminous and high-velocity data streams is usually performed manually (i.e. offline), making it an impractical strategy in production environments. To address this challenge, DataOps has emerged to achieve life-cycle automation of data processes using DevOps principles. However, determining the data quality based on a fitness scale constitutes a complex task within the framework of DataOps. This paper presents a novel Data Quality Scoring Operations (DQSOps) framework that yields a quality score for production data in DataOps workflows. The framework incorporates two scoring approaches, an ML prediction-based approach that predicts the data quality score and a standard-based approach that periodically produces the ground-truth scores based on assessing several data quality dimensions. We deploy the DQSOps framework in a real-world industrial use case. The results show that DQSOps achieves significant computational speedup rates compared to the conventional approach of data quality scoring while maintaining high prediction performance.Comment: 10 Pages The International Conference on Evaluation and Assessment in Software Engineering (EASE) conferenc

An information theoretic notion of software testability

CONTEXT: In software testing, Failed Error Propagation (FEP) is the situation in which a faulty program state occurs during the execution of the system under test (SUT) but this does not lead to incorrect output. It is known that FEP can adversely affect software testing and this has resulted in associated information theoretic measures. OBJECTIVE: To devise measures that can be used to assess the testability of the SUT. By testability, we mean how likely it is that a faulty program state, that occurs during testing, will lead to incorrect output. Previous work has considered a single program point rather than an entire program. METHOD: New, more fine-grained, measures were devised. Experiments were used to evaluate these and the previously defined measures (Squeeziness and Normalised Squeeziness). The experiments assessed how well these measures correlated with an estimate of the probability of FEP occurring during testing. Mutants were used to estimate this probability. RESULTS: A strong rank correlation was found between several of the measures and the probability of FEP. Importantly, this included the Normalised Squeeziness of the whole SUT, which is simpler to compute, or estimate, than most of the other measures considered. Additional experiments found that the measures were relatively insensitive to the choice of mutants and also test suite. CONCLUSION: There is scope to use information theoretic measures to estimate how prone an SUT is to FEP. As a result, there is potential to use such measures to prioritise testing or estimate how much testing an SUT might require

Mittaustiedon hallintajärjestelmän laadunvarmistuksen parantaminen käyttäen ajonaikaista verifiointia

This thesis investigates business process oriented automatic testing for meter data management system. The purpose is to improve the quality assurance process for GENERIS meter data management system. The thesis identifies the most important processes of GENERIS meter data management system based on laws, regulations and guidelines. The most important of the identified processes are described as business process modelling notation diagrams that can be used for test design. The thesis investigates feasibility of a new Quality Manager testing framework. The feasibility is analyzed by implementing a test case for market messaging process using the new framework. In addition, the feasibility of a virtual time management functionality for testing is analyzed. The new framework is also compared to an existing test automation tool. The effects of the new framework on a general software quality assurance process are analyzed. Case examples how the implemented test case improves the quality of the system are also presented. It is established that the new framework is feasible for testing even though the test development consumes significantly more resources than with the old tool. On the other hand, the test scripts developed with the new framework require less maintenance and are more versatile.Tässä opinnäytetyössä tutkitaan liiketoimintaprosessilähtöistä mittaustiedon hallintajärjestelmän automaattista testausta. Työn tarkoituksena on parantaa GENERISmittaustiedonhallintajärjestelmän laadunvarmistusprosessia. Työssä määritellään GENERIS-mittaustiedon hallintajärjestelmän tärkeimmän prosessit lakien, asetusten ja ohjeiden perusteella. Tärkeimmät tunnistetut prosessit on kuvattu bisnesprosessien mallinnuskaavioilla, joita voidaan käyttää testisuunnittelun pohjana. Työssä tutkitaan uuden Quality Manager -testikehyksen soveltuvuutta. Soveltuvuutta tutkitaan toteuttamalla automaattinen testitapaus markkinaviestinnän prosessille. Lisäksi analysoidaan testikehyksessä olevan virtuaalisen ajan hallinnan soveltuvuutta testaukseen. Uutta testikehystä verrataan myös soveltuvin osin vanhempaan testiautomaatiotyökaluun. Uuden testikehyksen vaikutuksia yleiseen laadunvarmistusprosessiin analysoidaan. Lisäksi esitetään konkreettisia esimerkkejä, kuinka kehitetty testitapaus parantaa tuotteen laatua. Testikehys näyttää olevan käyttökelpoinen työkalu, joskin testin kehittäminen vaatii huomattavasti enemmän resursseja vanhaan työkaluun verrattuna. Toisaalta uudella kehyksellä toteutetut testit vaativat vähemmän ylläpitoa ja ovat monipuolisempia

Oracle Assessment, Improvement and Placement

The oracle problem remains one of the key challenges in software testing, for which little automated support has been developed so far. This thesis analyses the prevalence of failed error propagation in programs with real faults to address the oracle placement problem and introduces an approach for iterative assessment and improvement of the oracles. To analyse failed error propagation in programs with real faults, we have conducted an empirical study, considering Defects4J, a benchmark of Java programs, of which we used all 6 projects available, 384 real bugs and 528 methods fixed to correct such bugs. The results indicate that the prevalence of failed error propagation is negligible. Moreover, the results on real faults differ from the results on mutants, indicating that if failed error propagation is taken into account, mutants are not a good surrogate of real faults. When measuring failed error propagation, for each method we use the strongest possible oracle as postcondition, which checks all externally observable program variables. The low prevalence of failed error propagation is caused by the presence of such a strong oracle, which usually is not available in practice. Therefore, there is a need for a technique to assess and improve existing weaker oracles. We propose a technique for assessing and improving test oracles, which necessarily places the human tester in the loop and is based on reducing the incidence of both false positives and false negatives. A proof showing that this approach results in an increase in the mutual information between the actual and perfect oracles is provided. The application of the approach to five real-world subjects shows that the fault detection rate of the oracles after improvement increases, on average, by 48.6%. The further evaluation with 39 participants assessed the ability of humans to detect false positives and false negatives manually, without any tool support. The correct classification rate achieved by humans in this case is poor (29%) indicating how helpful our automated approach can be for developers. The comparison of humans’ ability to improve oracles with and without the tool in a study with 29 other participants also empirically validates the effectiveness of the approach

Symbolic Programming of Distributed Cyber-Physical Systems

Cyber-Physical Systems (CPSs) tightly integrate physical world phenomena and cyber aspects of computational units. The composition of physical, computational and communication systems demands different levels and types of abstraction as well as novel programming methodologies allowing for homogeneous programming, knowledge representation and exchange on heterogeneous devices. Current modeling approaches, frameworks and architectures result fairly inadequate to the task, especially when resource-constrained devices are involved. This work proposes symbolic computation as an effective solution to program resource constrained CPS devices with code maintaining strict ties to high-level specifications expressed in natural language while supporting interoperability among heterogeneous devices. Design, architectural, programming, and deployment aspects of CPSs are addressed through a single formalism unifying the specification of both cyber and physical parts of CPSs. In particular, programming patterns are modeled as sequences of words adhering to natural language syntax and semantics. Given a software under test (SUT), i.e. an input program expressed as a natural language sentence, formal specifications are used to generate oracles for sentence verification and to generate input test cases. The choice of natural language inspired programming supplies a mechanism for the development of the same software on different hardware platforms, ensuring interoperability among heterogeneous devices. Formal specifications also permit to generate stress tests in order to verify that program components behave as expected in repeated execution. In order to make high-level symbolic programs run on real hardware devices with no loss of expressivity during the translation of high-level specifications into an executable implementation, this work proposes a novel software architecture, Distributed Computing for Constrained Devices (DC4CD), as a supporting platform. The proposed architecture enables symbolic processing and distributed computing on devices with very limited energy, communication and processing capabilities that can be integrated into CPSs. In particular, DC4CD has been extensively used to test the symbolic distributed programming methodology on Wireless Sensor Networks (WSNs) that include nodes with actuation abilities. The platform offers networking abstractions for the exchange of symbolic code among peer devices and allows designers to change at runtime, even wirelessly on deployed nodes, not only the application code but also system code.Cyber-Physical Systems (CPSs) tightly integrate physical world phenomena and cyber aspects of computational units. The composition of physical, computational and communication systems demands different levels and types of abstraction as well as novel programming methodologies allowing for homogeneous programming, knowledge representation and exchange on heterogeneous devices. Current modeling approaches, frameworks and architectures result fairly inadequate to the task, especially when resource-constrained devices are involved. This work proposes symbolic computation as an effective solution to program resource constrained CPS devices with code maintaining strict ties to high-level specifications expressed in natural language while supporting interoperability among heterogeneous devices. Design, architectural, programming, and deployment aspects of CPSs are addressed through a single formalism unifying the specification of both cyber and physical parts of CPSs. In particular, programming patterns are modeled as sequences of words adhering to natural language syntax and semantics. Given a software under test (SUT), i.e. an input program expressed as a natural language sentence, formal specifications are used to generate oracles for sentence verification and to generate input test cases. The choice of natural language inspired programming supplies a mechanism for the development of the same software on different hardware platforms, ensuring interoperability among heterogeneous devices. Formal specifications also permit to generate stress tests in order to verify that program components behave as expected in repeated execution. In order to make high-level symbolic programs run on real hardware devices with no loss of expressivity during the translation of high-level specifications into an executable implementation, this work proposes a novel software architecture, Distributed Computing for Constrained Devices (DC4CD), as a supporting platform. The proposed architecture enables symbolic processing and distributed computing on devices with very limited energy, communication and processing capabilities that can be integrated into CPSs. In particular, DC4CD has been extensively used to test the symbolic distributed programming methodology on Wireless Sensor Networks (WSNs) that include nodes with actuation abilities. The platform offers networking abstractions for the exchange of symbolic code among peer devices and allows designers to change at runtime, even wirelessly on deployed nodes, not only the application code but also system code