Self-Checked Metamorphic Testing of Monte Carlo Simulation

Photon propagation in biological tissues can be modeled with Monte Carlo simulations numerically. However, testing a such program is difficult due to the unknown character of the test oracles. Although approaches based on Beer-Lambert law, van de Hulst's table or Radiative Transfer Equation (RTE) can be used for testing the Monte Carlo modeling programs, these approaches are only applied to the programs that are designed for homogeneous media. A rigorous way for testing the Monte Carlo modeling programs for heterogeneous media is needed.   Metamorphic testing, as an effective approach for testing systems that do not have test oracles, is one of possible supplementary approaches to test a Monte Carlo modeling program for heterogeneous media. In metamorphic testing, instead of verifying the correctness of a test output, the satisfaction of a metamorphic relation of the test outputs is checked. If a violation of the metamorphic relation is found, the system implementation must have some faults. However, checking only the metamorphic relations is not good enough to ensure the testing quality. Randomly or accidently generated incorrect outputs may satisfy a metamorphic relation as well. Therefore, it is necessary to provide a systematic approach to measure the test effectiveness of a metamorphic testing, to choose metamorphic relations, and to generate test input data.   In this thesis, we propose a new approach called self-checked metamorphic testing. In our new approach, the original metamorphic testing is extended with the evaluation of the adequacy of testing coverage criteria to measure the quality of a metamorphic testing, to guide the creation of metamorphic relations, to generate testing inputs, and to investigate the found exceptions. The effectiveness of this approach has been demonstrated through testing a parallel Monte Carlo modeling program we developed for simulating photon propagation in human skins.   This thesis contains three parts of work. In first part, the enhanced Monte Carlo modeling program was used to preliminarily study the relationship between the height of the collection lens and the contrast values of the reflectance image of the system. In second part, the homogenous part of the Monte Carlo program was validated with van de Hulst's table method, which compares the simulation results with the calculated values on van de Hulst's table. The third and the main part of the thesis is applying the self-checked metamorphic testing approach to test the Monte Carlo modeling program.  M.S

Towards Automated Metamorphic Test Identification for Ocean System Models

Metamorphic testing seeks to verify software in the absence of test oracles. Our application domain is ocean system modeling, where test oracles rarely exist, but where symmetries of the simulated physical systems are known. The input data set is large owing to the requirements of the application domain. This paper presents work in progress for the automated generation of metamorphic test scenarios using machine learning. We extended our previously proposed method [1] to identify metamorphic relations with reduced computational complexity. Initially, we represent metamorphic relations as identity maps. We construct a cost function that minimizes for identifying a metamorphic relation orthogonal to previously found metamorphic relations and penalize for the identity map. A machine learning algorithm is used to identify all possible metamorphic relations minimizing the defined cost function. We propose applying dimensionality reduction techniques to identify attributes in the input which have high variance among the identified metamorphic relations. We apply mutation on these selected attributes to identify distinct metamorphic relations with reduced computational complexity. For experimental evaluation, we subject the two implementations of an ocean-modeling application to the proposed method to present the use of metamorphic relations to test the two implementations of this application.Comment: 5 Pages, 1 Figur

The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study

Context: Machine learning (ML) may enable effective automated test generation. Objective: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges. Methods: We perform a systematic mapping on a sample of 102 publications. Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning - often based on neural networks - and reinforcement learning - often based on Q-learning - are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion: Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed - and how they are applied - benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.Comment: Under submission to Software Testing, Verification, and Reliability journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an earlier study that this study extends

Pressure-temperature-time constraints for exhumation of ultrahigh-pressure metamorphic rocks, North Qaidam terrane, Western China

Ultrahigh-pressure rocks of a Paleozoic continental suture zone are exposed in the southeastern North Qaidam terrane (Dulan region). Garnetite sample D119 contains minor Czo+Qtz+Chl+Ttn; rutile inclusions in titanite suggest that titanite replaced rutile during decompression. Pressure-temperature estimates of sample D119 are 16.1-18.4 kbar and 485-520 °C. Sample D130B is a mafic band in calc-silicate gneiss, with garnet porphyroblasts in a fine-grained Hbl-Cpx-Pl-Qtz symplectite that is interpreted as former omphacite. D130B symplectite pressure-temperature estimates are 7.7-9.4 kbar and 623-708 °C. Titanite U-Pb ages of 419.7±3.1 Ma and 415.9±4.2 Ma are interpreted to date retrogression of D119 and D130B, respectively. Average exhumation and cooling rates from ultrahigh-pressure to retrograde conditions are 6.4±6.0 mm/yr and 30±27 °C/Ma (D119) and 7.9±6.1 mm/yr and 3.9±8.0 °C/Ma (D130B). These data are compatible with early, rapid exhumation of a coherent block large enough to experience nearly-isothermal decompression in its insulated interior (D130B), while colder crust conductively refrigerates the block\u27s margins (D119)

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles

It is challenging to test applications and functions for which the correct output for arbitrary input cannot be known in advance, e.g. some computational science or machine learning applications. In the absence of a test oracle, one approach to testing these applications is to use metamorphic testing: existing test case input is modified to produce new test cases in such a manner that, when given the new input, the application should produce an output that can be easily be computed based on the original output. That is, if input x produces output f(x), then we create input x' such that we can predict f(x') based on f(x); if the application or function does not produce the expected output, then a defect must exist, and either f(x) or f(x') (or both) is wrong. By using metamorphic testing, we are able to provide built-in 'pseudo-oracles' for these so-called 'nontestable programs' that have no test oracles. In this paper, we describe an approach in which a function's metamorphic properties are specified using an extension to the Java Modeling Language (JML), a behavioral interface specification language that is used to support the 'design by contract' paradigm in Java applications. Our implementation, called Corduroy, pre-processes these specifications and generates test code that can be executed using JML runtime assertion checking, for ensuring that the specifications hold during program execution. In addition to presenting our approach and implementation, we also describe our findings from case studies in which we apply our technique to applications without test oracles

Mapping the Structure and Evolution of Software Testing Research Over the Past Three Decades

Background: The field of software testing is growing and rapidly-evolving. Aims: Based on keywords assigned to publications, we seek to identify predominant research topics and understand how they are connected and have evolved. Method: We apply co-word analysis to map the topology of testing research as a network where author-assigned keywords are connected by edges indicating co-occurrence in publications. Keywords are clustered based on edge density and frequency of connection. We examine the most popular keywords, summarize clusters into high-level research topics, examine how topics connect, and examine how the field is changing. Results: Testing research can be divided into 16 high-level topics and 18 subtopics. Creation guidance, automated test generation, evolution and maintenance, and test oracles have particularly strong connections to other topics, highlighting their multidisciplinary nature. Emerging keywords relate to web and mobile apps, machine learning, energy consumption, automated program repair and test generation, while emerging connections have formed between web apps, test oracles, and machine learning with many topics. Random and requirements-based testing show potential decline. Conclusions: Our observations, advice, and map data offer a deeper understanding of the field and inspiration regarding challenges and connections to explore.Comment: To appear, Journal of Systems and Softwar

A computer code for forward calculation and inversion of the H/V spectral ratio under the diffuse field assumption

During a quarter of a century, the main characteristics of the horizontal-to-vertical spectral ratio of ambient noise HVSRN have been extensively used for site effect assessment. In spite of the uncertainties about the optimum theoretical model to describe these observations, several schemes for inversion of the full HVSRN curve for near surface surveying have been developed over the last decade. In this work, a computer code for forward calculation of H/V spectra based on the diffuse field assumption (DFA) is presented and tested.It takes advantage of the recently stated connection between the HVSRN and the elastodynamic Green's function which arises from the ambient noise interferometry theory. The algorithm allows for (1) a natural calculation of the Green's functions imaginary parts by using suitable contour integrals in the complex wavenumber plane, and (2) separate calculation of the contributions of Rayleigh, Love, P-SV and SH waves as well. The stability of the algorithm at high frequencies is preserved by means of an adaptation of the Wang's orthonormalization method to the calculation of dispersion curves, surface-waves medium responses and contributions of body waves. This code has been combined with a variety of inversion methods to make up a powerful tool for passive seismic surveying.Comment: Published in Computers & Geosciences 97, 67-7

Control-Theoretical Perspective in Feedback-Based Systems Testing

Self-Adaptive Systems (SAS) and Cyber-Physical Systems (CPS) have received significant attention in recent computer engineering research. This is due to their ability to improve the level of autonomy of engineering artefacts. In both cases, this autonomy increase is achieved through feedback. Feedback is the iteration of sens- ing and actuation to respectively acquire knowledge about the current state of said artefacts and steer them toward a desired state or behaviour. In this thesis we dis- cuss the challenges that the introduction of feedback poses on the verification and validation process for such systems, more specifically, on their testing. We highlight three types of new challenges with respect to traditional software testing: alteration of testing input and output definition, and intertwining of components with different nature. Said challenges affect the ways we can define different elements of the test- ing process: coverage criteria, testing set-ups, test-case generation strategies, and oracles in the testing process. This thesis consists of a collection of three papers and contributes to the definition of each of the mentioned testing elements. In terms of coverage criteria for SAS, Paper I proposes the casting of the testing problem, to a semi-infinite optimisation problem. This allows to leverage the Scenario Theory from the field of robust control, and provide a worst-case probabilistic bound on a given performance metric of the system under test. For what concerns the definition of testing set-ups for control-based CPS, Paper II investigates the implications of the use of different abstractions (i.e., the use of implemented or emulated compo- nents) on the significance of the testing. The paper provides evidence that confutes the common assumption present in previous literature on the existence of a hierar- chy among commonly used testing set-ups. Finally, regarding the test-case gener- ation and oracle definition, Paper III defines the problem of stress testing control- based CPS software. We contribute to the generation and identification of stress test cases for such software by proposing a novel test case parametrisation. Leveraging the proposed parametrisation we define metamorphic relations on the expected be- haviour of the system under test. We use said relations for the development of stress testing approach and sanity checks on the testing results

HIRESSS: a physically based slope stability simulator for HPC applications

HIRESSS (<b>HI</b>gh <b>RE</b>solution <b>S</b>lope <b>S</b>tability <b>S</b>imulator) is a physically based distributed slope stability simulator for analyzing shallow landslide triggering conditions in real time and on large areas using parallel computational techniques. The physical model proposed is composed of two parts: hydrological and geotechnical. The hydrological model receives the rainfall data as dynamical input and provides the pressure head as perturbation to the geotechnical stability model that computes the factor of safety (FS) in probabilistic terms. The hydrological model is based on an analytical solution of an approximated form of the Richards equation under the wet condition hypothesis and it is introduced as a modeled form of hydraulic diffusivity to improve the hydrological response. The geotechnical stability model is based on an infinite slope model that takes into account the unsaturated soil condition. During the slope stability analysis the proposed model takes into account the increase in strength and cohesion due to matric suction in unsaturated soil, where the pressure head is negative. Moreover, the soil mass variation on partially saturated soil caused by water infiltration is modeled. <br><br> The model is then inserted into a Monte Carlo simulation, to manage the typical uncertainty in the values of the input geotechnical and hydrological parameters, which is a common weak point of deterministic models. The Monte Carlo simulation manages a probability distribution of input parameters providing results in terms of slope failure probability. The developed software uses the computational power offered by multicore and multiprocessor hardware, from modern workstations to supercomputing facilities (HPC), to achieve the simulation in reasonable runtimes, compatible with civil protection real time monitoring. <br><br> A first test of HIRESSS in three different areas is presented to evaluate the reliability of the results and the runtime performance on large areas