Metamorphic Relation Patterns for Query-Based Systems

Searching and displaying data based on user queries is a key feature of most software applications such as information systems, web portals, web APIs, and data analytic platforms. The large volume of data managed by these types of systems, henceforth called query-based systems (QBS), makes them extremely hard to test due to the difficulty to assess whether the output of a query is correct, the so-called oracle problem. Metamorphic testing has proved to be a very effective approach to alleviate the oracle problem in QBS, enabling the detection of bugs in data repositories, large e-commerce sites, and some of the most used software applications worldwide such as Google Search and YouTube. We have observed, however, that the metamorphic relations used to test different types of QBS are very similar, regardless of their domain, since all of them exploit standard query features such as filtering and ordering. Inspired by this finding, in this paper we present a catalogue of metamorphic relation patterns to assist testers in the identification and inference of metamorphic relations in QBS. For the definition of the patterns we resorted to the root of most query languages: relational algebra. We show how the proposed patterns help in the identification of metamorphic relations in the e-commerce platform PrestaShop, the email service Gmail, and the mobile application of video streaming HBO.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-

Metamorphic Testing of RESTful Web APIs

Web Application Programming Interfaces (APIs) allow systems to interact with each other over the network. Modern Web APIs often adhere to the REST architectural style, being referred to as RESTful Web APIs. RESTful Web APIs are decomposed into multiple resources (e.g., a video in the YouTube API) that clients can manipulate through HTTP interactions. Testing Web APIs is critical but challenging due to the difficulty to assess the correctness of API responses, i.e., the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting relations (so-called metamorphic relations) among multiple executions of the program under test. In this paper, we present a metamorphic testing approach for the detection of faults in RESTful Web APIs. We first propose six abstract relations that capture the shape of many of the metamorphic relations found in RESTful Web APIs, we call these Metamorphic Relation Output Patterns (MROPs). Each MROP can then be instantiated into one or more concrete metamorphic relations. The approach was evaluated using both automatically seeded and real faults in six subject Web APIs. Among other results, we identified 60 metamorphic relations (instances of the proposed MROPs) in the Web APIs of Spotify and YouTube. Each metamorphic relation was implemented using both random and manual test data, running over 4.7K automated tests. As a result, 11 issues were detected (3 in Spotify and 8 in YouTube), 10 of them confirmed by the API developers or reproduced by other users, supporting the effectiveness of the approach.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P12-TIC-186

Metamorphic relations for enhancing system understanding and use

Modern information technology paradigms, such as online services and off-the-shelf products, often involve a wide variety of users with different or even conflicting objectives. Every software output may satisfy some users, but may also fail to satisfy others. Furthermore, users often do not know the internal working mechanisms of the systems. This situation is quite different from bespoke software, where developers and users usually know each other. This paper proposes an approach to help users to better understand the software that they use, and thereby more easily achieve their objectives—even when they do not fully understand how the system is implemented. Our approach borrows the concept of metamorphic relations from the field of metamorphic testing (MT), using it in an innovative way that extends beyond MT. We also propose a "symmetry" metamorphic relation pattern and a "change direction" metamorphic relation input pattern that can be used to derive multiple concrete metamorphic relations. Empirical studies reveal previously unknown failures in some of the most popular applications in the world, and show how our approach can help users to better understand and better use the systems. The empirical results provide strong evidence of the simplicity, applicability, and effectiveness of our methodology

On automatic testing of web search engines

Web search engines are very important because they are the means by which people retrieve information from the World Wide Web. However, testing these web search engines is difficult because there are no test oracles, so this research proposes seven new metrics based on the idea of metamorphic relations to alleviate the oracle problem in search engine testing. Using these metrics, our method can test search engines automatically in the absence of an ideal oracle. Using this method, we further conduct large-scale empirical studies to investigate and compare the qualities of four major search engines, namely, Google (www.google.com), Baidu (www.baidu.com), Bing (www.bing.com), and Chinese Bing (www.bing.com.cn). Our empirical studies involve more than 50 million queries sent to the search engines across 9 months, and about 300 GB data collected from the search engine responses. It is found that different search engines have significantly different performance and that the nature of the query terms can have a significant impact on the performance of the search engines. These empirical study results demonstrate that our method can effectively alleviate the oracle problem in search engine testing, and can help both developers and users to obtain a better understanding of the search engine behaviour under different operational profiles

Geological and structural analysis of the Hwange area-Northwest Zimbabwe: using remotely sensed data and geographic information systems (GIS)

There is a continuous need to locate more targets for coal exploration and evaluation of geological structures in the north-west coalfields in Zimbabwe. Conventional methods of analysing geological structures and field mapping are being hindered by inaccessibility of some areas and thick covers of Recent sediments. Remote sensing has been found to be a valuable method of identifying lithologic units and geological structures in the· area. Integration of the remotely sensed data in a 2D GIS resulted in recognition of spatial relationships between lithologic units, geological structures , coal seams and vegetation patterns. The Hwange area constitutes the western part of the Mid-Zambezi Karoo basin. The area consist of a wide spectrum of rocks ranging from Precambrian gneisses, Proterozoic schists and granulites, Karoo sediments to Tertiary and Recent sands. The area has been affected by a number of faults and shears some of which post date the Karoo sediments. These faults are an expression of the major tectonic events associated with this area. Some of the faults have been attributed to the effects of the Zambezi Rift System. Fault zones in the area, such as the Deka, Entuba and Inyantue Zones have been recognised as part of this system and these divide the Lower Karoo rocks into different coalfields. To try and evaluate the outcrop patterns and geological structures in the Hwange area, all the available geological and structural data were captured in a spatial database. The diversity of data incorporated in the spatial database demanded the need for a structured database design approach. The Entity-Relationship model was used to conceptualise the geological data of the ' Hwange area This model was transformed into the Relational Model that formed the implementation model of the database. Landsat 5 TM data covering the area from the Zimbabwean winter (20 June 1984) path 172, row 73 were also analysed for the information required to locate Karoo rift faults and the distribution of lithologic units associated with coal. The use of directional filters in the E-W and NE-SW directions and vegetation reflection characteristics during the dry season (June 1984) proved very effective in mapping fractures in the Karoo rocks. Landsat TM image enhancement techniques such as principal components analysis, edge enhancement, decorrelation stretching, band ratios; and colour composites made following these techniques, allowed mapping of different lithological units and discrimination between Karoo rocks and the crystalline basement rocks. Lineament analysis defined E-W, ENE-WSW, NE-SW and NW-SE conjugate sets of lineaments. The first three sets are related to the regional fracture zones of the Zambezi rift system The Entuba fault zone was found to be associated with most of the fractures affecting the Hwange coalfields. These have a dominant NE-SW and ENE-WSW trend in the Western Areas, Wankie Concession, Chaba, Entuba and Sinamatella coalfields. The E-W trending fracture set is dominated by joint sets in the Karoo basalt covering the north-west portion of the Hwange Coalfields. These show no relationship with the linear features of the Zambezi Rift system The NW-SE trending lineaments are dominantly developed on tilted bedding planes in the Karoo rocks as well as a few sparse joints in the Karoo basalt. Overlaying enhanced Landsat TM images on mapped faults and lithology data in a GIS revealed a number of features along the Entuba zone which were not previously known. The south-western part of the Entuba inlier was shown to consist of a synformal fold plunging to the south and bound on both sides by strike slip faults. Several kinematic indicators such as displacement of sedimentary strata have shown that the Entuba fault displays right lateral strike-slip coupled with dipslip movement. Proximity analysis using borehole data (depth to top and bottom of a coal seam) showed that most of the lineaments in the area are normal faults which have caused considerable displacements of the main coal seam Comparison of seam depth across most of these faults within coalfields and from one field to another shows that local and regional variations in depths of the main seam is primarily a function of vertical displacements along the faults over and above variations in the morphology of the pre-Karoo floor. The Entuba field was found to have greatest vertical variations over very short distances across faults, with depths varying from 60m to 520m from west to east over distances of less than 500m This part of the field has been partly affected by extensive normal faults, some of which can be traced for more than 10km. In the Hwange area, the Karoo rocks have been down faulted into a rift margin which is in turn divided into smaller fault blocks by intra-rift faulting. The shape of the fault blocks are further controlled by the orientation of the post-Karoo faults which have also down faulted the main coal seam Exploration activity in the area should also seek to establish the locations of these faults to help further decipher variations in depths of coal seams

Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps

Millions of people use navigation software every day to commute and travel. In addition, many systems rely upon the correctness of navigation software to function, ranging from directions applications to self-driving machinery. Navigation software is difficult to test because it is hard or very expensive to evaluate its output. This difficulty is generally known as the oracle problem, a fundamental challenge in software testing. In this study, we propose a metamorphic testing strategy to alleviate the oracle problem in testing navigation software, and conduct a case study by testing the Google Maps mobile app, its web service API, and its graphical user interface. The results show that our strategy is effective with the detection of several real-life bugs in Google Maps. This study is the first work on automated testing of navigation software with the detection of real-life bugs

QoS-aware Metamorphic Testing: An Elevation Case Study

Elevators are among the oldest and most widespread transportation systems, yet their complexity increases rapidly to satisfy customization demands and to meet quality of service requirements. Verification and validation tasks in this context are costly, since they rely on the manual intervention of domain experts at some points of the process. This is mainly due to the difficulty to assess whether the elevators behave as expected in the different test scenarios, the so-called test oracle problem. Metamorphic testing is a thriving testing technique that alleviates the oracle problem by reasoning on the relations among multiple executions of the system under test, the so-called metamorphic relations. In this practical experience paper, we report on the application of metamorphic testing to verify an industrial elevator dispatcher. Together with domain experts from the elevation sector, we defined multiple metamorphic relations that consider domain-specific quality of service measures. Evaluation results with seeded faults show that the approach is effective at detecting faults automatically