Improving Automated Software Testing while re-engineering legacy systems in the absence of documentation

Legacy software systems are essential assets that contain an organizations' valuable business logic. Because of outdated technologies and methods used in these systems, they are challenging to maintain and expand. Therefore, organizations need to decide whether to redevelop or re-engineer the legacy system. Although in most cases, re-engineering is the safer and less expensive choice, it has risks such as failure to meet the expected quality and delays due to testing blockades. These risks are even more severe when the legacy system does not have adequate documentation. A comprehensive testing strategy, which includes automated tests and reliable test cases, can substantially reduce the risks. To mitigate the hazards associated with re-engineering, we have conducted three studies in this thesis to improve the testing process. Our rst study introduces a new testing model for the re-engineering process and investigates test automation solutions to detect defects in the early re-engineering stages. We implemented this model on the Cold Region Hydrological Model (CRHM) application and discovered bugs that would not likely have been found manually. Although this approach helped us discover great numbers of software defects, designing test cases is very time-consuming due to the lack of documentation, especially for large systems. Therefore, in our second study, we investigated an approach to generate test cases from user footprints automatically. To do this, we extended an existing tool to collect user actions and legacy system reactions, including database and le system changes. Then we analyzed the data based on the order of user actions and time of them and generated human-readable test cases. Our evaluation shows that this approach can detect more bugs than other existing tools. Moreover, the test cases generated using this approach contain detailed oracles that make them suitable for both black-box and white-box testing. Many scienti c legacy systems such as CRHM are data-driven; they take large amounts of data as input and produce massive data after applying mathematical models. Applying test cases and nding bugs is more demanding when we are dealing with large amounts of data. Hence in our third study, we created a comparative visualization tool (ComVis) to compare a legacy system's output after each change. Visualization helps testers to nd data issues resulting from newly introduced bugs. Twenty participants took part in a user study in which they were asked to nd data issued using ComVis and embedded CRHM visualization tool. Our user study shows that ComVis can nd 51% more data issues than embedded visualization tools in the legacy system can. Also, results from the NASA-TLX assessment and thematic analysis of open-ended questions about each task show users prefer to use ComVis over the built-in visualization tool. We believe our introduced approaches and developed systems will signi cantly reduce the risks associated with the re-engineering process. i

Similarity-based Web Element Localization for Robust Test Automation

Non-robust (fragile) test execution is a commonly reported challenge in GUI-based test automation, despite much research and several proposed solutions. A test script needs to be resilient to (minor) changes in the tested application but, at the same time, fail when detecting potential issues that require investigation. Test script fragility is a multi-faceted problem. However, one crucial challenge is how to reliably identify and locate the correct target web elements when the website evolves between releases or otherwise fail and report an issue. This article proposes and evaluates a novel approach called similarity-based web element localization (Similo), which leverages information from multiple web element locator parameters to identify a target element using a weighted similarity score. This experimental study compares Similo to a baseline approach for web element localization. To get an extensive empirical basis, we target 48 of the most popular websites on the Internet in our evaluation. Robustness is considered by counting the number of web elements found in a recent website version compared to how many of these existed in an older version. Results of the experiment show that Similo outperforms the baseline; it failed to locate the correct target web element in 91 out of 801 considered cases (i.e., 11%) compared to 214 failed cases (i.e., 27%) for the baseline approach. The time efficiency of Similo was also considered, where the average time to locate a web element was determined to be 4 milliseconds. However, since the cost of web interactions (e.g., a click) is typically on the order of hundreds of milliseconds, the additional computational demands of Similo can be considered negligible. This study presents evidence that quantifying the similarity between multiple attributes of web elements when trying to locate them, as in our proposed Similo approach, is beneficial. With acceptable efficiency, Similo gives significantly higher effectiveness (i.e., robustness) than the baseline web element localization approach

HTML5-pohjaisten hybridimobiilisovellusten käyttöliittymätestauksen automatisoinnin hyödyllisyys: Tapaustutkimus

While the first research papers on GUI test automation date back to the 1990s, its use in the software industry is still relatively rare. Traditionally, GUIs have been tested manually only and software automation has focused on the lower levels of testing such as unit testing or integration testing. The main reason for that is the complexity of GUIs compared to the lower software components. However, the manual testing of GUIs is a tedious and time-consuming process that involves repetitive and dull tasks, since the same tests need to be executed repeatedly on every testing iteration of the software under testing. The main goal with GUI test automation is to automate those steps and by doing so improve the cost-efficiency of the testing process and free the testers’ time on more meaningful and useful tasks. The previous research on GUI test automation reveals contradicting results. Some of the research has found GUI test automation to be both beneficial and cost-efficient and while others have found results suggesting the exact opposite. The contradicting results from previous research and the unclarity on the benefits, challenges, limitations and impediments of GUI test automation worked as the main driver for this thesis. The research was conducted as a combination of a literature review on the subject and a case study of three HTML5-based hybrid mobile application projects in the mobile development unit of one of the biggest IT companies in Finland.Käyttöliittymätestiautomaation käyttö yrityksissä on verrattain harvinaista, vaikka ensimmäiset tutkimukset aiheesta ovat 1990-luvulta. Perinteisesti käyttöliittymiä on testattu manuaalisesti ja ohjelmistotestausautomaatio keskitetty ohjelmiston alempien tasojen testaamiseen yksikkö- ja integraatiotesteillä. Pääsyy tähän on käyttöliittymien monimutkaisuus verrattuna alemman tason ohjelmistokomponentteihin. Käyttöliittymän testaaminen manuaalisesti on kuitenkin vaivalloinen, aikaa vaativa ja toisteinen prosessi, koska samat testit suoritetaan jokaisella testiajolla. Käyttöliittymätestauksen automatisoinnin päätavoite on testauksen kustannus-tehokkuuden parantaminen ja testaajien ajan vapauttaminen olennaisempiin tehtäviin. Aikaisemmat tutkimustulokset käyttöliittymätestiautomaatioon liittyen ovat ristiriitaisia. Osa tutkimuksista on todennut käyttöliittymätestiautomaation olevan hyödyllistä ja kustannustehokasta ja osa on päätynyt päinvastaisiin tuloksiin. Tämän työn päämotivaattoreina toimivat aiemman tutkimuksen ristiriitaiset tulokset ja epäselvyys käyttöliittymätestiautomaation hyödyllisyydestä ja kustannustehokkuudesta. Työn päätavoitteena oli tutkia voiko käyttöliittymätestiautomaation käyttö olla hyödyllistä ja kustannustehokasta. Työ koostuu kirjallisuuskatsauksesta ja kolmen HTML5-pohjaisen hybridimobiilisovelluksen tapaustutkimuksesta testiautomaation hyödyllisyyteen ja kustannustehokkuuteen liittyen

A Survey of the Selenium Ecosystem

Selenium is often considered the de-facto standard framework for end-to-end web testing nowadays. It allows practitioners to drive web browsers (such as Chrome, Firefox, Edge, or Opera) in an automated fashion using different language bindings (such as Java, Python, or JavaScript, among others). The term ecosystem, referring to the open-source software domain, includes various components, tools, and other interrelated elements sharing the same technological background. This article presents a descriptive survey aimed to understand how the community uses Selenium and its ecosystem. This survey is structured in seven categories: Selenium foundations, test development, system under test, test infrastructure, other frameworks, community, and personal experience. In light of the current state of Selenium, we analyze future challenges and opportunities around it.This work has been supported by the European Commission under the H2020 project "MICADO" (GA-822717), by the Government of Spain through the project "BugBirth" (RTI2018-101963-B-100), by the Regional Government of Madrid (CM) through the project "EDGEDATA-CM" (P2018/TCS-4499) cofunded by FSE & FEDER, and by the project "Analytics using sensor data for FlatCity" (MINECO/ERDF, EU) funded in part by the Spanish Agencia Estatal de Investigación (AEI) under Grant TIN2016-77158-C4-1-R and in part by the European Regional Development Fund (ERDF)

Agile Testing: Improving the Process : Case Descom

The thesis was assigned by Descom, a marketing and technology company based in Jyväskylä. The aim of the thesis was to research the current state of testing inside the organization, and to improve on the existing processes and practices. The thesis was carried out as a design research (applied action research), because the focus was improving already existing processes inside a company. The theory base contains a wide range of subjects from agile development models, the testing process, and process improvement models to agile testing. Without a solid base of multiple aspects it would have been impossible to understand how the testing works as a process and how it could have been improved. As Descom uses agile development it was necessary to follow the same principles throughout the writing of the thesis and on results. As a result information was provided for the company about the current state of testing procedures at Descom and how to improve the testing and processes in the future. The documentation already existing for testing such as the test plan and test report were updated. New documents such as a process improvement plan based on Critical Testing Processes, test strategy and testing policy were also created. Figures of the testing process, and the processes for all test types in use were created to be used as a visual aid for understanding the testing as whole at Descom.Opinnäytetyön toimeksianto tuli Descomilta, joka on Jyväskylästä lähtöisin oleva markkinointi ja teknologia yritys. Työn tavoitteena oli tutkia testauksen tilaa organisaatiossa ja kehittää olemassa olevia prosesseja ja käytäntöjä. Tutkimusmenetelmäksi valikoitui kehittämistutkimus, koska painotus oli olemassa olevien prosessien kehityksessä yrityksen sisällä. Teoriapohjassa käsiteltiin monia aiheita ketterästä sovelluskehityksestä, testausprosessista ja prosessi kehityksestä aina ketterään testaukseen asti. Ilman kattavaa pohjaa monille osa-alueille, olisi ollut mahdotonta ymmärtää miten testaus toimii prosessina ja miten sitä pystyy kehittämään. Descom toimii ketterän sovelluskehityksen mukaisesti projekteissaan, joten oli tärkeää seurata samoja ketteriä periaatteita läpi opinnäytetyön kirjoittamisen ja tuloksissa. Tuloksena saatiin tietoa yritykselle, siitä miten testaus on toiminut Descomilla ja kuinka testausta ja prosesseja tulisi kehittää tulevaisuudessa. Myös aiemmin olemassa olleet testausdokumentit päivitettiin. Uusina dokumentteina laadittiin suunnitelma prosessikehitykseen, joka perustui Critical Testing Processes –malliin, testausstrategia ja testauspolitiikka. Prosessikuvaus tehtiin kaavioita käyttäen, joilla kuvattiin prosessi kokonaisuutena sekä käytettävät testaustasot

BIM Assisted Design Process Automation for Pre-Engineered Buildings (PEB)

The effective adoption and implementation of Building Information Modeling (BIM) is still challenging for the construction industry. However, studies and reports show a significant increase in the rate of BIM implementation and adoption in mainstream construction activities over the last five years. In contrast, Pre-Engineered Building (PEB) construction, a specialized construction system which provides a very efficient approach for construction of primarily industrial buildings, has not seen the same uptake in BIM implementation and adoption. The thesis reviews the benefits and the main applications of BIM for the PEB industry as well as challenges of its practical implementation. To facilitate the implementation of BIM in the PEB industry, a BIM framework is adapted from Pre-fabrication (Pre-fab) industry and new workflows, process maps, and data-exchange strategies are developed. As the PEB industry traditionally makes significant use of automation in its design and fabrication process, accordingly this work investigates the technical challenges of incorporating automation into the proposed BIM process. Two new BIM concepts, “Planar Concept” and “Floating LOD”, are then developed and implemented as a solution to these challenges. To define the proper input/output criteria for automated BIM design processes, a numerical study was performed to identify an “Optimum LOD”. A software implementation embodying the research outcomes was developed to illustrate the feasibility of the results. Its step-by-step deployment is analyzed and discussed using an example industry PEB design project. Further, the impact of this work is extended by integrating the developed BIM framework and automated design process with wind engineering design activities and tools and procurement systems. The study concludes that the deployment of the proposed BIM framework could significantly address existing issues in project design through to operation processes found in the PEB industry. Also, the results indicate the developed concepts have the potential for supporting the application of automation in the other sectors of the general construction industry. This thesis is written using the Integrated Article format and includes various complementary studies

A web-based Decision Support System (DSS) to assist SMEs to broker risks and rewards for BIM adoption

Building Information Modelling (BIM) usage in the UK’s construction industry has recorded a significant increase over the last few years. However, available evidence suggests that BIM adoption amongst larger construction firms and innovators seems be dominant, while the uptake of BIM by Small and Medium Sized enterprises (SMEs) remains relatively poor. Consequently, SMEs are currently lagging behind and are losing out in winning publicly funded projects. SMEs have not fully recognised the benefits of using BIM in project delivery. Guidance and frameworks to assist SMEs to make an informed decision about BIM adoption are currently lacking. It appears that SMEs are yet to be convinced that BIM is beneficial to them, and remain concerned about the potential risks to their business. Guidance and frameworks to assist SMEs in making an informed decision about BIM adoption are currently lacking.This study seeks to bridge this gap and provide a decision-support system (DSS) to assist in the analysis of the risks and rewards of adopting BIM by SMEs, in project delivery. As a result, a conceptual framework was developed to give a theoretical foundation to the study of brokering risks and rewards in the adoption of BIM for project delivery. This framework is comprehensive and includes trading off risks and rewards associated with several criteria, such as stage of involvement, project value, funding, and the procurement route chosen. The approach was validated by a representative sample of BIM users. The results of the validation of the framework provided an informed basis for the development of the DSS. The latter was validated by a sample of SMEs, according to several criteria such as ease of use of the Graphical user interface (GUI), quality of information, level of information presented, trading off risks and rewards of adoption of BIM in project delivery.The findings of the framework validation revealed that early design stage, project size between £5m and £50m, private funding, and integrated project delivery procurement are the best opportunities that enable SMEs to maximise the benefits and minimise the risks, when adopting BIM. Regarding the DSS validation, most participants reported that they had found the DSS easy to use, especially the GUI. They were also positive about the level and quality of information and knowledge provided by the DSS. In particular, they found the DSS informative to broker risks and rewards for BIM adoption