Empirical studies on exploratory software testing

Exploratory software testing (ET) is a practically relevant approach to software testing that lacks scientific knowledge. In ET, the tester's work is not based on predesigned and documented test cases. Instead, testing is guided by a higher-level plan or mission, and the testing work involves parallel test design, test execution, and learning. One of the distinct characteristics of ET is that the tester designs the tests during ET and uses information gained to design new and better tests continuously. The ET approach relies on testers' skills and experience. The main claimed benefits of ET are the tester's ability to apply personal knowledge and creativity during testing as well as effectiveness, efficiency, and agility in terms of adapting to changes and working with imperfect documentation. In this thesis, the ET approach has been studied using empirical research methods. Two case studies, one controlled experiment, and two field studies were performed to address three research goals: defining ET and understanding its applicability based on the literature; empirically investigating the benefits and shortcomings of ET; and providing empirically based results on how the ET approach is applied in practice. This research identifies different approaches to ET in industry and describes concrete testing practices. The role of the tester's personal knowledge is identified in the literature, and this research provides a detailed analysis of the application of personal knowledge in failure detection using ET. The main conclusions of this work are that ET can be as effective as test case-based approaches and even more efficient in certain contexts. The testers are capable of utilizing their personal knowledge in failure detection, and the role of personal knowledge is important in the ET approach. In addition, software testing in product organizations seems to involve multiple diverse organizational groups, and ET was found to be an applicable approach to engage domain experts in testing. The main implications of this thesis are introducing the exploratory testing approach to the research community and motivating its relevance by providing empirical studies in industry. In addition, the results of the effectiveness and efficiency of ET as well as the qualitative data on exploratory testing practices and the detailed analysis of knowledge in exploratory testing work are valuable for the research community. The main practical implications include presenting the benefits and applicability of the ET approach along with the potential shortcomings and providing empirical evidence regarding the benefits of ET

Refining the PoinTER “human firewall” pentesting framework

PurposePenetration tests have become a valuable tool in the cyber security defence strategy, in terms of detecting vulnerabilities. Although penetration testing has traditionally focused on technical aspects, the field has started to realise the importance of the human in the organisation, and the need to ensure that humans are resistant to cyber-attacks. To achieve this, some organisations “pentest” their employees, testing their resilience and ability to detect and repel human-targeted attacks. In a previous paper we reported on PoinTER (Prepare TEst Remediate), a human pentesting framework, tailored to the needs of SMEs. In this paper, we propose improvements to refine our framework. The improvements are based on a derived set of ethical principles that have been subjected to ethical scrutiny.MethodologyWe conducted a systematic literature review of academic research, a review of actual hacker techniques, industry recommendations and official body advice related to social engineering techniques. To meet our requirements to have an ethical human pentesting framework, we compiled a list of ethical principles from the research literature which we used to filter out techniques deemed unethical.FindingsDrawing on social engineering techniques from academic research, reported by the hacker community, industry recommendations and official body advice and subjecting each technique to ethical inspection, using a comprehensive list of ethical principles, we propose the refined GDPR compliant and privacy respecting PoinTER Framework. The list of ethical principles, we suggest, could also inform ethical technical pentests.OriginalityPrevious work has considered penetration testing humans, but few have produced a comprehensive framework such as PoinTER. PoinTER has been rigorously derived from multiple sources and ethically scrutinised through inspection, using a comprehensive list of ethical principles derived from the research literature

Survey on Exploratory Software Testing

Uuriv testimine on testimismeetod, millele pani aluse Cem Kaner 1983. aastal. Käesoleva magistritöö eesmärk on analüüsida, millises ulatuses kasutatakse uurivat testimist Eesti ja Soome tarkvaraettevõtetes ning milline on nende ettevõtete kogemus uuriva testimise osas. Samuti on eesmärgiks otsida seoseid rollide, töökogemuse, asukoha, ettevõtte suuruse, ettevõtte ajaloo, testimise korralduse ja uuriva testimise kasutamise viisi vahel. Lisaks pakub huvi, mida testijad arvavad uuriva testimise meetodist, millised on uuriva testimise eelised ja puudused ning kuidas plaanitakse uurivat testimist täiustada. Eesmärgi saavutamiseks viidi läbi küsitlus isikute hulgas, kes on tööalaselt seotud testimisega. Kahe kuu jooksul saatsid vastused 61 küsitletut, kellest 50 on pärit Eestist ja Soomest. Töös tehakse kokkuvõte saadud vastustest ning vastatakse kolmele peamisele uurimisküsimusele, milleks on: Mis iseloomustab küsitlusele vastanuid, kes kohaldavad uurivat testimist Eestis ja Soomes ja milline kogemus on nende ettevõtetel uuriva testimise kasutamisega? Millised faktorid mõjutavad uuriva testimise kasutamist Eesti ja Soome ettevõtetes? Mida arvavad tarkvaratehnikud uurivast testimisest? Küsitlusele vastanute hulgas on kõige rohkem testijaid (48%) ja juhtivaid testijaid (36%). Ligi 50% vastanutest on rohkem kui 5 aastat töökogemust. Eesti testijatest kasutab uurivat testimist suurem osa (50%) kui Soome testijatest (38%). Uurivat testimist kohaldatakse väikestes ettevõtetes (vähem kui 50 töötajat, 56%) rohkem kui suurtes (32%). Üle 50% ettevõtetest, kus uurivat testimist kohaldatakse, on vanemad kui 5 aastat. Ligikaudu 76% vastanutest, kes kasutavad uurivat testimist, kuuluvad oma ettevõttes eraldiseisvasse testimisorganisatsiooni. Töökogemus tundub olevat Eestis ja Soomes peamine faktor, mis mõjutab, kas uurivat testimist kasutatakse või mitte. Vastanud, kellel on pikaajaline töökogemus, kasutavad uurivat testimist rohkem kui need, kellel on väiksem töökogemus. Ettevõtte suurus on märgatavalt erinev Eesti ja Soome uurivat testimist kohaldavate ettevõtete vahel. Eestis eelistavad kasutada uurivat testimist vastanud väikestest ettevõtetest, kuid Soomes kasutavad uurivat testimist rohkem suurtest ettevõtetest pärit vastajad. Vastajad toovad välja, et uuriv testimine soosib loovust ning on efektiivne. Samas arvavad mõned vastanud, et uuriv testimine nõuab testijalt suuremat testimisalast võimekust, mis tähendab, et uuriv testimine ei sobi igaühele. Mõned vastanud saavad kasutada uurivat testimist vaid vähestes olukordades, mistõttu leiavad nad, et uuriv testimine ei ole paindlik. Peamine probleem uuriva testimisega on, et seda on raske salvestada. Enamik vastanutest on valmis muutma uuriva testimise salvestamise strateegiat.Exploratory Testing (ET) is a testing method that was created by Cem Kaner in 1983 [1]. [Pleease double-check: In the references (Section 6) it says Kaner, Falk and Nguyen, and the year is 1993. ET was created in 1983, but this was mentioned in references [6] in 1993]The objective of this thesis is to investigate to what extent ET is currently applied in Estonian and Finnish software companies and what experience the companies have with using ET. Also, we want to know more about the relationship between role, working experience, location, company size, company history, type of test organization and the way how ET is used. In addition, we try to find out how testers think about ET, e.g., the advantages and disadvantages of ET, and how they plan to improve ET. To achieve the objective of this thesis, we decided to use a survey to gather replies from testing relevant roles. After over 2 months, we got 61 replies. 50 are from Estonia and Finland. We summarize all information and answer three main research questions of this study, which are: RQ 1: What characterizes those respondents that apply ET in Estonia and Finland and what experience do their companies have with using ET? RQ 2: Which factors have an influence on using ET in Estonian and Finnish companies? RQ 3: How do software engineers think about ET? Respondents in Estonian and Finnish companies who apply ET are mostly testers (48%) and test managers (36%). Close to 50% of the respondents have more than five years of working experiences. More Estonian testers (50%) use ET than Finnish testers (38%). ET has been applied in small companies (less than 50 employees, 56%) relatively more often than in large companies (32%). Over 50% of the companies applying ET are older than 5 years. About 76% of the respondents who use ET are members of a separate testing organization within their company. “Working experience” seems to be the main factor that influences whether a company applies ET in Estonia and Finland. Respondents who have much working experience use ET more than those who have less working experience.“Company size” is significantly different between those companies which apply ET in Estonia as compared to Finland. Respondents who are from smaller companies in Estonia prefer using ET, but in Finland, bigger companies’ respondents use ET more than those who are from smaller companies. Respondents consider that ET supports creativity, and ET are efficient and effective. Meanwhile some respondents think ET requires higher testing ability for testers, which means ET does not suit for everyone. Some respondents can use ET for so limit testing situations that make them considering that ET is inflexible. The main problem which ET brings is“hard to record”. Most respondents are willing to change the recording strategy of ET

How is Exploratory Testing Used? : A State-of-the-Practice Survey

Jufo_ID: 59031 ; lyhenne ESEM.Context: Exploratory Testing has experienced a rise in popularity in the industry with the emergence of agile development practices, yet it remains unclear, in which domains and how it is used in practice. Goal: To study how software engineers understand and apply the principles of exploratory testing, as well as the specific advantages and difficulties they experience. Method: We conducted an online survey in the period June to August 2013 among Estonian and Finnish software developers and testers. Results: Our main findings are that the majority of testers, developers, and test managers using ET, (1) apply ET to usability- critical, performance-critical, security-critical and safety-critical software to a high degree; (2) use ET very flexibly in all types of test levels, activities, and phases; (3) perceive ET as an approach that supports creativity during testing and that is effective and efficient; and (4) find that ET is not easy to use and has little tool support. Conclusions: The high degree of application of ET in critical domains is particularly interesting and indicates a need for future research to obtain a better understanding of the effects of ET in these domains. In addition, our findings suggest that more support to ET users should be given (guidance and tools).Peer reviewe

Intercultural Communication and Collaboration in Software Development

Master's thesis in Business administration (BE501)Many Global Software firms outsource parts of their operations to other countries due to cheaper labour costs and to find employees with high knowledge within a specific area. Collaboration and communication between different nationalities aredifficult not only due to the distance geographically,but also distances in cultural aspects. The purpose of this study is to investigatedifferent factors that impact intercultural communication and collaboration in software development firms. Previous research in this field shows various factors impacting intercultural communication and collaboration in software development companies. To investigate these factors exploratory mixed-method design has been used. Initially, qualitative methodology was used to collect data from semi-structured interviews. Eight interviews were conducted with various work roles within a single Norwegian firm. The interviewees explain how various factors impact their communication and collaboration withcolleagues inIndia and Ukraine. These results have been analysed. Secondly, a quantitative survey was conducted based on Isern ́s (2014) eight identity factors of intercultural communication. Other factors based on the findings from the interviewshave also been discussed

FuzzTheREST - Intelligent Automated Blackbox RESTful API Fuzzer

In recent years, the pervasive influence of technology has deeply intertwined with human life, impacting diverse fields. This relationship has evolved into a dependency, with software systems playing a pivotal role, necessitating a high level of trust. Today, a substantial portion of software is accessed through Application Programming Interfaces, particularly web APIs, which predominantly adhere to the Representational State Transfer architecture. However, this architectural choice introduces a wide range of potential vulnerabilities, which are available and accessible at a network level. The significance of Software testing becomes evident when considering the widespread use of software in various daily tasks that impact personal safety and security, making the identification and assessment of faulty software of paramount importance. In this thesis, FuzzTheREST, a black-box RESTful API fuzzy testing framework, is introduced with the primary aim of addressing the challenges associated with understanding the context of each system under test and conducting comprehensive automated testing using diverse inputs. Operating from a black-box perspective, this fuzzer leverages Reinforcement Learning to efficiently uncover vulnerabilities in RESTful APIs by optimizing input values and combinations, relying on mutation methods for input exploration. The system's value is further enhanced through the provision of a thoroughly documented vulnerability discovery process for the user. This proposal stands out for its emphasis on explainability and the application of RL to learn the context of each API, thus eliminating the necessity for source code knowledge and expediting the testing process. The developed solution adheres rigorously to software engineering best practices and incorporates a novel Reinforcement Learning algorithm, comprising a customized environment for API Fuzzy Testing and a Multi-table Q-Learning Agent. The quality and applicability of the tool developed are also assessed, relying on the results achieved on two case studies, involving the Petstore API and an Emotion Detection module which was part of the CyberFactory#1 European research project. The results demonstrate the tool's effectiveness in discovering vulnerabilities, having found 7 different vulnerabilities and the agents' ability to learn different API contexts relying on API responses while maintaining reasonable code coverage levels.Ultimamente, a influência da tecnologia espalhou-se pela vida humana de uma forma abrangente, afetando uma grande diversidade dos seus aspetos. Com a evolução tecnológica esta acabou por se tornar uma dependência. Os sistemas de software começam assim a desempenhar um papel crucial, o que em contrapartida obriga a um elevado grau de confiança. Atualmente, uma parte substancial do software é implementada em formato de Web APIs, que na sua maioria seguem a arquitetura de transferência de estado representacional. No entanto, esta introduz uma série vulnerabilidade. A importância dos testes de software torna-se evidente quando consideramos o amplo uso de software em várias tarefas diárias que afetam a segurança, elevando ainda mais a importância da identificação e mitigação de falhas de software. Nesta tese é apresentado o FuzzTheREST, uma framework de teste fuzzy de APIs RESTful num modelo caixa preta, com o objetivo principal de abordar os desafios relacionados com a compreensão do contexto de cada sistema sob teste e a realização de testes automatizados usando uma variedade de possíveis valores. Este fuzzer utiliza aprendizagem por reforço de forma a compreender o contexto da API que está sob teste de forma a guiar a geração de valores de teste, recorrendo a métodos de mutação, para descobrir vulnerabilidades nas mesmas. Todo o processo desempenhado pelo sistema é devidamente documentado para que o utilizador possa tomar ações mediante os resultados obtidos. Esta explicabilidade e aplicação de inteligência artificial para aprender o contexto de cada API, eliminando a necessidade de analisar código fonte e acelerando o processo de testagem, enaltece e distingue a solução proposta de outras. A solução desenvolvida adere estritamente às melhores práticas de engenharia de software e inclui um novo algoritmo de aprendizagem por reforço, que compreende um ambiente personalizado para testagem Fuzzy de APIs e um Agente de QLearning com múltiplas Q-tables. A qualidade e aplicabilidade da ferramenta desenvolvida também são avaliadas com base nos resultados obtidos em dois casos de estudo, que envolvem a conhecida API Petstore e um módulo de Deteção de Emoções que fez parte do projeto de investigação europeu CyberFactory#1. Os resultados demonstram a eficácia da ferramenta na descoberta de vulnerabilidades, tendo identificado 7 vulnerabilidades distintas, e a capacidade dos agentes em aprender diferentes contextos de API com base nas respostas da mesma, mantendo níveis de cobertura aceitáveis

Enhancing the test and evaluation process: implementing agile development, test automation, and model-based systems engineering concepts

2020 Fall.Includes bibliographical references.With the growing complexity of modern systems, traditional testing methods are falling short. Test documentation suites used to verify the software for these types of large, complex systems can become bloated and unclear, leading to extremely long execution times and confusing, unmanageable test procedures. Additionally, the complexity of these systems can prevent the rapid understanding of complicated system concepts and behaviors, which is a necessary part of keeping up with the demands of modern testing efforts. Opportunities for optimization and innovation exist within the Test and Evaluation (T&E) domain, evidenced by the emergence of automated testing frameworks and iterative testing methodologies. Further opportunities lie with the directed expansion and application of related concepts such as Model-Based Systems Engineering (MBSE). This dissertation documents the development and implementation of three methods of enhancing the T&E field when applied to a real-world project. First, the development methodology of the system was transitioned from Waterfall to Agile, providing a more responsive approach when creating new features. Second, the Test Automation Framework (TAF) was developed, enabling the automatic execution of test procedures. Third, a method of test documentation using the Systems Modeling Language (SysML) was created, adopting concepts from MBSE to standardize the planning and analysis of test procedures. This dissertation provides the results of applying the three concepts to the development process of an airborne Electronic Warfare Management System (EWMS), which interfaces with onboard and offboard aircraft systems to receive and process the threat environment, providing the pilot or crew with a response solution for the protection of the aircraft. This system is representative of a traditional, long-term aerospace project that has been constantly upgraded over its lifetime. Over a two-year period, this new process produced a number of qualitative and quantitative results, including improving the quality and organization of the test documentation suite, reducing the minimum time to execute the test procedures, enabling the earlier identification of defects, and increasing the overall quality of the system under test. The application of these concepts generated many lessons learned, which are also provided. Transitioning a project's development methodology, modernizing the test approach, and introducing a new system of test documentation may provide significant benefits to the development of a system, but these types of process changes must be weighed against the needs of the project. This dissertation provides details of the effort to improve the effectiveness of the T&E process on an example project, as a framework for possible implementation on similar systems