Automating performance testing of an application platform

Tietokoneohjelmat kehittyvät jatkuvasti monimutkaisemmaksi, minkä vuoksi niiden testauksen merkitys korostuu. Manuaaliseen ohjelmistojen testaukseen kuluu tavallisesti paljon aikaa ja resursseja. Testauksen kuluja ja testaukseen tarvittavaa työmäärää voidaan vähentää käyttämällä testiautomaatiota, jolloin osa testien suorituksesta jätetään automaattisen testaustyökalun tehtäväksi. Suorituskykytestauksella selvitetään järjestelmän kykyä suoriutua tehtävästään kuorman alla, ja sen avulla voidaan havaita pullonkauloja, jotka heikentävät järjestelmän suorituskykyä. Järjestelmät, joiden suorituskyky ei ole riittävä, aiheuttavat ylimääräisiä kuluja ja vähentävät järjestelmästä saatavia hyötyjä. Tehokas ja luotettava suorituskykytestaus voidaan suorittaa vain automaattisen testaustyökalun avulla. Sovellusalusta on joukko yhteisen rakenteen muodostavia alijärjestelmiä ja ohjelmointirajapintoja, joiden varaan sovelluksia voidaan kehittää. Tässä työssä toteutettiin eräälle sovellusalustalle suorituskykytestausjärjestelmä, joka mahdollistaa automaattisen suorituskykytestauksen. Työn tavoitteena oli selvittää, miten sovellusalustan suorituskykytestit ja niiden raportointi voidaan automatisoida, jotta saadaan hyödyllisiä ja vertailukelpoisia tuloksia järjestelmän suorituskyvystä. Toteutettu suorituskykytestausjärjestelmä mahdollistaa sovellusalustan palvelinympäristön ja sovellusalustan varaan kehitettyjen sovellusten suorituskyvyn testaamisen. Testausjärjestelmä tarjoaa riittävät ominaisuudet suorituskykytestien toteuttamiseen ja tulosten tarkasteluun, mutta testausjärjestelmää voidaan edelleen kehittää monella eri tavalla. Testausjärjestelmällä on toteutettu kattavat suorituskykytestit sovellusalustan palvelinympäristölle, joten testausjärjestelmän toteutusta voidaan pitää onnistuneena

Jatkuvien menetelmien parantaminen järjestelmäpiiri kehityksessä

This work is about continuous practices in embedded System-on-Chip development. Continuous practices include continuous integration, continuous delivery, and continuous development. These practices mean committing small code changes often to the repository’s main branch. Then the changes are automatically tested and integrated with the rest of the system. In the case of continuous deployment, all the changes are automatically deployed to production without any human interaction. Continuous practices are meant to make development faster and more effective, give feedback faster and improve quality by reducing bugs. These tasks are important in today’s industry which is continuously changing, and customer satisfaction is as important as ever. This creates the demand to deliver new products and updates rapidly along with high quality. In the Nokia Networks’ System-on-Chip department there was a need to increase the level of automated processes by improving the continuous practices. This work studies continuous practices based on literature and identifies ways to improve continuous practice processes used at System-on-Chip development. The implementation of this work was done at the System-on-Chip departments’ software unit where the current state was analysed. The main improvement points found in the analysis were related to automated function, investment and working habits. Based on the analysis the implementation plan was formed. The implementation included adding more functions to the continuous integration server, improving feedback and making the results more visible. These were done by creating more Jenkins jobs and integrating Robot Framework to the testing. All the improvements were not possible to do within the time scope or without the support of the whole department. Therefore, those problem points were analysed, and detailed plans were formed to solve them in the near future

Enhancing the test and evaluation process: implementing agile development, test automation, and model-based systems engineering concepts

2020 Fall.Includes bibliographical references.With the growing complexity of modern systems, traditional testing methods are falling short. Test documentation suites used to verify the software for these types of large, complex systems can become bloated and unclear, leading to extremely long execution times and confusing, unmanageable test procedures. Additionally, the complexity of these systems can prevent the rapid understanding of complicated system concepts and behaviors, which is a necessary part of keeping up with the demands of modern testing efforts. Opportunities for optimization and innovation exist within the Test and Evaluation (T&E) domain, evidenced by the emergence of automated testing frameworks and iterative testing methodologies. Further opportunities lie with the directed expansion and application of related concepts such as Model-Based Systems Engineering (MBSE). This dissertation documents the development and implementation of three methods of enhancing the T&E field when applied to a real-world project. First, the development methodology of the system was transitioned from Waterfall to Agile, providing a more responsive approach when creating new features. Second, the Test Automation Framework (TAF) was developed, enabling the automatic execution of test procedures. Third, a method of test documentation using the Systems Modeling Language (SysML) was created, adopting concepts from MBSE to standardize the planning and analysis of test procedures. This dissertation provides the results of applying the three concepts to the development process of an airborne Electronic Warfare Management System (EWMS), which interfaces with onboard and offboard aircraft systems to receive and process the threat environment, providing the pilot or crew with a response solution for the protection of the aircraft. This system is representative of a traditional, long-term aerospace project that has been constantly upgraded over its lifetime. Over a two-year period, this new process produced a number of qualitative and quantitative results, including improving the quality and organization of the test documentation suite, reducing the minimum time to execute the test procedures, enabling the earlier identification of defects, and increasing the overall quality of the system under test. The application of these concepts generated many lessons learned, which are also provided. Transitioning a project's development methodology, modernizing the test approach, and introducing a new system of test documentation may provide significant benefits to the development of a system, but these types of process changes must be weighed against the needs of the project. This dissertation provides details of the effort to improve the effectiveness of the T&E process on an example project, as a framework for possible implementation on similar systems

Software Testing: An Analysis of the Impacts of Test Automation on Software’s Cost, Quality and Time

Software testing is an essential yet time consuming and tedious task in the software development cycle despite the accessibility of most capable quality assurance teams and tools. Test automation is widely being utilised within the software industry to provide increased testing capacity to ensure high product quality and reliability. This thesis will specifically be addressing automated testing whereby test cases are manually written and executed automated. Test automation has its benefits, drawbacks, and impacts on different stages of development. Furthermore, there is often a disconnect between non-technical and technical roles, where non-technical roles (e.g., management) predominantly strive to reduce costs and delivery time whereas technical roles are often driven by quality and completeness. Although it is widely understood that there are challenges with adopting and using automated testing, there is a lack of evidence to understand the different attitudes toward automated testing, focusing specifically on why it is not adopted. In this thesis, the author has surveyed practitioners within the software industry from different roles to determine common trends and draw conclusions. A two-stage approach is presented, comprising of a comprehensive descriptive analysis and the use of Principle Component Analysis (PCA). In total, 81 participants were provided with a series of 22 questions and their responses were compared against job role types and experience levels. In summary, 6 key findings are presented covering expertise, time, cost, tools and techniques, utilisation, organisation and capacity

Software defect prediction using maximal information coefficient and fast correlation-based filter feature selection

Software quality ensures that applications that are developed are failure free. Some modern systems are intricate, due to the complexity of their information processes. Software fault prediction is an important quality assurance activity, since it is a mechanism that correctly predicts the defect proneness of modules and classifies modules that saves resources, time and developers’ efforts. In this study, a model that selects relevant features that can be used in defect prediction was proposed. The literature was reviewed and it revealed that process metrics are better predictors of defects in version systems and are based on historic source code over time. These metrics are extracted from the source-code module and include, for example, the number of additions and deletions from the source code, the number of distinct committers and the number of modified lines. In this research, defect prediction was conducted using open source software (OSS) of software product line(s) (SPL), hence process metrics were chosen. Data sets that are used in defect prediction may contain non-significant and redundant attributes that may affect the accuracy of machine-learning algorithms. In order to improve the prediction accuracy of classification models, features that are significant in the defect prediction process are utilised. In machine learning, feature selection techniques are applied in the identification of the relevant data. Feature selection is a pre-processing step that helps to reduce the dimensionality of data in machine learning. Feature selection techniques include information theoretic methods that are based on the entropy concept. This study experimented the efficiency of the feature selection techniques. It was realised that software defect prediction using significant attributes improves the prediction accuracy. A novel MICFastCR model, which is based on the Maximal Information Coefficient (MIC) was developed to select significant attributes and Fast Correlation Based Filter (FCBF) to eliminate redundant attributes. Machine learning algorithms were then run to predict software defects. The MICFastCR achieved the highest prediction accuracy as reported by various performance measures.School of ComputingPh. D. (Computer Science