Software Test Management Tool Evaluation Framework

Tarkvara testimine on korduvalt tõestanud oma olulisust tarkvara arenduse juures viimase kümnendi jooksul. Tarkvara testimise tunnustuse kasvuga on esile kerkinud paljud elektroonilised testide haldamissüsteemid (THS). Kuigi nende hindamiseks on mitmeid võimalusi, pole me siiski leidnud selleks ühtselt aktsepteeritud meetodit. Me usume, et see on probleem, mida tuleks uurida, sest THS hindamine on sageli subjektiivne, sõltudes pigem hindaja arvamusest kui objektiivsest lähenemisest. Sama mure on ka kvaliteedikontrolli meeskondade juhtidel, kui neil palutakse hinnata, kas THS, mida neil kasutatakse, vastab ettevõtte vajadustele. Mõistmaks THS hindamise olulisust, uurisime me testimisprotsesside alast kirjandust ning analüüsisime hetkel olemasolevaid rakendusi. Seejärel kaardistasime tuvastatud testimisprotsessid ning nende väljundid. Läbi viidud analüüsi tulemusena saadud andmete põhjal koostasime veebiküsitluse ning saatsime Eesti IT-firmadele. Uuringu tulemuste põhjal koostasime me THS hindamisraamistiku, mis aitab ettevõtetel mõõta, kas ostetav THS on joondatud firma eesmärkidega, ning vähendab hinnangu andmisel subjektiivsust. Meie raamistik võimaldab testimis- ning projektijuhtidel mõista, kas nende ettevõttes kasutusel olev rakendus vastab firma ootustele. Veendumaks loodud hindamisprotsessi kasutatavuses, viisime kvaliteedikontrolli spetsialistide seas läbi täiendava uuringu, mis kinnitas meie ootusi. Meie lõputöö edasi arendamiseks on mitmeid võimalusi. Raamistikust võib luua veebirakenduse, et seda oleks kergem kasutada või laiemalt levitada. Samuti tuleks uurimust laiendada, kaasates ning analüüsides teiste euroopa riikide IT-firmade THS nõudeid. Kindlasti ei saa mainimata jätta, et THS nõudeid tuleks aja möödudes täiendada vastavalt uutele trendidele kvaliteedikontrollis. Lõpetuseks me usume, et käesoleva lõputöö tulemus, THS hindamisraamistik, on praktiline ning vajalik panus tarkvara kvaliteedikontrolli kogukonnale.Software testing has proven its value for software development increasingly over the last decade. With the recognition of the benefits of software testing, several software test management tools (TMT) have emerged on the market. Although there exist different approaches, there is no method for a systematic TMT assessment. This is a problem because to our knowledge, evaluating TMT is rather a subjective task, heavily depending on the evaluators’ opinions rather than based on the objective approach. The same problem applies when test managers are asked to evaluate whether their currently used TMT meets the company’s expectations. In order to understand the importance and neccessity of TMT evaluation we perform a literature study on software testing processes and existing TMT market studies. Then we map together the identified test activities and test artifacts. The results help us formulate and design an online questionnaire and perform a TMT survey within the Estonian IT companies. Based on the survey results, a framework for evaluating TMT software is created. Such a framework could potentially help companies to measure the TMT suitability to company’s goals and to decrease subjectivity of the TMT assessment. The framework also provides test and project managers the understanding whether their current TMTs meet the company’s expectations. We validate the framework with a case study performed among Quality Assurance specialists to collect information on the framework usability. Possibilities for future work based on this thesis are numerous. The framework can be made into an application for ease of use and wider distribution. Expanding the research onto other European countries is another viable choice. Also expanding the TMT requirements based on new trends in testing can be taken into consideration. In conclusion, we believe this thesis contributes to the testing community with a practical TMT evaluation method

Testing-based process for component substitutability

Software components have emerged to ease the assembly of software systems. However, updates of systems by substitution or upgrades of components demand careful management due to stability risks of deployed systems. Replacement components must be properly evaluated to identify if they provide the expected behaviour affected by substitution. To address this problem, this paper proposes a substitutability assessment process in which the regular compatibility analysis is complemented with the use of black-box testing criteria. The purpose is to observe the components' behaviour by analysing their internal functions of data transformation, which fulfils the observability testing metric. The approach is conceptually based on the technique Back-to-Back testing. When a component should be replaced, a specific Test Suite TS is built in order to represent its behavioural facets, viz. a Component Behaviour TS. This TS is later exercised on candidate upgrades or replacement components with the purpose of identifying the required compatibility. Automation of the process is supported through the testooj tool, which constrains the conditions and steps of the whole process in order to provide a rigorous and reliable approach.Fil: Flores, Andrés Pablo. Universidad Nacional del Comahue. Facultad de Informática. Departamento Ingeniería de Sistemas; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Patagonia Confluencia; ArgentinaFil: Polo, Macario. Universidad de Castilla-La Mancha; Españ

STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS

It is typical for digital instrumentation and controls, used to manage significant risk, to undergo substantial amounts of scrutiny. The equipment must be proven to have the necessary level of design integrity. The details of the scrutiny vary based on the particular industry, but the ultimate goal is to provide sufficient evidence that the equipment will operate successfully when performing their required functions. To be able to stand up to the scrutiny and more importantly, successfully perform the required safety functions, the equipment must be designed to defend against random hardware failures and also to prevent systematic faults. These design activities must also have been documented in a manner that sufficiently proves their adequacy. The variability in the requirements of the different industries makes this task difficult for instrumentation and controls equipment manufacturers. To assist the manufacturers in dealing with these differences, a standardization of requirements is needed to facilitate clear communication of expectations. The IEC 61508 set of standards exists to fulfill this role, but it is not yet universally embraced. After that occurs, various industries, from nuclear power generation to oil & gas production, will benefit from the existence of a wider range of equipment that has been designed to perform in these critical roles and that also includes the evidence necessary to prove its integrity. The manufacturers will then be able to enjoy the benefit of having a larger customer base interested in their products. The use of IEC 61508 will also help industries avoid significant amounts of uncertainty when selecting commercial off-the-shelf equipment. It is currently understood that it cannot be assumed that a typical commercial manufacturer’s equipment designs and associated design activities will be adequate to allow for success in these high risk applications. In contrast, a manufacturer that seeks to comply with IEC 61508 and seeks to achieve certification by an independent third party can be assumed to be better suited for meeting the needs of these demanding situations. Use of these manufacturers help to avoid substantial uncertainty and risk

Software Engineering Laboratory Series: Proceedings of the Twenty-Second Annual Software Engineering Workshop

The Software Engineering Laboratory (SEL) is an organization sponsored by NASA/GSFC and created to investigate the effectiveness of software engineering technologies when applied to the development of application software. The activities, findings, and recommendations of the SEL are recorded in the Software Engineering Laboratory Series, a continuing series of reports that includes this document

Integrating Oracle PeopleSoft Campus Solution to External Applications

An integration solution must sustain multiple PeopleSoft upgrades, which is necessary to preserve investment in system integrations. Since the underlying structures and connection technologies of PeopleSoft have been and can be migrated from version to version in order to enhance features and performance, it is critical for any external component of integration to be built based on publicly visible interfaces of the PeopleSoft component. We have developed a standard-based solution to integrate “PeopleSoft Campus Solution” into “Microsoft SharePoint” using Web services generated by PeopleSoft’s Pure Internet Architecture. We have illustrated such kind of integration in two examples that emulate some of the imminent problems in the University’s current information systems between the PeopleSoft Campus and SharePoint Workflow. The methodology used in this is applicable to integrations of general COTS software systems into modern enterprise information systems

Strategies for the intelligent selection of components

It is becoming common to build applications as component-intensive systems - a mixture of fresh code and existing components. For application developers the selection of components to incorporate is key to overall system quality - so they want the `best\u27. For each selection task, the application developer will de ne requirements for the ideal component and use them to select the most suitable one. While many software selection processes exist there is a lack of repeatable, usable, exible, automated processes with tool support. This investigation has focussed on nding and implementing strategies to enhance the selection of software components. The study was built around four research elements, targeting characterisation, process, strategies and evaluation. A Post-positivist methodology was used with the Spiral Development Model structuring the investigation. Data for the study is generated using a range of qualitative and quantitative methods including a survey approach, a range of case studies and quasiexperiments to focus on the speci c tuning of tools and techniques. Evaluation and review are integral to the SDM: a Goal-Question-Metric (GQM)-based approach was applied to every Spiral

Hardware Certification for Real-time Safety-critical Systems: State of the Art

This paper discusses issues related to the RTCA document DO-254 Design Assurance Guidance for Airborne Electronic Hardware and its consequences for hardware certification. In particular, problems related to circuits’ compliance with DO-254 in avionics and other industries are considered. Extensive literature review of the subject is given, including current views on and experiences of chip manufacturers and EDA industry with qualification of hardware design tools, including formal approaches to hardware verification. Some results of the authors’ own study on tool qualification are presented

The Importance of 'Risk Radar' in Software Risk Management: A Case of a Malaysian Company

‘Risk radar’ is applied to a company in Malaysia, a discussion on the implementation, implications and recommendation highlighted in this paper. The scope of this study has been an analysis of risk management and risk exposure of software projects practices in the company. This study also provided the evident that the successes of the several software that goes into the Malaysian market, depending on how risk management and its plan in software development as in the case of the selected company. It also exposed on how significant is the risk management contributing to cost effective and growth. Findings also included using 80/20 rules or Pareto Principle, 80% of the risks item listed by Boehm in Ten (10) Top Risks are due to 20% of sources (i.e. soft risks). Empirical studies have shown that 80% of the software rework comes from 20% of the problems, and that many of these critical problems involve neglect of off nominal requirements and all these negligence are caused by human (soft risk) (Boehm, 1989; Boehm-Basili, 2001; Standish Group Chaos Study Report (STANDISH), 1995).The company must implement the propose system to ensure that good practice and successful implementation of software risk management is the key factor to successful creation of software that are marketable and high quality benchmarking of plant industrial solutions. It could contribute to gain competitive advantage by at least 50% of project cost due to risks such as rework, budget overruns cost overrun, content deficiencies and etc.; ability to sustain due to minimum impact by software risks; and ability to own the technology rather than uses the technology with reasonable cost in development and always meet or exceed customer requirements

Developmental Flight Test Lessons Learned from Open Architecture Software in the Mission Computer of the U.S. Navy E-2C Group II Aircraft

The Naval Air Systems Command commissioned the E-2C Hawkeye Group II Mission Computer Replacement Program and tasked Air Test and Evaluation Squadron Two-Zero and the E-2C Integrated Test Team to evaluate the integration of the form, fit, and function of the OL-698/ASQ Mission Computer Replacement (MCR) for replacement of the Litton L-304 Mission Computer in the E-2C Group II configured aircraft. As part of the life cycle support of the E-2C aircraft, the MCR configuration fields a new, more reliable Commercial-off-the-Shelf (COTS) hardware system and preserves the original software investment by emulating the existing Litton Instructional Set Architecture (LISA) legacy code. Incorporating Northrop Grumman Space Technology’s Reconfigurable Processor for Legacy Applications Code Execution (RePLACE) software re-hosting technique, the investment in the LISA software is maintained. Conducting developmental test of robust software systems, such as the MCR and its associated software, provided dramatically different challenges than traditional developmental testing. A series of lessons were learned through particular discrepancies and deficiencies discovered through the six month flight test period. Specific deficiencies illustrate where proper planning could ease the difficulties encountered in software testing. Keys to successful developmental software tests include having the appropriate personnel on the test team with the proper equipment and capability. Equally important, inadequate configuration management creates more problems than fixes. Software re-programming can provide faster fixes than traditional developmental test. The flexibility of software programming makes configuration management a challenge as multiple versions become available in a short amount of time. Multiple versions of software heighten the risk of configuration management breakdown during limited amount of available flight tests. Each re-programmed version potentially fixes targeted deficiencies, but can also cause new issues in functional areas already tested. Inherently, regression testing impacts the schedule. Software testing requires a realistic schedule that the author believes should compensate for anticipated problems. Data collection, reduction, and analysis always prove to be valuable in developmental testing. A solid instrumentation plan for data collection from all parties involved in flight tests, especially data link network tests, are critical for trouble shooting discovered deficiencies. Software testing is relatively new to the developmental test world and can be seen as the way of the future. Software upgrades lure program managers into a potentially cost effective option in the face of aging avionics systems. With realistic planning and configuration management, the cost and performance effectiveness of software upgrades and development is more likely to become realized