Angeli GZ

Dierickx P

MacIntosh MJ

English

ePubs: the open archive for STFC research publications

                           This is a copy of the published version, or version of record, available on the publisher’s website. This version does not track changes, errata, or withdrawals on the publisher’s site. Published version information: Citation: MJ MacIntosh, GZ Angeli, ed. and P Dierickx, ed. Test management and reporting using DOORS. In Modeling, Systems Engineering, and Project Management for Astronomy X, Montréal, Canada, 17-23 Jul 2022, (2021): 59   DOI: 10.1117/12.2630015  Copyright 2022 Society of Photo-Optical Instrumentation Engineers (SPIE). One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this publication for a fee or for commercial purposes, and modification of the contents of the publication are prohibited. This version is made available in accordance with publisher policies. Please cite only the published version using the reference above. This is the citation assigned by the publisher at the time of issuing the APV. Please check the publisher’s website for any updates.  This item was retrieved from ePubs, the Open Access archive of the Science and Technology Facilities Council, UK. Please contact epublications@stfc.ac.uk or go to http://epubs.stfc.ac.uk/ for further information and policies. Test management and reporting using DOORS Michael MacIntosh PROCEEDINGS OF SPIESPIEDigitalLibrary.org/conference-proceedings-of-spieTest management and reportingusing DOORSMichael MacIntoshMichael J. MacIntosh, "Test management and reporting using DOORS," Proc.SPIE 12187, Modeling, Systems Engineering, and Project Management forAstronomy X, 121871N (25 August 2022); doi: 10.1117/12.2630015Event: SPIE Astronomical Telescopes + Instrumentation, 2022, Montréal,Québec, CanadaDownloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022  Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      Test management and reporting using DOORS  Michael J MacIntosh* UKRI, STFC, UK Astronomy Technology, Royal Observatory, Blackford Hill, EH9 3HJ ABSTRACT   This paper proposes the usage of IBM Engineering Requirements Management DOORS and IBM Engineering Lifecycle Optimization Publishing for system test management and reporting. DOORS is used to capture the technical requirements, test specifications and test reports as well as establishing links between the requirements and tests specifications. DOORS custom attributes capture additional information such as object type, verification methods and success criteria. Engineering Publishing interacts with the DOORS database to produce document style reports including Requirements Specifications, Test Plans, Test Reports and compliance matrixes. To illustrate the usage of these tools, data from the NIX IR Imager for the ERIS instrument is used. Keywords: Requirements Management, Test Management, Reporting, Compliance Matrix, ERIS  1. INTRODUCTION  Engineering Requirements Management DOORS1 is a requirements management tool which has been used for many years at the UK Astronomy Technology Centre (UK ATC) on several different projects. DOORS stores data in objects contained within modules. As well as the main object text, each object has associated attributes containing additional information (metadata) which can be used both within DOORS and by the Engineering Lifecycle Optimization – Publishing tool. The Publishing tool ingests information from one or more DOORS modules and, via user defined templates, generates document style reports. This paper describes how to use DOORS in conjunction with the Publishing tool to extract data from test specification and test report modules to generate document style reports. To illustrate the usage of the tool, data from the NIX IR Imager for the ERIS3 instrument is used. 2. ATTRIBUTES AND LINKS ARE KEY Custom attributes are used both to store module specific metadata associated with DOORS objects and to control the operation of the Publishing tool. Users can create custom attributes according to the project needs. Before starting a new project, it is good practice to review the information required by DOORS and the Publishing tool and create a corresponding set of custom attributes and types. For example, for a DOORS requirements module it would be useful to define custom attributes for the requirements identifier, requirements type and compliance status. Similarly for a test specifications module, attributes such as test procedure, time to run test and test result would be appropriate. DOORS supports several different attribute types including strings, integers, and enumerations. The latter type is particularly useful when the value of an attribute needs to be constrained to a fixed set of values, for example, a compliance status attribute might have values of Compliant, Partially Compliant and Non-Compliant. Custom types can be created in DOORS to store the enumeration values. To easily distinguish custom attributes and types for the in-built ones it is useful to add a prefix to the name. At the UK ATC ‘SYS- ‘, and ‘TYP-‘are used as prefixes for custom attributes and types, respectively. DOORS custom attributes and types are module specific, so it is important to maintain consistency between related modules. At the UK ATC, custom attributes and types are defined globally for a project and all modules created for that project use the same set of attributes and types even if only a subset of these is used in a particular module. It is then easier to propagate any changes to the custom attributes or types to all the modules in a project.  *mike.macintosh@stfc.ac.uk; https://www.technologysi.stfc.ac.uk/Pages/United-Kingdom-Astronomy-Technology-Centre.aspx  Modeling, Systems Engineering, and Project Management for Astronomy X, edited byGeorge Z. Angeli, Philippe Dierickx, Proc. of SPIE Vol. 12187, 121871N© 2022 SPIE · 0277-786X · doi: 10.1117/12.2630015Proc. of SPIE Vol. 12187  121871N-1Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      For the Publishing tool, attributes are used to control the operation of the tool. One of the key attributes to define is an SYS-ObjType enumeration attribute. This is used to define a type for each DOORS object so that the Publishing tool can process the object appropriately. For example, enumeration values could be defined for heading, figure, table, requirement, and test specification objects amongst other things. Other attributes can be created to define information associated with an object. For example, figure and table objects might have a caption whose value is stored in a caption attribute, for example, SYS-Caption. Figure 1 and Figure 2 show the custom attributes and types, respectively, from the NIX project.   Figure 1. NIX custom attributes   Figure 2. NIX custom types  DOORS has the capability of creating links between any pair of objects in the same or different modules. For example, you might want to associate a particular test specification with one or more requirements by creating links between the corresponding objects. The Publishing tool can follow the DOORS links and pull-out information from the linked object. In the previous example, the Publishing tool could retrieve the details of the requirements linked to a particular test and include this information in a test report. Proc. of SPIE Vol. 12187  121871N-2Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      3. TEST MANAGEMENT  At the UK ATC, test specification modules use the main object text to define the test description and custom attributes to define additional information such as the test procedure and estimated time needed to complete the test. Links are created between each test specification object and the corresponding requirements. These links can be used both internally within DOORS and by the Publishing tool to extract information from the linked objects. Once a particular test has been completed, the result is recorded in the test result attribute (SYS-TestResults) of the corresponding test specification object and details of the test are added to the test results module. Additionally, the compliance status attribute in the corresponding requirements objects can be updated according to the test result. The Publishing tool can be used to create a user defined document style report from the test specification module which can include any required data from the linked requirements objects.  Figure 3 show an extract from the NIX test specifications module. The red arrows to the right of the ‘NIX Test Specifications’ column indicate that links exist to other objects, in this case the links are to the NIX Requirements module.  Figure 3. Extract from the NIX Test Specifications module  Proc. of SPIE Vol. 12187  121871N-3Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      4. TEST REPORTING  The test report module used at the UK ATC is somewhat different from the other DOORS modules in that for each test report a consistent set of objects is created to define the key information associated with a particular test such as the test configuration, results, and conclusions. Custom attributes are used to define the start and end of each such set of objects. It is often necessary to repeat a test, either at different points in the project lifecycle or to address changes in the system architecture or configuration, or to address a failure during testing. The test result module has a SYS-TestRunNo attribute which is used to allow different test runs to be captured. Test runs with a positive SYS-TestRunNo value are output by the Publishing tool to the test report whilst those with a negative value are not. This allows all the test runs to be captured in the DOORS database whilst only reporting the final test results. To make things more manageable a useful tip is to delete, but not purge, older test runs from DOORS. The old test run data is retained in DOORS but is not visible unless the ‘Show deleted objects’ flag is set within the tool. The deleted test results can always be recovered if it is desired to create a report including the older test runs. To create a test report document from the DOORS test report module, a Publishing template is written which defines the modules from which data is to be collected and the logic for processing the data, based on the attributes defined in the DOORS modules. Similar Publishing templates can be created, to generate document style reports from other DOORS modules such as the Requirements Specification or Test Specification and to generate other types of report such as a compliance matrix. The Publishing tool is very flexible and can be used to generate sophisticated document style reports not just from DOORS modules but from a variety of other data sources. The downside of this flexibility is that there is a steep learning curve to become proficient in creating document templates. However, with careful planning, once a document template exists it can be reused on future projects if the attributes are defined and used in the consistent way. To illustrate test report publishing, Figure 4 shows an overview of the process using the NIX dark current test as an example. For clarity Figure 5 shows an expanded version of the NIX dark current report.   Figure 4. Overview of the test report publication process using the NIX dark current as an example. Proc. of SPIE Vol. 12187  121871N-4Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use       Figure 5. Expanded version of the NIX dark current test report. Proc. of SPIE Vol. 12187  121871N-5Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use       The details for part of the document template used to generate the NIX test report is shown in Figure 6. The extract shows the part of the template which creates the test heading in the document and the Requirements verified table. The orange blocks in Figure 6 represent different data sources. The test heading comes from the first data source, namely the NIX Test Specification module and the entries in the Requirements verified table come from the second data source, namely the NIX Requirements module, by following the links from the first data source.   Figure 6. Extract from the NIX Test Report Publishing template  The data for the remainder of example test report shown in Figure 5 comes from the test results DOORS module. Each test report entry is split into a consistent set of sections each starting with a heading object including Configuration, Observation blocks, Results and Conclusions. In turn each of these heading objects can be followed by one or more objects which provide further information related to the test. These objects can contain sub-headings, text, images, or tables. At the UK ATC, images and tables are entered as Word OLE objects as these can be more easily and consistently handled by the Publishing tool compared to the built-in DOORS picture and table objects. Figure 7 shows an extract from the DOORS test results module for NIX which was used to generate the sample test report shown in Figure 5.  Proc. of SPIE Vol. 12187  121871N-6Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use       Figure 7. Extract from the NIX test report module  The start of each test run block is marked by an object with SYS-ObjType set to Information and SYS-InfoType set to TestRun. The end of each block is marked by an object with SYS-ObjType set to Information and SYS-InfoType set to TestRunFiles. The TestRunFiles objects are used to record details of any data files associated with the test. A Publishing template could be written to extract details of all the test files associated with a particular project. The approach described in this paper is very flexible and can easily be modified to suit the needs of a particular project. The key is to define an appropriate set of DOORS attributes and links and create a Publishing tool template to format the document style report in the desired manner. 5. CONCLUSIONS  This paper shows how the DOORS and Publishing tools can be used together to manage test specifications and test reports for a project or group of projects, and subsequently create flexible and versatile document style reports for distribution to interested parties. They key to the effective use of the tools is the creation of a set of custom attributes and associated types which both capture the metadata required in DOORS and guide the Publishing tool to generate the required reports. Additionally, by utilizing the linking capabilities in DOORS, the publishing tool can extract information from the linked objects for inclusion in the report. With careful planning the same Publishing template can be used on many projects if a consistent set of attributes and types are used throughout all the DOORS modules involved.   Proc. of SPIE Vol. 12187  121871N-7Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      REFERENCES [1] IBM, “Engineering Requirements Management DOORS”, 05 May 2021, https://www.ibm.com/docs/en/ermd/9.7.0 (05 May 2021). [2] IBM, “Engineering Lifecycle Optimization Publishing”, 27 Oct 2021, https://www.ibm.com/docs/en/elo p/7.0.2 (27 Oct 2021) [3] Davies, R. et al,” ERIS: revitalising an adaptive optics instrument for the VLT”, arXiv:1807.05089v1 [astro-ph.IM], 2018. Proc. of SPIE Vol. 12187  121871N-8Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 26 Oct 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

Test management and reporting using DOORS

https://epubs.stfc.ac.uk/manifestation/53814317/STFC-APV-2022-091.pdf

Test management and reporting using DOORS

Abstract

Similar works

Full text

Available Versions

ePubs: the open archive for STFC research publications