Verification, Analytical Validation, and Clinical Validation (V3): The Foundation of Determining Fit-for-Purpose for Biometric Monitoring Technologies (BioMeTs)

Digital medicine is an interdisciplinary field, drawing together stakeholders with expertize in engineering, manufacturing, clinical science, data science, biostatistics, regulatory science, ethics, patient advocacy, and healthcare policy, to name a few. Although this diversity is undoubtedly valuable, it can lead to confusion regarding terminology and best practices. There are many instances, as we detail in this paper, where a single term is used by different groups to mean different things, as well as cases where multiple terms are used to describe essentially the same concept. Our intent is to clarify core terminology and best practices for the evaluation of Biometric Monitoring Technologies (BioMeTs), without unnecessarily introducing new terms. We focus on the evaluation of BioMeTs as fit-for-purpose for use in clinical trials. However, our intent is for this framework to be instructional to all users of digital measurement tools, regardless of setting or intended use. We propose and describe a three-component framework intended to provide a foundational evaluation framework for BioMeTs. This framework includes (1) verification, (2) analytical validation, and (3) clinical validation. We aim for this common vocabulary to enable more effective communication and collaboration, generate a common and meaningful evidence base for BioMeTs, and improve the accessibility of the digital medicine field

Validation and verification of expert systems

Validation and verification (V&V) are procedures used to evaluate system structure or behavior with respect to a set of requirements. Although expert systems are often developed as a series of prototypes without requirements, it is not possible to perform V&V on any system for which requirements have not been prepared. In addition, there are special problems associated with the evaluation of expert systems that do not arise in the evaluation of conventional systems, such as verification of the completeness and accuracy of the knowledge base. The criticality of most NASA missions make it important to be able to certify the performance of the expert systems used to support these mission. Recommendations for the most appropriate method for integrating V&V into the Expert System Development Methodology (ESDM) and suggestions for the most suitable approaches for each stage of ESDM development are presented

Testing in the incremental design and development of complex products

Testing is an important aspect of design and development which consumes significant time and resource in many companies. However, it has received less research attention than many other activities in product development, and especially, very few publications report empirical studies of engineering testing. Such studies are needed to establish the importance of testing and inform the development of pragmatic support methods. This paper combines insights from literature study with findings from three empirical studies of testing. The case studies concern incrementally developed complex products in the automotive domain. A description of testing practice as observed in these studies is provided, confirming that testing activities are used for multiple purposes depending on the context, and are intertwined with design from start to finish of the development process, not done after it as many models depict. Descriptive process models are developed to indicate some of the key insights, and opportunities for further research are suggested

Civil Space Technology Initiative: a First Step

This is the first published overview of OAST's focused program, the Civil Space Technology Initiative, (CSTI) which started in FY88. This publication describes the goals, technical approach, current status, and plans for CSTI. Periodic updates are planned

Gateway Modeling and Simulation Plan

This plan institutes direction across the Gateway Program and the Element Projects to ensure that Cross Program M&S are produced in a manner that (1) generate the artifacts required for NASA-STD-7009 compliance, (2) ensures interoperability of M&S exchanged and integrated across the program and, (3) drives integrated development efforts to provide cross-domain integrated simulation of the Gateway elements, space environment, and operational scenarios. This direction is flowed down via contractual enforcement to prime contractors and includes both the GMS requirements specified in this plan and the NASASTD- 7009 derived requirements necessary for compliance. Grounding principles for management of Gateway Models and Simulations (M&S) are derived from the Columbia Accident Investigation Board (CAIB) report and the Diaz team report, A Renewed Commitment to Excellence. As an outcome of these reports, and in response to Action 4 of the Diaz team report, the NASA Standard for Models and Simulations, NASA-STD-7009 was developed. The standard establishes M&S requirements for development and use activities to ensure proper capture and communication of M&S pedigree and credibility information to Gateway program decision makers. Through the course of the Gateway program life cycle M&S will be heavily relied upon to conduct analysis, test products, support operations activities, enable informed decision making and ultimately to certify the Gateway with an acceptable level of risk to crew and mission. To reduce risk associated with M&S influenced decisions, this plan applies the NASA-STD-7009 requirements to produce the artifacts that support credibility assessments and ensure the information is communicated to program management

Integrating IVHM and Asset Design

Integrated Vehicle Health Management (IVHM) describes a set of capabilities that enable effective and efficient maintenance and operation of the target vehicle. It accounts for the collection of data, conducting analysis, and supporting the decision-making process for sustainment and operation. The design of IVHM systems endeavours to account for all causes of failure in a disciplined, systems engineering, manner. With industry striving to reduce through-life cost, IVHM is a powerful tool to give forewarning of impending failure and hence control over the outcome. Benefits have been realised from this approach across a number of different sectors but, hindering our ability to realise further benefit from this maturing technology, is the fact that IVHM is still treated as added on to the design of the asset, rather than being a sub-system in its own right, fully integrated with the asset design. The elevation and integration of IVHM in this way will enable architectures to be chosen that accommodate health ready sub-systems from the supply chain and design trade-offs to be made, to name but two major benefits. Barriers to IVHM being integrated with the asset design are examined in this paper. The paper presents progress in overcoming them, and suggests potential solutions for those that remain. It addresses the IVHM system design from a systems engineering perspective and the integration with the asset design will be described within an industrial design process

Principles and Concepts of Agent-Based Modelling for Developing Geospatial Simulations

The aim of this paper is to outline fundamental concepts and principles of the Agent-Based Modelling (ABM) paradigm, with particular reference to the development of geospatial simulations. The paper begins with a brief definition of modelling, followed by a classification of model types, and a comment regarding a shift (in certain circumstances) towards modelling systems at the individual-level. In particular, automata approaches (e.g. Cellular Automata, CA, and ABM) have been particularly popular, with ABM moving to the fore. A definition of agents and agent-based models is given; identifying their advantages and disadvantages, especially in relation to geospatial modelling. The potential use of agent-based models is discussed, and how-to instructions for developing an agent-based model are provided. Types of simulation / modelling systems available for ABM are defined, supplemented with criteria to consider before choosing a particular system for a modelling endeavour. Information pertaining to a selection of simulation / modelling systems (Swarm, MASON, Repast, StarLogo, NetLogo, OBEUS, AgentSheets and AnyLogic) is provided, categorised by their licensing policy (open source, shareware / freeware and proprietary systems). The evaluation (i.e. verification, calibration, validation and analysis) of agent-based models and their output is examined, and noteworthy applications are discussed.Geographical Information Systems (GIS) are a particularly useful medium for representing model input and output of a geospatial nature. However, GIS are not well suited to dynamic modelling (e.g. ABM). In particular, problems of representing time and change within GIS are highlighted. Consequently, this paper explores the opportunity of linking (through coupling or integration / embedding) a GIS with a simulation / modelling system purposely built, and therefore better suited to supporting the requirements of ABM. This paper concludes with a synthesis of the discussion that has proceeded. The aim of this paper is to outline fundamental concepts and principles of the Agent-Based Modelling (ABM) paradigm, with particular reference to the development of geospatial simulations. The paper begins with a brief definition of modelling, followed by a classification of model types, and a comment regarding a shift (in certain circumstances) towards modelling systems at the individual-level. In particular, automata approaches (e.g. Cellular Automata, CA, and ABM) have been particularly popular, with ABM moving to the fore. A definition of agents and agent-based models is given; identifying their advantages and disadvantages, especially in relation to geospatial modelling. The potential use of agent-based models is discussed, and how-to instructions for developing an agent-based model are provided. Types of simulation / modelling systems available for ABM are defined, supplemented with criteria to consider before choosing a particular system for a modelling endeavour. Information pertaining to a selection of simulation / modelling systems (Swarm, MASON, Repast, StarLogo, NetLogo, OBEUS, AgentSheets and AnyLogic) is provided, categorised by their licensing policy (open source, shareware / freeware and proprietary systems). The evaluation (i.e. verification, calibration, validation and analysis) of agent-based models and their output is examined, and noteworthy applications are discussed.Geographical Information Systems (GIS) are a particularly useful medium for representing model input and output of a geospatial nature. However, GIS are not well suited to dynamic modelling (e.g. ABM). In particular, problems of representing time and change within GIS are highlighted. Consequently, this paper explores the opportunity of linking (through coupling or integration / embedding) a GIS with a simulation / modelling system purposely built, and therefore better suited to supporting the requirements of ABM. This paper concludes with a synthesis of the discussion that has proceeded

Key challenges in agent-based modelling for geo-spatial simulation

Agent-based modelling (ABM) is fast becoming the dominant paradigm in social simulation due primarily to a worldview that suggests that complex systems emerge from the bottom-up, are highly decentralised, and are composed of a multitude of heterogeneous objects called agents. These agents act with some purpose and their interaction, usually through time and space, generates emergent order, often at higher levels than those at which such agents operate. ABM however raises as many challenges as it seeks to resolve. It is the purpose of this paper to catalogue these challenges and to illustrate them using three somewhat different agent-based models applied to city systems. The seven challenges we pose involve: the purpose for which the model is built, the extent to which the model is rooted in independent theory, the extent to which the model can be replicated, the ways the model might be verified, calibrated and validated, the way model dynamics are represented in terms of agent interactions, the extent to which the model is operational, and the way the model can be communicated and shared with others. Once catalogued, we then illustrate these challenges with a pedestrian model for emergency evacuation in central London, a hypothetical model of residential segregation tuned to London data which elaborates the standard Schelling (1971) model, and an agent-based residential location built according to spatial interactions principles, calibrated to trip data for Greater London. The ambiguities posed by this new style of modelling are drawn out as conclusions

1992 NASA Life Support Systems Analysis workshop

The 1992 Life Support Systems Analysis Workshop was sponsored by NASA's Office of Aeronautics and Space Technology (OAST) to integrate the inputs from, disseminate information to, and foster communication among NASA, industry, and academic specialists. The workshop continued discussion and definition of key issues identified in the 1991 workshop, including: (1) modeling and experimental validation; (2) definition of systems analysis evaluation criteria; (3) integration of modeling at multiple levels; and (4) assessment of process control modeling approaches. Through both the 1991 and 1992 workshops, NASA has continued to seek input from industry and university chemical process modeling and analysis experts, and to introduce and apply new systems analysis approaches to life support systems. The workshop included technical presentations, discussions, and interactive planning, with sufficient time allocated for discussion of both technology status and technology development recommendations. Key personnel currently involved with life support technology developments from NASA, industry, and academia provided input to the status and priorities of current and future systems analysis methods and requirements

Integrating IVHM and asset design

Integrated Vehicle Health Management (IVHM) describes a set of capabilities that enable effective and efficient maintenance and operation of the target vehicle. It accounts for the collecting of data, conducting analysis, and supporting the decision-making process for sustainment and operation. The design of IVHM systems endeavours to account for all causes of failure in a disciplined, systems engineering, manner. With industry striving to reduce through-life cost, IVHM is a powerful tool to give forewarning of impending failure and hence control over the outcome. Benefits have been realised from this approach across a number of different sectors but, hindering our ability to realise further benefit from this maturing technology, is the fact that IVHM is still treated as added on to the design of the asset, rather than being a sub-system in its own right, fully integrated with the asset design. The elevation and integration of IVHM in this way will enable architectures to be chosen that accommodate health ready sub-systems from the supply chain and design trade-offs to be made, to name but two major benefits. Barriers to IVHM being integrated with the asset design are examined in this paper. The paper presents progress in overcoming them, and suggests potential solutions for those that remain. It addresses the IVHM system design from a systems engineering perspective and the integration with the asset design will be described within an industrial design process