Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI

We demonstrate a unified approach to rigorous design of safety-critical autonomous systems using the VerifAI toolkit for formal analysis of AI-based systems. VerifAI provides an integrated toolchain for tasks spanning the design process, including modeling, falsification, debugging, and ML component retraining. We evaluate all of these applications in an industrial case study on an experimental autonomous aircraft taxiing system developed by Boeing, which uses a neural network to track the centerline of a runway. We define runway scenarios using the Scenic probabilistic programming language, and use them to drive tests in the X-Plane flight simulator. We first perform falsification, automatically finding environment conditions causing the system to violate its specification by deviating significantly from the centerline (or even leaving the runway entirely). Next, we use counterexample analysis to identify distinct failure cases, and confirm their root causes with specialized testing. Finally, we use the results of falsification and debugging to retrain the network, eliminating several failure cases and improving the overall performance of the closed-loop system.Comment: Full version of a CAV 2020 pape

The engineering roles of requirements and specification

The distinction between requirements and specification is often confused in practice. This obstructs the system validation process, because it is unclear what exactly should be validated, and against what it should be validated. The reference model of Gunter et al. addresses this difficulty by providing a framework within which requirements can be distinguished from specification. It separates world phenomena from machine phenomena. However, it does not explain how the characterization can be used to help assure system validity. In this paper, we enhance the reference model to account for certain key elements that are necessary to expose and clarify the distinction and the link between requirements and specification. We use the enhanced version to present a more refined picture of validity, where validation has two steps that can be undertaken separately. We use this picture to question whether the “what the system will do, not how it will do it ” paradigm is useful in describing how to construct a specification, and propose an alternative. Finally, we present the requirements and specification for an illustrative example based on a runway incursion prevention system, with the ArchiTRIO formal language in a UML-like environment, to show how this might be done in practice.

Synthetic and Enhanced Vision Systems for NextGen (SEVS) Simulation and Flight Test Performance Evaluation

The Synthetic and Enhanced Vision Systems for NextGen (SEVS) simulation and flight tests are jointly sponsored by NASA's Aviation Safety Program, Vehicle Systems Safety Technology project and the Federal Aviation Administration (FAA). The flight tests were conducted by a team of Honeywell, Gulfstream Aerospace Corporation and NASA personnel with the goal of obtaining pilot-in-the-loop test data for flight validation, verification, and demonstration of selected SEVS operational and system-level performance capabilities. Nine test flights (38 flight hours) were conducted over the summer and fall of 2011. The evaluations were flown in Gulfstream.s G450 flight test aircraft outfitted with the SEVS technology under very low visibility instrument meteorological conditions. Evaluation pilots flew 108 approaches in low visibility weather conditions (600 ft to 2400 ft visibility) into various airports from Louisiana to Maine. In-situ flight performance and subjective workload and acceptability data were collected in collaboration with ground simulation studies at LaRC.s Research Flight Deck simulator

Space benefits: The secondary application of aerospace technology in other sectors of the economy

Benefit cases of aerospace technology utilization are presented for manufacturing, transportation, utilities, and health. General, organization, geographic, and field center indexes are included

Implementation of the Recommendations of the Presidential Commission on the Space Shuttle Challenger Accident

This final status report describes the actions taken by NASA in response to the recommendations of the Presidential Commission on the Space Shuttle Challenger Accident (Mission 51-L). The Commission recommendations and NASA's responses to them are summarized in the Executive Summary, which is accompanied by a schedule showing significant program milestones. A detailed discussion of the activities undertaken by NASA to implement each of the nine Commission recommendations is included and other related NASA actions required for safe return to flight are discussed. A copy of the interim plan submitted to the President one year ago and other significant reference documents are included as appendixes

Addressee Errors in ATC Communications: The Call Sign Problem

Communication errors involving aircraft call signs were portrayed in reports of 462 hazardous incidents voluntarily submitted to the ASRS during an approximate four-year period. These errors resulted in confusion, disorder, and uncoordinated traffic conditions and produced the following types of operational anomalies: altitude deviations, wrong-way headings, aborted takeoffs, go arounds, runway incursions, missed crossing altitude restrictions, descents toward high terrain, and traffic conflicts in flight and on the ground. Analysis of the report set resulted in identification of five categories of errors involving call signs: (1) faulty radio usage techniques, (2) call sign loss or smearing due to frequency congestion, (3) confusion resulting from similar sounding call signs, (4) airmen misses of call signs leading to failures to acknowledge or readback, and (5) controller failures regarding confirmation of acknowledgements or readbacks. These error categories are described in detail and several associated hazard mitigating measures that might be aken are considered

Advanced Manned Launch System (AMLS) study

To assure national leadership in space operations and exploration in the future, NASA must be able to provide cost effective and operationally efficient space transportation. Several NASA studies and the joint NASA/DoD Space Transportation Architecture Studies (STAS) have shown the need for a multi-vehicle space transportation system with designs driven by enhanced operations and low costs. NASA is currently studying an advanced manned launch system (AMLS) approach to transport crew and cargo to the Space Station Freedom. Several single and multiple stage systems from air-breathing to all-rocket concepts are being examined in a series of studies potential replacements for the Space Shuttle launch system in the 2000-2010 time frame. Rockwell International Corporation, under contract to the NASA Langley Research Center, has analyzed a two-stage all-rocket concept to determine whether this class of vehicles is appropriate for the AMLS function. The results of the pre-phase A study are discussed

Human factors of flight-deck checklists: The normal checklist

Although the aircraft checklist has long been regarded as the foundation of pilot standardization and cockpit safety, it has escaped the scrutiny of the human factors profession. The improper use, or the non-use, of the normal checklist by flight crews is often cited as the probable cause or at least a contributing factor to aircraft accidents. An attempt is made to analyze the normal checklist, its functions, format, design, length, usage, and the limitations of the humans who must interact with it. The development of the checklist from the certification of a new model to its delivery and use by the customer are discussed. The influence of the government, particularly the FAA Principle Operations Inspector, the manufacturer's philosophy, the airline's culture, and the end user, the pilot, influence the ultimate design and usage of this device. The effects of airline mergers and acquisitions on checklist usage and design are noted. In addition, the interaction between production pressures and checklist usage and checklist management are addressed. Finally, a list of design guidelines for normal checklists is provided

Workshop on Verification and Theorem Proving for Continuous Systems (NetCA Workshop 2005)

Oxford, UK, 26 August 200

Classification and reduction of pilot error

Human error is a primary or contributing factor in about two-thirds of commercial aviation accidents worldwide. With the ultimate goal of reducing pilot error accidents, this contract effort is aimed at understanding the factors underlying error events and reducing the probability of certain types of errors by modifying underlying factors such as flight deck design and procedures. A review of the literature relevant to error classification was conducted. Classification includes categorizing types of errors, the information processing mechanisms and factors underlying them, and identifying factor-mechanism-error relationships. The classification scheme developed by Jens Rasmussen was adopted because it provided a comprehensive yet basic error classification shell or structure that could easily accommodate addition of details on domain-specific factors. For these purposes, factors specific to the aviation environment were incorporated. Hypotheses concerning the relationship of a small number of underlying factors, information processing mechanisms, and error types types identified in the classification scheme were formulated. ASRS data were reviewed and a simulation experiment was performed to evaluate and quantify the hypotheses