A model-based approach for detection of runways and other objects in image sequences acquired using an on-board camera

This research was initiated as a part of the Advanced Sensor and Imaging System Technology (ASSIST) program at NASA Langley Research Center. The primary goal of this research is the development of image analysis algorithms for the detection of runways and other objects using an on-board camera. Initial effort was concentrated on images acquired using a passive millimeter wave (PMMW) sensor. The images obtained using PMMW sensors under poor visibility conditions due to atmospheric fog are characterized by very low spatial resolution but good image contrast compared to those images obtained using sensors operating in the visible spectrum. Algorithms developed for analyzing these images using a model of the runway and other objects are described in Part 1 of this report. Experimental verification of these algorithms was limited to a sequence of images simulated from a single frame of PMMW image. Subsequent development and evaluation of algorithms was done using video image sequences. These images have better spatial and temporal resolution compared to PMMW images. Algorithms for reliable recognition of runways and accurate estimation of spatial position of stationary objects on the ground have been developed and evaluated using several image sequences. These algorithms are described in Part 2 of this report. A list of all publications resulting from this work is also included

Real-time Accurate Runway Detection based on Airborne Multi-sensors Fusion

Existing methods of runway detection are more focused on image processing for remote sensing images based on computer vision techniques. However, these algorithms are too complicated and time-consuming to meet the demand for real-time airborne application. This paper proposes a novel runway detection method based on airborne multi-sensors data fusion which works in a coarse-to-fine hierarchical architecture. At the coarse layer, a vision projection model from world coordinate system to image coordinate system is built by fusing airborne navigation data and forward-looking sensing images, then a runway region of interest (ROI) is extracted from a whole image by the model. Furthermore, EDLines which is a real-time line segments detector is applied to extract straight line segments from ROI at the fine layer, and fragmented line segments generated by EDLines are linked into two long runway lines. Finally, some unique runway features (e.g. vanishing point and runway direction) are used to recognise airport runway. The proposed method is tested on an image dataset provided by a flight simulation system. The experimental results show that the method has advantages in terms of speed, recognition rate and false alarm rate

Aspects of Synthetic Vision Display Systems and the Best Practices of the NASA's SVS Project

NASA s Synthetic Vision Systems (SVS) Project conducted research aimed at eliminating visibility-induced errors and low visibility conditions as causal factors in civil aircraft accidents while enabling the operational benefits of clear day flight operations regardless of actual outside visibility. SVS takes advantage of many enabling technologies to achieve this capability including, for example, the Global Positioning System (GPS), data links, radar, imaging sensors, geospatial databases, advanced display media and three dimensional video graphics processors. Integration of these technologies to achieve the SVS concept provides pilots with high-integrity information that improves situational awareness with respect to terrain, obstacles, traffic, and flight path. This paper attempts to emphasize the system aspects of SVS - true systems, rather than just terrain on a flight display - and to document from an historical viewpoint many of the best practices that evolved during the SVS Project from the perspective of some of the NASA researchers most heavily involved in its execution. The Integrated SVS Concepts are envisagements of what production-grade Synthetic Vision systems might, or perhaps should, be in order to provide the desired functional capabilities that eliminate low visibility as a causal factor to accidents and enable clear-day operational benefits regardless of visibility conditions

Detection of Obstacles in Monocular Image Sequences

The ability to detect and locate runways/taxiways and obstacles in images captured using on-board sensors is an essential first step in the automation of low-altitude flight, landing, takeoff, and taxiing phase of aircraft navigation. Automation of these functions under different weather and lighting situations, can be facilitated by using sensors of different modalities. An aircraft-based Synthetic Vision System (SVS), with sensors of different modalities mounted on-board, complements the current ground-based systems in functions such as detection and prevention of potential runway collisions, airport surface navigation, and landing and takeoff in all weather conditions. In this report, we address the problem of detection of objects in monocular image sequences obtained from two types of sensors, a Passive Millimeter Wave (PMMW) sensor and a video camera mounted on-board a landing aircraft. Since the sensors differ in their spatial resolution, and the quality of the images obtained using these sensors is not the same, different approaches are used for detecting obstacles depending on the sensor type. These approaches are described separately in two parts of this report. The goal of the first part of the report is to develop a method for detecting runways/taxiways and objects on the runway in a sequence of images obtained from a moving PMMW sensor. Since the sensor resolution is low and the image quality is very poor, we propose a model-based approach for detecting runways/taxiways. We use the approximate runway model and the position information of the camera provided by the Global Positioning System (GPS) to define regions of interest in the image plane to search for the image features corresponding to the runway markers. Once the runway region is identified, we use histogram-based thresholding to detect obstacles on the runway and regions outside the runway. This algorithm is tested using image sequences simulated from a single real PMMW image

Analytical evaluation of ILM sensors, volume 1

The functional requirements and operating environment constraints are defined for an independent landing monitor ILM which provides the flight crew with an independent assessment of the operation of the primary automatic landing system. The capabilities of radars, TV, forward looking infrared radiometers, multilateration, microwave radiometers, interferometers, and nuclear sensing concepts to meet the ILM conditions are analyzed. The most critical need for the ILM appears in the landing sequence from 1000 to 2000 meters from threshold through rollout. Of the sensing concepts analyzed, the following show potential of becoming feasible ILM's: redundant microwave landings systems, precision approach radar, airborne triangulation radar, multilateration with radar altimetry, and nuclear sensing

Cognitively-Engineered Multisensor Data Fusion Systems for Military Applications

The fusion of imagery from multiple sensors is a field of research that has been gaining prominence in the scientific community in recent years. The technical aspects of combining multisensory information have been and are currently being studied extensively. However, the cognitive aspects of multisensor data fusion have not received so much attention. Prior research in the field of cognitive engineering has shown that the cognitive aspects of any human-machine system should be taken into consideration in order to achieve systems that are both safe and useful. The goal of this research was to model how humans interpret multisensory data, and to evaluate the value of a cognitively-engineered multisensory data fusion system as an effective, time-saving means of presenting information in high- stress situations. Specifically, this research used principles from cognitive engineering to design, implement, and evaluate a multisensor data fusion system for pilots in high-stress situations. Two preliminary studies were performed, and concurrent protocol analysis was conducted to determine how humans interpret and mentally fuse information from multiple sensors in both low- and high-stress environments. This information was used to develop a model for human processing of information from multiple data sources. This model was then implemented in the development of algorithms for fusing imagery from several disparate sensors (visible and infrared). The model and the system as a whole were empirically evaluated in an experiment with fighter pilots in a simulated combat environment. The results show that the model is an accurate depiction of how humans interpret information from multiple disparate sensors, and that the algorithms show promise for assisting fighter pilots in quicker and more accurate target identification

Evaluation of Fused Synthetic and Enhanced Vision Display Concepts for Low-Visibility Approach and Landing

NASA is developing revolutionary crew-vehicle interface technologies that strive to proactively overcome aircraft safety barriers that would otherwise constrain the full realization of the next generation air transportation system. A piloted simulation experiment was conducted to evaluate the complementary use of Synthetic and Enhanced Vision technologies. Specific focus was placed on new techniques for integration and/or fusion of Enhanced and Synthetic Vision and its impact within a two-crew flight deck during low-visibility approach and landing operations. Overall, the experimental data showed that significant improvements in situation awareness, without concomitant increases in workload and display clutter, could be provided by the integration and/or fusion of synthetic and enhanced vision technologies for the pilot-flying and the pilot-not-flying. Improvements in lateral path control performance were realized when the Head-Up Display concepts included a tunnel, independent of the imagery (enhanced vision or fusion of enhanced and synthetic vision) presented with it. During non-normal operations, the ability of the crew to handle substantial navigational errors and runway incursions were neither improved nor adversely impacted by the display concepts. The addition of Enhanced Vision may not, of itself, provide an improvement in runway incursion detection without being specifically tailored for this application

Proceedings of the Augmented VIsual Display (AVID) Research Workshop

The papers, abstracts, and presentations were presented at a three day workshop focused on sensor modeling and simulation, and image enhancement, processing, and fusion. The technical sessions emphasized how sensor technology can be used to create visual imagery adequate for aircraft control and operations. Participants from industry, government, and academic laboratories contributed to panels on Sensor Systems, Sensor Modeling, Sensor Fusion, Image Processing (Computer and Human Vision), and Image Evaluation and Metrics

Investigation of advanced navigation and guidance system concepts for all-weather rotorcraft operations

Results are presented of a survey conducted of active helicopter operators to determine the extent to which they wish to operate in IMC conditions, the visibility limits under which they would operate, the revenue benefits to be gained, and the percent of aircraft cost they would pay for such increased capability. Candidate systems were examined for capability to meet the requirements of a mission model constructed to represent the modes of flight normally encountered in low visibility conditions. Recommendations are made for development of high resolution radar, simulation of the control display system for steep approaches, and for development of an obstacle sensing system for detecting wires. A cost feasibility analysis is included

Airborne Wind Shear Detection and Warning Systems: Third Combined Manufacturers' and Technologists' Conference, part 2

The Third Combined Manufacturers' and Technologists' Conference was held in Hampton, Va., on October 16-18, 1990. The purpose of the meeting was to transfer significant on-going results of the NASA/FAA joint Airborne Wind Shear Program to the technical industry and to pose problems of current concern to the combined group. It also provided a forum for manufacturers to review forward-look technology concepts and for technologists to gain an understanding of the problems encountered by the manufacturers during the development of airborne equipment and the FAA certification requirements