3-D Scene Reconstruction from Aerial Imagery

3-D scene reconstructions derived from Structure from Motion (SfM) and Multi-View Stereo (MVS) techniques were analyzed to determine the optimal reconnaissance flight characteristics suitable for target reconstruction. In support of this goal, a preliminary study of a simple 3-D geometric object facilitated the analysis of convergence angles and number of camera frames within a controlled environment. Reconstruction accuracy measurements revealed at least 3 camera frames and a 6 convergence angle were required to achieve results reminiscent of the original structure. The central investigative effort sought the applicability of certain airborne reconnaissance flight profiles to reconstructing ground targets. The data sets included images collected within a synthetic 3-D urban environment along circular, linear and s-curve aerial flight profiles equipped with agile and non-agile sensors. S-curve and dynamically controlled linear flight paths provided superior results, whereas with sufficient data conditioning and combination of orthogonal flight paths, all flight paths produced quality reconstructions under a wide variety of operational considerations

Roving vehicle motion control Final report

Roving vehicle motion control for unmanned planetary and lunar exploratio

Towards a Common Software/Hardware Methodology for Future Advanced Driver Assistance Systems

The European research project DESERVE (DEvelopment platform for Safe and Efficient dRiVE, 2012-2015) had the aim of designing and developing a platform tool to cope with the continuously increasing complexity and the simultaneous need to reduce cost for future embedded Advanced Driver Assistance Systems (ADAS). For this purpose, the DESERVE platform profits from cross-domain software reuse, standardization of automotive software component interfaces, and easy but safety-compliant integration of heterogeneous modules. This enables the development of a new generation of ADAS applications, which challengingly combine different functions, sensors, actuators, hardware platforms, and Human Machine Interfaces (HMI). This book presents the different results of the DESERVE project concerning the ADAS development platform, test case functions, and validation and evaluation of different approaches. The reader is invited to substantiate the content of this book with the deliverables published during the DESERVE project. Technical topics discussed in this book include:Modern ADAS development platforms;Design space exploration;Driving modelling;Video-based and Radar-based ADAS functions;HMI for ADAS;Vehicle-hardware-in-the-loop validation system

A Semantic-based Framework for Digital Survey of Architectural Heritage

The research aims at developing a framework for semantic-based digital survey of architectural heritage. Rooted in knowledge-based modeling which extracts mathematical constraints of geometry from architectural treatises, as-built information of architecture obtained from image-based modeling is integrated with the ideal model in BIM platform. The knowledge-based modeling transforms the geometry and parametric relation of architectural components from 2D printings to 3D digital models, and create large amount variations based on shape grammar in real time thanks to parametric modeling. It also provides prior knowledge for semantically segmenting unorganized survey data. The emergence of SfM (Structure from Motion) provides access to reconstruct large complex architectural scenes with high flexibility, low cost and full automation, but low reliability of metric accuracy. We solve this problem by combing photogrammetric approaches which consists of camera configuration, image enhancement, and bundle adjustment, etc. Experiments show the accuracy of image-based modeling following our workflow is comparable to that from range-based modeling. We also demonstrate positive results of our optimized approach in digital reconstruction of portico where low-texture-vault and dramatical transition of illumination bring huge difficulties in the workflow without optimization. Once the as-built model is obtained, it is integrated with the ideal model in BIM platform which allows multiple data enrichment. In spite of its promising prospect in AEC industry, BIM is developed with limited consideration of reverse-engineering from survey data. Besides representing the architectural heritage in parallel ways (ideal model and as-built model) and comparing their difference, we concern how to create as-built model in BIM software which is still an open area to be addressed. The research is supposed to be fundamental for research of architectural history, documentation and conservation of architectural heritage, and renovation of existing buildings

Towards a Common Software/Hardware Methodology for Future Advanced Driver Assistance Systems

The European research project DESERVE (DEvelopment platform for Safe and Efficient dRiVE, 2012-2015) had the aim of designing and developing a platform tool to cope with the continuously increasing complexity and the simultaneous need to reduce cost for future embedded Advanced Driver Assistance Systems (ADAS). For this purpose, the DESERVE platform profits from cross-domain software reuse, standardization of automotive software component interfaces, and easy but safety-compliant integration of heterogeneous modules. This enables the development of a new generation of ADAS applications, which challengingly combine different functions, sensors, actuators, hardware platforms, and Human Machine Interfaces (HMI). This book presents the different results of the DESERVE project concerning the ADAS development platform, test case functions, and validation and evaluation of different approaches. The reader is invited to substantiate the content of this book with the deliverables published during the DESERVE project. Technical topics discussed in this book include:Modern ADAS development platforms;Design space exploration;Driving modelling;Video-based and Radar-based ADAS functions;HMI for ADAS;Vehicle-hardware-in-the-loop validation system

Establishing a Quantifiable Model of Whale Shark Avoidance Behaviours to Anthropogenic Impacts in Tourism Encounters to Inform Management Actions

As the world's largest living fish, the whale shark has received much scientific attention in recent years, although despite this a great deal is still unknown on the life history and behavioural ecology of these majestic sharks. Whale shark related tourism has exploded in the last two decades from only a few sites in the 1990s to more than 12 sites internationally, allowing it to become a highly lucrative industry based upon this Vulnerable species. This study assesses the effects of anthropogenic impact on the sharks’ avoidance behaviours within modern day tourism encounters, and provides recommendations on how to control and reduce unnecessary disturbance to the species. By means of stereo-photogrammetry, continuous high definition videos of human-animal interactions were recorded and analyzed for behavioural changes against pre-selected independant variables. The use of Stereo-photogrammetry imagery also allowed for the accumulation of repeatable, proximity measurements of swimmer distance to the shark, permitting more precise and accurate results. Avoidance behaviours of 33 individual whale sharks were monitored during typical tourism encounters (n=75). A total of 192 search hours were documented over the collection periods, which incorporated three-aggregation sites spanning the Indian Ocean (the Seychelles, the Philipines & Mozambique). A generalized linear model demonstrated that proximity of swimmers to the shark was found to be significant (p=0.0295) in explaining the probability of the whale sharks showing disturbed behaviour. A proportional odds plot for proximity was developed to give an indication of the animals disturbance level in tourism interactions. At recommended distances of three metres from the sides of the shark, there is on average a 42% chance of disturbance, while at the distance of four metres from the tail area results showed a 31% chance of overall disturbance. The true estimate for either distance is likely to lie between 22-53% respectively with regards to the uncertainty around the mean predictions. Whale shark tourism is viewed as a potential means of protecting this threatened species, while also providing a sustainable livelihood for local communities and tourism providers. Management recommendations presented offer suggestions on how to tackle concerns over proximity distances and links to disturbance. Additionally judgments for future research endeavors into assessing both the impacts of uncontrolled tourism and participants behaviour

Inertial-aided Visual Perception of Geometry and Semantics

We describe components of a visual perception system to understand the geometry and semantics of the three-dimensional scene by utilizing monocular cameras and inertial measurement units (IMUs). The use of the two sensor modalities is motivated by the wide availability of the camera-IMU sensor packages present in mobile devices from phones to cars, and their complementary sensing capabilities: IMUs can track the motion of the sensor platform over a short period of time accurately, and provide a scaled and gravity-aligned global reference frame, while cameras can capture rich photometric signatures of the scene, and provide relative motion constraints between images up to scale. We first show that visual 3D reconstruction can be improved by leveraging the global orientation frame -- easily inferred from inertials. In the gravity-aligned global orientation frame, a shape prior can be imposed in depth prediction from a single image, where the normal vectors to surfaces of objects of certain classes tend to align with gravity or orthogonal to it. Adding such a prior to baseline methods for monocular depth prediction yields improvements beyond the state-of-the-art and illustrates the power of utilizing inertials in 3D reconstruction. The global reference provided by inertials is not only gravity-aligned but also scaled, which is exploited in depth completion: We describe a method to infer dense metric depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to “self-supervision,” measuring photometric consistency across time, forward-backward pose consistency, and geometric compatibility with the sparse point cloud. We also launch the first visual-inertial + depth dataset (dubbed ``VOID''), which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, and show state-of-the-art performance on it.In addition to dense geometry, the camera-IMU sensor package can also be used to recover the semantics of the scene. We present two methods to augment a point cloud map with class-labeled objects represented in the form of either scaled and oriented bounding boxes or CAD models. The tradeoff of the two shape representation resides in their generality and capability to model detailed structures. While being more generic, 3D bounding boxes fail to model the details of the objects, whereas CAD models preserve the finest shape details but require more computation and are limited to previously seen objects. Nevertheless, both methods populate an unknown environment with 3D objects placed in a Euclidean reference frame inferred causally and on-line using monocular video along with inertial sensors. Besides, both methods include bottom-up and top-down components, whereby deep networks trained for detection provide likelihood scores for object hypotheses provided by a nonlinear filter, whose state serves as memory. We test our methods on KITTI and SceneNN datasets, and also introduce the VISMA dataset, which contains ground truth pose, point-cloud map, and object models, along with time-stamped inertial measurements.To reduce the drift of the visual-inertial SLAM system -- a building block of all the visual perception systems we have built, we introduce an efficient loop closure detection approach based on the idea of hierarchical pooling of image descriptors. We also open-sourced a full-fledged SLAM system equipped with mapping and loop closure capabilities. The code is publicly available at https://github.com/ucla-vision/xivo

Detection of changes through visual alerts and comparisons using a multi-layered display.

The Multi-Layered Displays (MLD) comprise two LCD screens mounted one in front of the other, allowing the presentation of information on both screens. This physical separation produces depth without requiring glasses. This research evaluated the utility of the MLD for change detection tasks, particularly in operational environments. Change Blindness refers to the failure to detect changes when the change happens during a visual disruption. The literature equates these visual disruptions with the types of interruptions that occur regularly in work situations. Change blindness is more likely to occur when operators monitor dynamic situations spread over several screens, when there are popup messages, and when there are frequent interruptions which are likely to block the visual transients that signal a change. Four laboratory experiments were conducted to evaluate the utility of the MLD for change detection tasks. The results from the experiments revealed that, when depth is used as a visual cue, the depth of the MLD has a different effect on the detection of expected changes and unexpected events. When the depth of the MLD is used as a comparison tool, the detection of differences is limited to translation differences in simple stimuli with a white background. These results call into question previous claims made for the MLD regarding operational change detection. In addition, observations and interviews were used to explore whether change blindness occurred in an operational command room. The results suggested that operators develop strategies to recover from interruptions and multitasking. These results call into doubt the wisdom of applying change detection theories to real world operational settings. More importantly, the research serves as a reminder that cognitive limitations found in the laboratory are not always found in real world environments

Detection of changes through visual alerts and comparisons using a multi-layered display

The Multi-Layered Displays (MLD) comprise two LCD screens mounted one in front of the other, allowing the presentation of information on both screens. This physical separation produces depth without requiring glasses. This research evaluated the utility of the MLD for change detection tasks, particularly in operational environments. Change Blindness refers to the failure to detect changes when the change happens during a visual disruption. The literature equates these visual disruptions with the types of interruptions that occur regularly in work situations. Change blindness is more likely to occur when operators monitor dynamic situations spread over several screens, when there are popup messages, and when there are frequent interruptions which are likely to block the visual transients that signal a change. Four laboratory experiments were conducted to evaluate the utility of the MLD for change detection tasks. The results from the experiments revealed that, when depth is used as a visual cue, the depth of the MLD has a different effect on the detection of expected changes and unexpected events. When the depth of the MLD is used as a comparison tool, the detection of differences is limited to translation differences in simple stimuli with a white background. These results call into question previous claims made for the MLD regarding operational change detection. In addition, observations and interviews were used to explore whether change blindness occurred in an operational command room. The results suggested that operators develop strategies to recover from interruptions and multitasking. These results call into doubt the wisdom of applying change detection theories to real world operational settings. More importantly, the research serves as a reminder that cognitive limitations found in the laboratory are not always found in real world environments