Ray Tracing Gems

This book is a must-have for anyone serious about rendering in real time. With the announcement of new ray tracing APIs and hardware to support them, developers can easily create real-time applications with ray tracing as a core component. As ray tracing on the GPU becomes faster, it will play a more central role in real-time rendering. Ray Tracing Gems provides key building blocks for developers of games, architectural applications, visualizations, and more. Experts in rendering share their knowledge by explaining everything from nitty-gritty techniques that will improve any ray tracer to mastery of the new capabilities of current and future hardware. What you'll learn: The latest ray tracing techniques for developing real-time applications in multiple domains Guidance, advice, and best practices for rendering applications with Microsoft DirectX Raytracing (DXR) How to implement high-performance graphics for interactive visualizations, games, simulations, and more Who this book is for: Developers who are looking to leverage the latest APIs and GPU technology for real-time rendering and ray tracing Students looking to learn about best practices in these areas Enthusiasts who want to understand and experiment with their new GPU

Introspective Perception for Mobile Robots

Perception algorithms that provide estimates of their uncertainty are crucial to the development of autonomous robots that can operate in challenging and uncontrolled environments. Such perception algorithms provide the means for having risk-aware robots that reason about the probability of successfully completing a task when planning. There exist perception algorithms that come with models of their uncertainty; however, these models are often developed with assumptions, such as perfect data associations, that do not hold in the real world. Hence the resultant estimated uncertainty is a weak lower bound. To tackle this problem we present introspective perception - a novel approach for predicting accurate estimates of the uncertainty of perception algorithms deployed on mobile robots. By exploiting sensing redundancy and consistency constraints naturally present in the data collected by a mobile robot, introspective perception learns an empirical model of the error distribution of perception algorithms in the deployment environment and in an autonomously supervised manner. In this paper, we present the general theory of introspective perception and demonstrate successful implementations for two different perception tasks. We provide empirical results on challenging real-robot data for introspective stereo depth estimation and introspective visual simultaneous localization and mapping and show that they learn to predict their uncertainty with high accuracy and leverage this information to significantly reduce state estimation errors for an autonomous mobile robot

Viljelykasvien tunnistaminen Sentinel-2 -satelliittikuvien avulla Suomessa

European Union member countries are obligated to control the validity of Common Agricultural Policy subsidy applications. Each member country performs manual inspection for at least 5% of these subsidy applications. This is both expensive and a considerable administrative burden. According to European Union, the crop type identifcation process in Common Agricultural Policy could be carried out using remote sensing or orthophoto imagery for an alternative to physical inspections by competent authorities. Automated crop type identifcation would reduce the costs signifcantly. This master’s thesis addressed the crop identifcation with optical Sentinel-2 satellite imagery in Finland. The aim was to investigate whether it was possible to reliably identify the crop growing in land parcels by using machine learning classifcation methods. This thesis presented an automated approach of identifying crops. Multiple different machine learning classifcation algorithms were trained and tested to find out the most suitable processing method, time period and classifcation algorithm by utilizing the land parcels obtained from the Finnish Agency for Rural Affairs. The developed processing method and most of the tested classifcation algorithms were able to perform relatively well in crop identifcation in cloudy growth period 2017 of Finland. Therefore, the developed method could be applied to different use cases and cloudy weather conditions. The further development and training of the classifcation algorithms could make it possible to utilize this approach in Finland as well as in other EU countries for the Common Agricultural Policy control and possibly in numerous other tasks.Euroopan Unionin jäsenmaiden on noudatettava yhteisen maatalouspolitiikan tarjoamien maataloustukihakemusten valvomista. Jokainen jäsenmaa suorittaa manuaalisen valvonnan vähintään 5% tukihakemuksista. Tämä on sekä kallista, että huomattava hallinnollinen taakka. Euroopan Unionin mukaan viljelykasvin tunnistamisprosessin voisi suorittaa kaukokartoitus- tai ortokuvien avulla paikan päällä tehtävien tarkastuksien sijaan. Automaattinen viljelykasvin tunnistaminen vähentäisi valvonnan kustannuksia huomattavasti. Tämä diplomityö käsitteli viljelykasvien tunnistamista optisten Sentinel-2 satelliittikuvien avulla Suomessa. Tarkoitus oli tutkia, pystyttäisiinkö koneoppimista hyödyntävien luokittelualgortimien avulla tunnistamaan pelloilla kasvavia maatalouskasveja. Tämä diplomityö esitteli automaattisen lähestymistavan viljelykasvin tunnistamiselle. Useaa erilaisia luokittelualgrotimia opetettiin ja testattiin kaikkein sopivimman prosessointimenetelmän, ajankohdan ja luokittelualgoritmin löytämiseksi Suomen oloihin Maaseutuviraston tarjoamien peltolohkojen avulla. Kehitetty prosessointimenetelmä ja suurin osa testatuista luokittelualgoritmeistä suoriutuivat suhteellisen hyvin viljelykasvin tunnistamisesta Suomen vuoden 2017 pilvisellä kasvukaudella. Tämän vuoksi on mahdollista, että kehitetty prosessointimenetelmää voisi hyödyntää myös erilaisissa ilmastoissa ja eri käyttötapauksissa. Jatkokehityksen ja lisäopetuksen avulla luokittelumenetelmät voisivat mahdollistaa tämän lähestymistavan hyödyntämistä yleisen maatalouspolitiikan maataloustukihakemusten valvontaan Suomessa ja myös muissa EU-maissa muiden käyttötapausten lisäksi

S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

Autonomous driving simulation system plays a crucial role in enhancing self-driving data and simulating complex and rare traffic scenarios, ensuring navigation safety. However, traditional simulation systems, which often heavily rely on manual modeling and 2D image editing, struggled with scaling to extensive scenes and generating realistic simulation data. In this study, we present S-NeRF++, an innovative autonomous driving simulation system based on neural reconstruction. Trained on widely-used self-driving datasets such as nuScenes and Waymo, S-NeRF++ can generate a large number of realistic street scenes and foreground objects with high rendering quality as well as offering considerable flexibility in manipulation and simulation. Specifically, S-NeRF++ is an enhanced neural radiance field for synthesizing large-scale scenes and moving vehicles, with improved scene parameterization and camera pose learning. The system effectively utilizes noisy and sparse LiDAR data to refine training and address depth outliers, ensuring high quality reconstruction and novel-view rendering. It also provides a diverse foreground asset bank through reconstructing and generating different foreground vehicles to support comprehensive scenario creation. Moreover, we have developed an advanced foreground-background fusion pipeline that skillfully integrates illumination and shadow effects, further enhancing the realism of our simulations. With the high-quality simulated data provided by our S-NeRF++, we found the perception methods enjoy performance boost on several autonomous driving downstream tasks, which further demonstrate the effectiveness of our proposed simulator

Adaptive Vision Based Scene Registration for Outdoor Augmented Reality

Augmented Reality (AR) involves adding virtual content into real scenes. Scenes are viewed using a Head-Mounted Display or other display type. In order to place content into the user's view of a scene, the user's position and orientation relative to the scene, commonly referred to as their pose, must be determined accurately. This allows the objects to be placed in the correct positions and to remain there when the user moves or the scene changes. It is achieved by tracking the user in relation to their environment using a variety of technology. One technology which has proven to provide accurate results is computer vision. Computer vision involves a computer analysing images and achieving an understanding of them. This may be locating objects such as faces in the images, or in the case of AR, determining the pose of the user. One of the ultimate goals of AR systems is to be capable of operating under any condition. For example, a computer vision system must be robust under a range of different scene types, and under unpredictable environmental conditions due to variable illumination and weather. The majority of existing literature tests algorithms under the assumption of ideal or 'normal' imaging conditions. To ensure robustness under as many circumstances as possible it is also important to evaluate the systems under adverse conditions. This thesis seeks to analyse the effects that variable illumination has on computer vision algorithms. To enable this analysis, test data is required to isolate weather and illumination effects, without other factors such as changes in viewpoint that would bias the results. A new dataset is presented which also allows controlled viewpoint differences in the presence of weather and illumination changes. This is achieved by capturing video from a camera undergoing a repeatable motion sequence. Ground truth data is stored per frame allowing images from the same position under differing environmental conditions, to be easily extracted from the videos. An in depth analysis of six detection algorithms and five matching techniques demonstrates the impact that non-uniform illumination changes can have on vision algorithms. Specifically, shadows can degrade performance and reduce confidence in the system, decrease reliability, or even completely prevent successful operation. An investigation into approaches to improve performance yields techniques that can help reduce the impact of shadows. A novel algorithm is presented that merges reference data captured at different times, resulting in reference data with minimal shadow effects. This can significantly improve performance and reliability when operating on images containing shadow effects. These advances improve the robustness of computer vision systems and extend the range of conditions in which they can operate. This can increase the usefulness of the algorithms and the AR systems that employ them

Multi-view Relighting using a Geometry-Aware Network

International audienceWe propose the first learning-based algorithm that can relight images in a plausible and controllable manner given multiple views of an outdoor scene. In particular, we introduce a geometry-aware neural network that utilizes multiple geometry cues (normal maps, specular direction, etc.) and source and target shadow masks computed from a noisy proxy geometry obtained by multi-view stereo. Our model is a three-stage pipeline: two subnetworks refine the source and target shadow masks, and a third performs the final relighting. Furthermore, we introduce a novel representation for the shadow masks, which we call RGB shadow images. They reproject the colors from all views into the shadowed pixels and enable our network to cope with inacuraccies in the proxy and the non-locality of the shadow casting interactions. Acquiring large-scale multi-view relighting datasets for real scenes is challenging, so we train our network on photorealistic synthetic data. At train time, we also compute a noisy stereo-based geometric proxy, this time from the synthetic renderings. This allows us to bridge the gap between the real and synthetic domains. Our model generalizes well to real scenes. It can alter the illumination of drone footage, image-based renderings, textured mesh reconstructions, and even internet photo collections

Land and cryosphere products from Suomi NPP VIIRS: overview and status

[1] The Visible Infrared Imaging Radiometer Suite (VIIRS) instrument was launched in October 2011 as part of the Suomi National Polar-Orbiting Partnership (S-NPP). The VIIRS instrument was designed to improve upon the capabilities of the operational Advanced Very High Resolution Radiometer and provide observation continuity with NASA's Earth Observing System's Moderate Resolution Imaging Spectroradiometer (MODIS). Since the VIIRS first-light images were received in November 2011, NASA- and NOAA-funded scientists have been working to evaluate the instrument performance and generate land and cryosphere products to meet the needs of the NOAA operational users and the NASA science community. NOAA's focus has been on refining a suite of operational products known as Environmental Data Records (EDRs), which were developed according to project specifications under the National Polar-Orbiting Environmental Satellite System. The NASA S-NPP Science Team has focused on evaluating the EDRs for science use, developing and testing additional products to meet science data needs, and providing MODIS data product continuity. This paper presents to-date findings of the NASA Science Team's evaluation of the VIIRS land and cryosphere EDRs, specifically Surface Reflectance, Land Surface Temperature, Surface Albedo, Vegetation Indices, Surface Type, Active Fires, Snow Cover, Ice Surface Temperature, and Sea Ice Characterization. The study concludes that, for MODIS data product continuity and earth system science, an enhanced suite of land and cryosphere products and associated data system capabilities are needed beyond the EDRs currently available from the VIIRS

Towards A Self-calibrating Video Camera Network For Content Analysis And Forensics

Due to growing security concerns, video surveillance and monitoring has received an immense attention from both federal agencies and private firms. The main concern is that a single camera, even if allowed to rotate or translate, is not sufficient to cover a large area for video surveillance. A more general solution with wide range of applications is to allow the deployed cameras to have a non-overlapping field of view (FoV) and to, if possible, allow these cameras to move freely in 3D space. This thesis addresses the issue of how cameras in such a network can be calibrated and how the network as a whole can be calibrated, such that each camera as a unit in the network is aware of its orientation with respect to all the other cameras in the network. Different types of cameras might be present in a multiple camera network and novel techniques are presented for efficient calibration of these cameras. Specifically: (i) For a stationary camera, we derive new constraints on the Image of the Absolute Conic (IAC). These new constraints are shown to be intrinsic to IAC; (ii) For a scene where object shadows are cast on a ground plane, we track the shadows on the ground plane cast by at least two unknown stationary points, and utilize the tracked shadow positions to compute the horizon line and hence compute the camera intrinsic and extrinsic parameters; (iii) A novel solution to a scenario where a camera is observing pedestrians is presented. The uniqueness of formulation lies in recognizing two harmonic homologies present in the geometry obtained by observing pedestrians; (iv) For a freely moving camera, a novel practical method is proposed for its self-calibration which even allows it to change its internal parameters by zooming; and (v) due to the increased application of the pan-tilt-zoom (PTZ) cameras, a technique is presented that uses only two images to estimate five camera parameters. For an automatically configurable multi-camera network, having non-overlapping field of view and possibly containing moving cameras, a practical framework is proposed that determines the geometry of such a dynamic camera network. It is shown that only one automatically computed vanishing point and a line lying on any plane orthogonal to the vertical direction is sufficient to infer the geometry of a dynamic network. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on synthetic as well as on real data. Applications to path modeling, GPS coordinate estimation, and configuring mixed-reality environment are explored

New Remote Sensing Methods for Detecting and Quantifying Forest Disturbance and Regeneration in the Eastern United States

Forest disturbances, such as wildfires, the southern pine beetle, and the hemlock woolly adelgid, affect millions of hectares of forest in North America with significant implications for forest health and management. This dissertation presents new methods to quantify and monitor disturbance through time in the forests of the eastern United States using remotely sensed imagery from the Landsat family of satellites, detect clouds and cloud-shadow in imagery, generate composite images from the clear-sky regions of multiple images acquired at different times, delineate the extents of disturbance events, identify the years in which they occur, and label those events with an agent and severity. These methods operate at a 30x30 m spatial resolution and a yearly temporal resolution. Overall accuracy for cloud and cloud-shadow detection is 98.7% and is significantly better than a leading method. Overall accuracy for designating a specific space and time as disturbed, stable, or regenerating is 85%, and accuracy for labeling disturbance events with a causal agent ranges from 42% to 90%, depending on agent, with overall accuracy, excluding samples marked as `uncertain\u27, of 81%. Due to the high spatial resolution of the imagery and resulting output, these methods are valuable for managers interested in monitoring specific forested areas. Additionally, these methods enable the discovery and quantification of forest dynamics at larger spatial scales in a way other datasets cannot. Applying these methods over the entire extent of the eastern United States highlands reveals significant differences in disturbance frequency by ecoregion, from less than 1% of forested area per year in the Central Appalachians, to over 5% in the Piedmont. Yearly variations from these means are substantial, with disturbance frequency being twice as high as the mean in some years. Additionally, these analyses reveal that some disturbance agents, such as the southern pine beetle, exhibit periodic dynamics. Finally, although these methods are applied here to the problem of forest disturbance in the eastern United States, the core innovations are easily extended to other locations or even to other applications of landscape change, such as vegetation succession, shifting coastlines, or urbanization