Semantic Visual Localization

Robust visual localization under a wide range of viewing conditions is a fundamental problem in computer vision. Handling the difficult cases of this problem is not only very challenging but also of high practical relevance, e.g., in the context of life-long localization for augmented reality or autonomous robots. In this paper, we propose a novel approach based on a joint 3D geometric and semantic understanding of the world, enabling it to succeed under conditions where previous approaches failed. Our method leverages a novel generative model for descriptor learning, trained on semantic scene completion as an auxiliary task. The resulting 3D descriptors are robust to missing observations by encoding high-level 3D geometric and semantic information. Experiments on several challenging large-scale localization datasets demonstrate reliable localization under extreme viewpoint, illumination, and geometry changes

Semantic Cross-View Matching

Matching cross-view images is challenging because the appearance and viewpoints are significantly different. While low-level features based on gradient orientations or filter responses can drastically vary with such changes in viewpoint, semantic information of images however shows an invariant characteristic in this respect. Consequently, semantically labeled regions can be used for performing cross-view matching. In this paper, we therefore explore this idea and propose an automatic method for detecting and representing the semantic information of an RGB image with the goal of performing cross-view matching with a (non-RGB) geographic information system (GIS). A segmented image forms the input to our system with segments assigned to semantic concepts such as traffic signs, lakes, roads, foliage, etc. We design a descriptor to robustly capture both, the presence of semantic concepts and the spatial layout of those segments. Pairwise distances between the descriptors extracted from the GIS map and the query image are then used to generate a shortlist of the most promising locations with similar semantic concepts in a consistent spatial layout. An experimental evaluation with challenging query images and a large urban area shows promising results

Estimation of leaf area index and its sunlit portion from DSCOVR EPIC data: theoretical basis

This paper presents the theoretical basis of the algorithm designed for the generation of leaf area index and diurnal course of its sunlit portion from NASA's Earth Polychromatic Imaging Camera (EPIC) onboard NOAA's Deep Space Climate Observatory (DSCOVR). The Look-up-Table (LUT) approach implemented in the MODIS operational LAI/FPAR algorithm is adopted. The LUT, which is the heart of the approach, has been significantly modified. First, its parameterization incorporates the canopy hot spot phenomenon and recent advances in the theory of canopy spectral invariants. This allows more accurate decoupling of the structural and radiometric components of the measured Bidirectional Reflectance Factor (BRF), improves scaling properties of the LUT and consequently simplifies adjustments of the algorithm for data spatial resolution and spectral band compositions. Second, the stochastic radiative transfer equations are used to generate the LUT for all biome types. The equations naturally account for radiative effects of the three-dimensional canopy structure on the BRF and allow for an accurate discrimination between sunlit and shaded leaf areas. Third, the LUT entries are measurable, i.e., they can be independently derived from both below canopy measurements of the transmitted and above canopy measurements of reflected radiation fields. This feature makes possible direct validation of the LUT, facilitates identification of its deficiencies and development of refinements. Analyses of field data on canopy structure and leaf optics collected at 18 sites in the Hyytiälä forest in southern boreal zone in Finland and hyperspectral images acquired by the EO-1 Hyperion sensor support the theoretical basis.Shared Services Center NAS

Towards Personalized Image Retrieval

International audienceThis paper describes an approach to personalized image indexing and retrieval. To tackle the issue of subjectivity in Content-Based Image Retrieval (CBIR), users can define their own indexing vocabulary and make the system learn it. These indexing concepts may be both local (objects) and global (image ategories). The system guides the user in the selection of relevant training examples. Concept learning in the system is incremental and hierarchical: global concepts are built upon local concepts as well as low-level features. Similarity measures tuning is used to emphasize relevant features for a given concept. To illustrate the potential of this approach, an implementation of this model has been developed; preliminary results are given in this paper

Describing Textures in the Wild

Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.Comment: 13 pages; 12 figures Fixed misplaced affiliatio

Learning to See the Wood for the Trees: Deep Laser Localization in Urban and Natural Environments on a CPU

Localization in challenging, natural environments such as forests or woodlands is an important capability for many applications from guiding a robot navigating along a forest trail to monitoring vegetation growth with handheld sensors. In this work we explore laser-based localization in both urban and natural environments, which is suitable for online applications. We propose a deep learning approach capable of learning meaningful descriptors directly from 3D point clouds by comparing triplets (anchor, positive and negative examples). The approach learns a feature space representation for a set of segmented point clouds that are matched between a current and previous observations. Our learning method is tailored towards loop closure detection resulting in a small model which can be deployed using only a CPU. The proposed learning method would allow the full pipeline to run on robots with limited computational payload such as drones, quadrupeds or UGVs.Comment: Accepted for publication at RA-L/ICRA 2019. More info: https://ori.ox.ac.uk/esm-localizatio