Semantic segmentation priors for object discovery

© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Reliable object discovery in realistic indoor scenes is a necessity for many computer vision and service robot applications. In these scenes, semantic segmentation methods have made huge advances in recent years. Such methods can provide useful prior information for object discovery by removing false positives and by delineating object boundaries. We propose a novel method that combines bottom-up object discovery and semantic priors for producing generic object candidates in RGB-D images. We use a deep learning method for semantic segmentation to classify colour and depth superpixels into meaningful categories. Separately for each category, we use saliency to estimate the location and scale of objects, and superpixels to find their precise boundaries. Finally, object candidates of all categories are combined and ranked. We evaluate our approach on the NYU Depth V2 dataset and show that we outperform other state-of-the-art object discovery methods in terms of recall.Peer ReviewedPostprint (author's final draft

Saliency Methods for Object Discovery Based on Image and Depth Segmentation

Object discovery is a recent paradigm in computer and robotic vision where the process of interpreting an image starts by proposing a set of candidate regions that potentially correspond to objects; these candidates can be validated later on by object recognition modules or by robot interaction. In this thesis, we propose a novel method for object discovery that works on single RGB-D images and aims at achieving higher recall than current state-of-the-art methods with fewer candidates. Our approach uses saliency as a cue to roughly estimate the location and extent of the objects, and segmentation processes in order to identify the candidates' precise boundaries. We investigate the performance of four different segmentation methods based on colour, depth, an early and a late fusion of colour and depth, and conclude that the late fusion is the most successful. The object candidates are sorted according to a novel ranking strategy based on a combination of features such as 3D convexity and saliency. We evaluate our method and compare it to other state-of-the-art approaches in object discovery on challenging real world sequences from three different public datasets containing a high degree of clutter. The results show that our approach consistently outperforms the other methods. In the second part of this thesis, we turn to streams of images. Here, our goal is to generate as few object candidates per frame as necessary in order to find as many objects as possible throughout the sequence. Therefore, we propose to extend our object discovery system with a so called spatial inhibition of return mechanism to inhibit object candidates that correspond to objects that have already been generated in the past. The challenge here is to inhibit the candidates consistently with viewpoint change, and therefore, we root our inhibition of return mechanism in 3D spatial coordinates. In the final part of this thesis we show an application of our object discovery method to the task of salient object segmentation. The results show that our method achieves state-of-the-art performance

The First Brown Dwarf Discovered by the Backyard Worlds: Planet 9 Citizen Science Project

The Wide-field Infrared Survey Explorer (WISE) is a powerful tool for finding nearby brown dwarfs and searching for new planets in the outer solar system, especially with the incorporation of NEOWISE and NEOWISE-Reactivation data. So far, searches for brown dwarfs in WISE data have yet to take advantage of the full depth of the WISE images. To efficiently search this unexplored space via visual inspection, we have launched a new citizen science project, called "Backyard Worlds: Planet 9," which asks volunteers to examine short animations composed of difference images constructed from time-resolved WISE coadds. We report the discovery of the first new substellar object found by this project, WISEA J110125.95+540052.8, a T5.5 brown dwarf located approximately 34 pc from the Sun with a total proper motion of $\sim$0.7 as yr$^{-1}$. WISEA J110125.95+540052.8 has a WISE $W2$ magnitude of $W2=15.37 \pm 0.09$, this discovery demonstrates the ability of citizen scientists to identify moving objects via visual inspection that are 0.9 magnitudes fainter than the $W2$ single-exposure sensitivity, a threshold that has limited prior motion-based brown dwarf searches with WISE.Comment: 9 pages, 4 figures, 1 table. Accepted for publication in the Astrophysical Journal Letter

StampNet: unsupervised multi-class object discovery

Unsupervised object discovery in images involves uncovering recurring patterns that define objects and discriminates them against the background. This is more challenging than image clustering as the size and the location of the objects are not known: this adds additional degrees of freedom and increases the problem complexity. In this work, we propose StampNet, a novel autoencoding neural network that localizes shapes (objects) over a simple background in images and categorizes them simultaneously. StampNet consists of a discrete latent space that is used to categorize objects and to determine the location of the objects. The object categories are formed during the training, resulting in the discovery of a fixed set of objects. We present a set of experiments that demonstrate that StampNet is able to localize and cluster multiple overlapping shapes with varying complexity including the digits from the MNIST dataset. We also present an application of StampNet in the localization of pedestrians in overhead depth-maps

ULAS J234311.93-005034.0: A gravitational lens system selected from UKIDSS and SDSS

We report the discovery of a new gravitational lens system. This object, ULAS J234311.93-005034.0, is the first to be selected by using the new UKIRT Infrared Deep Sky Survey (UKIDSS), together with the Sloan Digital Sky Survey (SDSS). The ULAS J234311.93-005034.0 system contains a quasar at redshift 0.788 which is doubly imaged, with separation 1.4". The two quasar images have the same redshift and similar, though not identical, spectra. The lensing galaxy is detected by subtracting point-spread functions from R-band images taken with the Keck telescope. The lensing galaxy can also be detected by subtracting the spectra of the A and B images, since more of the galaxy light is likely to be present in the latter. No redshift is determined from the galaxy, although the shape of its spectrum suggests a redshift of about 0.3. The object's lens status is secure, due to the identification of two objects with the same redshift together with a lensing galaxy. Our imaging suggests that the lens is found in a cluster environment, in which candidate arc-like structures, that require confirmation, are visible in the vicinity. Further discoveries of lenses from the UKIDSS survey are likely as part of this programme, due to the depth of UKIDSS and its generally good seeing conditions.Comment: Accepted by MNRA

RGB-D datasets using microsoft kinect or similar sensors: a survey

RGB-D data has turned out to be a very useful representation of an indoor scene for solving fundamental computer vision problems. It takes the advantages of the color image that provides appearance information of an object and also the depth image that is immune to the variations in color, illumination, rotation angle and scale. With the invention of the low-cost Microsoft Kinect sensor, which was initially used for gaming and later became a popular device for computer vision, high quality RGB-D data can be acquired easily. In recent years, more and more RGB-D image/video datasets dedicated to various applications have become available, which are of great importance to benchmark the state-of-the-art. In this paper, we systematically survey popular RGB-D datasets for different applications including object recognition, scene classification, hand gesture recognition, 3D-simultaneous localization and mapping, and pose estimation. We provide the insights into the characteristics of each important dataset, and compare the popularity and the difficulty of those datasets. Overall, the main goal of this survey is to give a comprehensive description about the available RGB-D datasets and thus to guide researchers in the selection of suitable datasets for evaluating their algorithms

3D Object Discovery and Modeling Using Single RGB-D Images Containing Multiple Object Instances

Unsupervised object modeling is important in robotics, especially for handling a large set of objects. We present a method for unsupervised 3D object discovery, reconstruction, and localization that exploits multiple instances of an identical object contained in a single RGB-D image. The proposed method does not rely on segmentation, scene knowledge, or user input, and thus is easily scalable. Our method aims to find recurrent patterns in a single RGB-D image by utilizing appearance and geometry of the salient regions. We extract keypoints and match them in pairs based on their descriptors. We then generate triplets of the keypoints matching with each other using several geometric criteria to minimize false matches. The relative poses of the matched triplets are computed and clustered to discover sets of triplet pairs with similar relative poses. Triplets belonging to the same set are likely to belong to the same object and are used to construct an initial object model. Detection of remaining instances with the initial object model using RANSAC allows to further expand and refine the model. The automatically generated object models are both compact and descriptive. We show quantitative and qualitative results on RGB-D images with various objects including some from the Amazon Picking Challenge. We also demonstrate the use of our method in an object picking scenario with a robotic arm