Learning to Place New Objects

The ability to place objects in the environment is an important skill for a personal robot. An object should not only be placed stably, but should also be placed in its preferred location/orientation. For instance, a plate is preferred to be inserted vertically into the slot of a dish-rack as compared to be placed horizontally in it. Unstructured environments such as homes have a large variety of object types as well as of placing areas. Therefore our algorithms should be able to handle placing new object types and new placing areas. These reasons make placing a challenging manipulation task. In this work, we propose a supervised learning algorithm for finding good placements given the point-clouds of the object and the placing area. It learns to combine the features that capture support, stability and preferred placements using a shared sparsity structure in the parameters. Even when neither the object nor the placing area is seen previously in the training set, our algorithm predicts good placements. In extensive experiments, our method enables the robot to stably place several new objects in several new placing areas with 98% success-rate; and it placed the objects in their preferred placements in 92% of the cases

Learning to Singulate Objects using a Push Proposal Network

Learning to act in unstructured environments, such as cluttered piles of objects, poses a substantial challenge for manipulation robots. We present a novel neural network-based approach that separates unknown objects in clutter by selecting favourable push actions. Our network is trained from data collected through autonomous interaction of a PR2 robot with randomly organized tabletop scenes. The model is designed to propose meaningful push actions based on over-segmented RGB-D images. We evaluate our approach by singulating up to 8 unknown objects in clutter. We demonstrate that our method enables the robot to perform the task with a high success rate and a low number of required push actions. Our results based on real-world experiments show that our network is able to generalize to novel objects of various sizes and shapes, as well as to arbitrary object configurations. Videos of our experiments can be viewed at http://robotpush.cs.uni-freiburg.deComment: International Symposium on Robotics Research (ISRR) 2017, videos: http://robotpush.cs.uni-freiburg.d

Efficient Belief Propagation for Perception and Manipulation in Clutter

Autonomous service robots are required to perform tasks in common human indoor environments. To achieve goals associated with these tasks, the robot should continually perceive, reason its environment, and plan to manipulate objects, which we term as goal-directed manipulation. Perception remains the most challenging aspect of all stages, as common indoor environments typically pose problems in recognizing objects under inherent occlusions with physical interactions among themselves. Despite recent progress in the field of robot perception, accommodating perceptual uncertainty due to partial observations remains challenging and needs to be addressed to achieve the desired autonomy. In this dissertation, we address the problem of perception under uncertainty for robot manipulation in cluttered environments using generative inference methods. Specifically, we aim to enable robots to perceive partially observable environments by maintaining an approximate probability distribution as a belief over possible scene hypotheses. This belief representation captures uncertainty resulting from inter-object occlusions and physical interactions, which are inherently present in clutterred indoor environments. The research efforts presented in this thesis are towards developing appropriate state representations and inference techniques to generate and maintain such belief over contextually plausible scene states. We focus on providing the following features to generative inference while addressing the challenges due to occlusions: 1) generating and maintaining plausible scene hypotheses, 2) reducing the inference search space that typically grows exponentially with respect to the number of objects in a scene, 3) preserving scene hypotheses over continual observations. To generate and maintain plausible scene hypotheses, we propose physics informed scene estimation methods that combine a Newtonian physics engine within a particle based generative inference framework. The proposed variants of our method with and without a Monte Carlo step showed promising results on generating and maintaining plausible hypotheses under complete occlusions. We show that estimating such scenarios would not be possible by the commonly adopted 3D registration methods without the notion of a physical context that our method provides. To scale up the context informed inference to accommodate a larger number of objects, we describe a factorization of scene state into object and object-parts to perform collaborative particle-based inference. This resulted in the Pull Message Passing for Nonparametric Belief Propagation (PMPNBP) algorithm that caters to the demands of the high-dimensional multimodal nature of cluttered scenes while being computationally tractable. We demonstrate that PMPNBP is orders of magnitude faster than the state-of-the-art Nonparametric Belief Propagation method. Additionally, we show that PMPNBP successfully estimates poses of articulated objects under various simulated occlusion scenarios. To extend our PMPNBP algorithm for tracking object states over continuous observations, we explore ways to propose and preserve hypotheses effectively over time. This resulted in an augmentation-selection method, where hypotheses are drawn from various proposals followed by the selection of a subset using PMPNBP that explained the current state of the objects. We discuss and analyze our augmentation-selection method with its counterparts in belief propagation literature. Furthermore, we develop an inference pipeline for pose estimation and tracking of articulated objects in clutter. In this pipeline, the message passing module with the augmentation-selection method is informed by segmentation heatmaps from a trained neural network. In our experiments, we show that our proposed pipeline can effectively maintain belief and track articulated objects over a sequence of observations under occlusion.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163159/1/kdesingh_1.pd

EXPLORING THE ABILITY TO EMPLOY VIRTUAL 3D ENTITIES OUTDOORS AT RANGES BEYOND 20 METERS

The Army is procuring the Integrated Visual Augmentation System (IVAS) system to enable enhanced night vision, planning, and training capability. One known limitation of the IVAS system is the limited ability to portray virtual entities at far ranges in the outdoors due to light wash out, accurate positioning, and dynamic occlusion. The primary goal of this research was to evaluate fixed three-dimensional (3D) visualizations to support outdoor training for fire teams through squads, requiring target visualizations for 3D non-player characters or vehicles at ranges up to 300 m. Tools employed to achieve outdoor visualizations included GPS locational data with virtual entity placement, and sensors to adjust device light levels. This study was conducted with 20 military test subjects in three scenarios at the Naval Postgraduate School using a HoloLens II. Outdoor location considerations included shadows, background clutter, cars blocking the field of view, and the sun’s positioning. Users provided feedback on identifying the type of object, and the difficulty in finding the object. The results indicate GPS only aided in identification for objects up to 100 m. Animation had a statistically insignificant effect on identification of objects. Employment of software to adjust the light levels of the virtual objects aided in identification of objects at 200 m. This research develops a clearer understanding of requirements to enable the employment of mixed reality in outdoor training.Lieutenant Colonel, United States ArmyApproved for public release. Distribution is unlimited

Elements of design for indoor visualisation

Indoor visualisation has received little attention. Research related to indoor environments have primarily focussed on the data structuring, localisation and navigation components (Zlatanova et al., 2013). Visualisation is an integral component in addressing the diverse array of indoor environments. In simple words, 'What is the most efficient way to visualise the surrounding indoor environment so that the user can concisely understand their surroundings as well as facilitating the process of navigation?' This dissertation proposes a holistic approach that consists of two components. The significance of this approach is that it provides a robust and adaptable method in providing a standard to which indoor visualisation can be referenced against. The first component is a theoretical framework focussing on indoor visualisation and it comprises of principles from several disciplines such as geovisualisation, human-perception theory, spatial cognition, dynamic and 3D environments as well as accommodating emotional processes resulting from human-computer interaction. The second component is based on the theoretical framework and adopts a practical approach towards indoor visualisation. It consists of a set of design properties that can be used for the design of effective indoor visualisations. The framework is referred to as the "Elements of Design" framework. Both these components aim to provide a set of principles and guidelines that can be used as best practices for the design of indoor visualisations. In order to practically demonstrate the holistic indoor visualisation approach, multiple indoor visualisation renderings were developed. The visualisation renderings were represented in a three-dimensional virtual environment from a first-person perspective. Each rendering used the design framework differently. Also, each rendering was graded using a parallel chart that compares how the different visual elements were used per the rendering. The main findings were that the techniques/ renderings that used the visual elements effectively (enhanced human-perception) resulted in better acquisition and construction of knowledge about the surrounding indoor environment