Interactive multiple object learning with scanty human supervision

© 2016. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/We present a fast and online human-robot interaction approach that progressively learns multiple object classifiers using scanty human supervision. Given an input video stream recorded during the human robot interaction, the user just needs to annotate a small fraction of frames to compute object specific classifiers based on random ferns which share the same features. The resulting methodology is fast (in a few seconds, complex object appearances can be learned), versatile (it can be applied to unconstrained scenarios), scalable (real experiments show we can model up to 30 different object classes), and minimizes the amount of human intervention by leveraging the uncertainty measures associated to each classifier.; We thoroughly validate the approach on synthetic data and on real sequences acquired with a mobile platform in indoor and outdoor scenarios containing a multitude of different objects. We show that with little human assistance, we are able to build object classifiers robust to viewpoint changes, partial occlusions, varying lighting and cluttered backgrounds. (C) 2016 Elsevier Inc. All rights reserved.Peer ReviewedPostprint (author's final draft

Autonomous Movement Control of Coaxial Mobile Robot based on Aspect Ratio of Human Face for Public Relation Activity Using Stereo Thermal Camera

In recent years, robots that recognize people around them and provide guidance, information, and monitoring have been attracting attention. The mainstream of conventional human recognition technology is the method using a camera or laser range finder. However, it is difficult to recognize with a camera due to fluctuations in lighting 1), and it is often affected by the recognition environment such as misrecognition 2) with a person's leg and a chair's leg with a laser range finder. Therefore, we propose a human recognition method using a thermal camera that can visualize human heat. This study aims to realize human-following autonomous movement based on human recognition. In addition, the distance from the robot to the person is measured with a stereo thermal camera that uses two thermal cameras. A coaxial two-wheeled robot that is compact and capable of super-credit turning is used as a mobile robot. Finally, we conduct an autonomous movement experiment of a coaxial mobile robot based on human recognition by combining these. We performed human-following experiments on a coaxial two-wheeled robot based on human recognition using a stereo thermal camera and confirmed that it moves appropriately to the location where the recognized person is in multiple use cases (scenarios). However, the accuracy of distance measurement by stereo vision is inferior to that of laser measurement. It is necessary to improve it in the case of movement that requires more accuracy

Perspective distortion modeling for image measurements

A perspective distortion modelling for monocular view that is based on the fundamentals of perspective projection is presented in this work. Perspective projection is considered to be the most ideal and realistic model among others, which depicts image formation in monocular vision. There are many approaches trying to model and estimate the perspective effects in images. Some approaches try to learn and model the distortion parameters from a set of training data that work only for a predefined structure. None of the existing methods provide deep understanding of the nature of perspective problems. Perspective distortions, in fact, can be described by three different perspective effects. These effects are pose, distance and foreshortening. They are the cause of the aberrant appearance of object shapes in images. Understanding these phenomena have long been an interesting topic for artists, designers and scientists. In many cases, this problem has to be necessarily taken into consideration when dealing with image diagnostics, high and accurate image measurement, as well as accurate pose estimation from images. In this work, a perspective distortion model for every effect is developed while elaborating the nature of perspective effects. A distortion factor for every effect is derived, then followed by proposed methods, which allows extracting the true target pose and distance, and correcting image measurements

Vision-Based Object Recognition and 3-D Pose Estimation Using Conic Features

This thesis deals with monocular vision-based object recognition and 3-D pose estimation based on conic features. Conic features including circles and ellipses are frequently observed in many man-made objects in real word as well as have the merit of robustness potentially in feature extraction in vision-based applications. Although the 3-D pose estimation problem of conic features in 3-D space has been studied well since 1990, the previous work has not provided a unique solution completely for full 3-D pose parameters (i.e., 3-orientations and 3-positions) due to complexity from high nonlinearity of a general conic. This thesis, therefore, renews conic features in a new perspective on geometric invariants in both 3-D space and 2-D projective space, incorporating other geometric features with conics. First, as the most essential step in dealing with conics, this thesis shows that the pose parameters of a circular feature in 3-D space can be derived analytically from incorporating a coplanar point. A procedure of pose parameter recovery is described in detail, and its performance is evaluated and discussed in view of pose estimation errors and sensitivity. Second, it is also revealed that the pose of an elliptic feature can be resolved when two coplanar points are incorporated on the basis of the polarity of two points for a conic in 2-D projective space. This thesis proposes a series of algorithms to determine the 3-D pose parameters uniquely, and evaluates the proposed method through a measure of estimation performance and sensitivity depending on point locations. Third, a pair of two conics is dealt with, which is regarded as an extension of the idea of the incorporation scheme to another conic feature from point features. Under the polarity concept, this thesis proves that the problem involving a pair of two conics can be formulated with the problem of one ellipse with two points so that its solution is derived in the same form as in the ellipse case. In order to treat two or more conic objects as well as to deal with an object recognition problem, the rest of thesis concentrates on the theoretical foundation of multiple object recognition. First, some effective modeling approaches are described. A general object model is specially designed to model multiple objects for object recognition and pose recovery in view of spatial geometry. In particular, this thesis defines a pairwise conic model that can describes the geometrical relation between two conics invariantly in 2-D projective space, which consists of a pairwise conic (PC), a pairwise conic invariant (PCI), and a pairwise conic pole (PCP). Based on the two kinds of models, an object learning and recognition system is proposed as a general framework for multiple object recognition. Considering simplicity and flexibility in object learning stage, this thesis introduces a semi-automatic learning scheme to construct the multiple object model from a model image at once. To utilize geometric relations among multiple objects effectively in object recognition, this thesis specifies some feature functions based on the pairwise conic model, and then describes an object recognition method in a fashion of linear-chain conditional random field (CRF). In particular, as a post refinement step of the recognition, a geometric alignment procedure is also proposed in algorithmic details to improve recognition performance against noisy conditions. Last, the multiple object recognition method is evaluated intensively through two practical applications that deal with a place recognition and an elevator button recognition problem for service robots. A series of experiment results supports the effectiveness of the proposed method, maintaining reliable performance against noisy conditions in the presence of perspective distortion and partial object occlusions.Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Research objective and expected contribution . . . . . . . . . . . . . . . . . . 6 1.4 Organization of thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2 3-D Pose Estimation of a Circular Feature 10 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Preliminaries: an elliptic cone in 3-D space and its homogeneous representation in 2-D projective space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.1 Homogeneous representation . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.2 Principal planes of a cone versus diagonalization of a conic matrix Q . 16 2.3 3-D interpretation of a circular feature for 3-D pose estimation . . . . . . . . 19 2.3.1 3-D orientation estimation . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 3-D position estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.3 Composition of homogeneous transformation and discrimination for the unique solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.2 Evaluation of pose estimation performance . . . . . . . . . . . . . . . 29 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3 3-D Pose Estimation of an Elliptic Feature 35 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.2 Interpretation of an elliptic feature with coplanar points in 2-D projective space 38 3.2.1 The minimal number of points for pose estimation . . . . . . . . . . . 39 3.2.2 Analysis of possible constraints for relative positions of two points to an ellipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.3 Feature selection scheme for stable homography estimation . . . . . . 43 3.3 3-D pose estimation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.1 Extraction of triangular features from an elliptic object . . . . . . . . 47 3.3.2 Homography decomposition . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.3 Composition of homogeneous transformation matrix with unique solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4.2 Evaluation of the proposed method . . . . . . . . . . . . . . . . . . . . 54 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4 3-D Pose Estimation of a Pair of Conic Features 61 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.2 3-D pose estimation of a conic feature incorporated with line features . . . . 61 4.3 3-D pose estimation of a conic feature incorporated with another conic feature 63 4.3.1 Some examples of self-polar triangle and invariants . . . . . . . . . . . 65 4.3.2 3-D pose estimation of a pair of coplanar conics . . . . . . . . . . . . . 67 4.3.3 Examples of 3-D pose estimation of a conic feature incorporated with another conic feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Multiple Object Recognition Based on Pairwise Conic Model 77 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Learning of geometric relation of multiple objects . . . . . . . . . . . . . . . . 78 5.3 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.3.1 De_nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.4 Multiple object recognition based on pairwise conic model and conditional random _elds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.4.1 Graphical model for multiple object recognition . . . . . . . . . . . . . 86 5.4.2 Linear-chain conditional random _eld . . . . . . . . . . . . . . . . . . 87 5.4.3 Determination of low-level feature functions for multiple object recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.4.4 Range selection trick for e_ciently computing the costs of low-level feature functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.4.5 Evaluation of observation sequence . . . . . . . . . . . . . . . . . . . . 93 5.4.6 Object recognition based on hierarchical CRF . . . . . . . . . . . . . . 95 5.5 Geometric alignment algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6 Application to Place Recognition for Service Robots 105 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 6.2 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.2.1 Detection of 2-D geometric shapes . . . . . . . . . . . . . . . . . . . . 107 6.2.2 Examples of shape feature extraction . . . . . . . . . . . . . . . . . . . 109 6.3 Object modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6.3.1 A place model that describes multiple landmark objects . . . . . . . . 112 6.3.2 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.3.3 Incorporation of non-conic features with a pairwise conic model . . . . 114 6.4 Place learning and recognition system . . . . . . . . . . . . . . . . . . . . . . 121 6.4.1 HCRF-based recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.5 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.5.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.5.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 7 Application to Elevator Button Recognition 136 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 7.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 7.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7.2 Object modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 7.2.1 Geometric model for multiple button objects . . . . . . . . . . . . . . 140 7.2.2 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 7.3 Learning and recognition system . . . . . . . . . . . . . . . . . . . . . . . . . 141 7.3.1 Button object learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 7.3.2 CRF-based recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 7.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.4.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.4.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 151 7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 8 Concluding remarks 159 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 8.2 Further work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 References 161 Summary (in Korean) 16

Redundant Multi-Modal Integration of Machine Vision and Programmable Mechanical Manipulation for Scene Segmentation

The main idea in this paper is that one cannot discern the part-whole relationship of three-dimensional objects in a passive mode without a great deal of a priori information. Perceptual activity is exploratory, probing and searching. Physical scene segmentation is the first step in active perception. The task of perception is greatly simplified if one has to deal with only one object at a time. This work adapts the non-deterministic Turing machine model and develops strategies to control the interaction between sensors and actions for physical segmentation. Scene segmentation is formulated in graph theoretic terms as a graph generation/decomposition problem. The isomorphism between manipulation actions and graph decomposition operations is defined. The non-contact sensors generate the directed graphs representing the spatial relations among surface regions. The manipulator decomposes these graphs under contact sensor supervision. Assuming a finite number of sensors and actions and a goal state, that is reachable and measurable with the available sensors, the control strategies converge. This was experimentally verified in a real, noisy, and dynamic environment

Segmentation via Manipulation

The motivation for this paper is the observation that a static scene that contains more than one object/part most of the time cannot be segmented only by vision or in general by any non-contact sensing. Exception to this is only the case when the objects/parts are physically separated so that the non-contact sensor can measure this separation or one knows a great deal of a priori knowledge about the objects (their geometry, material, etc.). We assume no such knowledge is available. Instead, we assume that the scene is reachable with a manipulator. Hence the problem represents a class of problems of segmentation that occur on an assembly line, bin picking, organizing a desk top, and the like. What are the typical properties of this class of problems? 1. The objects are rigid. Their size and weight is such that they are manipulable with an suitable end effector. Their numbers on the scene is such that in a reasonable time each piece can be examined and manipulated, i.e the complexity of the scene is bounded. 2. The scene is accessible to the sensors, i.e the whole scene is visible, although some parts may be occluded, and reachable by the manipulator. 3. There is a well defined goal which is detectable by the available sensors. Specifically the goal maybe: an empty scene, or an organized/ ordered scene

Adaptive sensor-fusion of depth and color information for cognitive robotics

Proceedings of: 2011 IEEE International Conference on Robotics and Biomimetics (ROBIO), December 7-11, 2011, Phuket (Thailand)The presented work goes one step further than only combining data from different sensors. The corresponding points of an image and a 3D point cloud are determined through calibration. Color information is thereby assigned to every voxel in the overlapping area of a stereo camera system and a laser range finder. Then we analyze the image and search for the locations, which are especially susceptible to errors by both sensors. Depending on the ascertained situation, we try to correct or minimize errors. By analyzing and interpreting the images as well as removing errors we create an adaptive tool which improves multi-sensor fusion. This allows us to correct the fused data and to perfect the multi-modal sensor fusion or to predict the locations where the sensor information is vague or defective. The presented results demonstrate a clear improvement over standard procedures and show that other progress based on our work is possible.European Community's Seventh Framework Progra

Symbol Emergence in Robotics: A Survey

Humans can learn the use of language through physical interaction with their environment and semiotic communication with other people. It is very important to obtain a computational understanding of how humans can form a symbol system and obtain semiotic skills through their autonomous mental development. Recently, many studies have been conducted on the construction of robotic systems and machine-learning methods that can learn the use of language through embodied multimodal interaction with their environment and other systems. Understanding human social interactions and developing a robot that can smoothly communicate with human users in the long term, requires an understanding of the dynamics of symbol systems and is crucially important. The embodied cognition and social interaction of participants gradually change a symbol system in a constructive manner. In this paper, we introduce a field of research called symbol emergence in robotics (SER). SER is a constructive approach towards an emergent symbol system. The emergent symbol system is socially self-organized through both semiotic communications and physical interactions with autonomous cognitive developmental agents, i.e., humans and developmental robots. Specifically, we describe some state-of-art research topics concerning SER, e.g., multimodal categorization, word discovery, and a double articulation analysis, that enable a robot to obtain words and their embodied meanings from raw sensory--motor information, including visual information, haptic information, auditory information, and acoustic speech signals, in a totally unsupervised manner. Finally, we suggest future directions of research in SER.Comment: submitted to Advanced Robotic