Vision-Based Object Recognition and 3-D Pose Estimation Using Conic Features

This thesis deals with monocular vision-based object recognition and 3-D pose estimation based on conic features. Conic features including circles and ellipses are frequently observed in many man-made objects in real word as well as have the merit of robustness potentially in feature extraction in vision-based applications. Although the 3-D pose estimation problem of conic features in 3-D space has been studied well since 1990, the previous work has not provided a unique solution completely for full 3-D pose parameters (i.e., 3-orientations and 3-positions) due to complexity from high nonlinearity of a general conic. This thesis, therefore, renews conic features in a new perspective on geometric invariants in both 3-D space and 2-D projective space, incorporating other geometric features with conics. First, as the most essential step in dealing with conics, this thesis shows that the pose parameters of a circular feature in 3-D space can be derived analytically from incorporating a coplanar point. A procedure of pose parameter recovery is described in detail, and its performance is evaluated and discussed in view of pose estimation errors and sensitivity. Second, it is also revealed that the pose of an elliptic feature can be resolved when two coplanar points are incorporated on the basis of the polarity of two points for a conic in 2-D projective space. This thesis proposes a series of algorithms to determine the 3-D pose parameters uniquely, and evaluates the proposed method through a measure of estimation performance and sensitivity depending on point locations. Third, a pair of two conics is dealt with, which is regarded as an extension of the idea of the incorporation scheme to another conic feature from point features. Under the polarity concept, this thesis proves that the problem involving a pair of two conics can be formulated with the problem of one ellipse with two points so that its solution is derived in the same form as in the ellipse case. In order to treat two or more conic objects as well as to deal with an object recognition problem, the rest of thesis concentrates on the theoretical foundation of multiple object recognition. First, some effective modeling approaches are described. A general object model is specially designed to model multiple objects for object recognition and pose recovery in view of spatial geometry. In particular, this thesis defines a pairwise conic model that can describes the geometrical relation between two conics invariantly in 2-D projective space, which consists of a pairwise conic (PC), a pairwise conic invariant (PCI), and a pairwise conic pole (PCP). Based on the two kinds of models, an object learning and recognition system is proposed as a general framework for multiple object recognition. Considering simplicity and flexibility in object learning stage, this thesis introduces a semi-automatic learning scheme to construct the multiple object model from a model image at once. To utilize geometric relations among multiple objects effectively in object recognition, this thesis specifies some feature functions based on the pairwise conic model, and then describes an object recognition method in a fashion of linear-chain conditional random field (CRF). In particular, as a post refinement step of the recognition, a geometric alignment procedure is also proposed in algorithmic details to improve recognition performance against noisy conditions. Last, the multiple object recognition method is evaluated intensively through two practical applications that deal with a place recognition and an elevator button recognition problem for service robots. A series of experiment results supports the effectiveness of the proposed method, maintaining reliable performance against noisy conditions in the presence of perspective distortion and partial object occlusions.Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Research objective and expected contribution . . . . . . . . . . . . . . . . . . 6 1.4 Organization of thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2 3-D Pose Estimation of a Circular Feature 10 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Preliminaries: an elliptic cone in 3-D space and its homogeneous representation in 2-D projective space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.1 Homogeneous representation . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.2 Principal planes of a cone versus diagonalization of a conic matrix Q . 16 2.3 3-D interpretation of a circular feature for 3-D pose estimation . . . . . . . . 19 2.3.1 3-D orientation estimation . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 3-D position estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.3 Composition of homogeneous transformation and discrimination for the unique solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.2 Evaluation of pose estimation performance . . . . . . . . . . . . . . . 29 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3 3-D Pose Estimation of an Elliptic Feature 35 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.2 Interpretation of an elliptic feature with coplanar points in 2-D projective space 38 3.2.1 The minimal number of points for pose estimation . . . . . . . . . . . 39 3.2.2 Analysis of possible constraints for relative positions of two points to an ellipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.3 Feature selection scheme for stable homography estimation . . . . . . 43 3.3 3-D pose estimation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.1 Extraction of triangular features from an elliptic object . . . . . . . . 47 3.3.2 Homography decomposition . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.3 Composition of homogeneous transformation matrix with unique solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4.2 Evaluation of the proposed method . . . . . . . . . . . . . . . . . . . . 54 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4 3-D Pose Estimation of a Pair of Conic Features 61 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.2 3-D pose estimation of a conic feature incorporated with line features . . . . 61 4.3 3-D pose estimation of a conic feature incorporated with another conic feature 63 4.3.1 Some examples of self-polar triangle and invariants . . . . . . . . . . . 65 4.3.2 3-D pose estimation of a pair of coplanar conics . . . . . . . . . . . . . 67 4.3.3 Examples of 3-D pose estimation of a conic feature incorporated with another conic feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Multiple Object Recognition Based on Pairwise Conic Model 77 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Learning of geometric relation of multiple objects . . . . . . . . . . . . . . . . 78 5.3 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.3.1 De_nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.4 Multiple object recognition based on pairwise conic model and conditional random _elds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.4.1 Graphical model for multiple object recognition . . . . . . . . . . . . . 86 5.4.2 Linear-chain conditional random _eld . . . . . . . . . . . . . . . . . . 87 5.4.3 Determination of low-level feature functions for multiple object recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.4.4 Range selection trick for e_ciently computing the costs of low-level feature functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.4.5 Evaluation of observation sequence . . . . . . . . . . . . . . . . . . . . 93 5.4.6 Object recognition based on hierarchical CRF . . . . . . . . . . . . . . 95 5.5 Geometric alignment algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6 Application to Place Recognition for Service Robots 105 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 6.2 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.2.1 Detection of 2-D geometric shapes . . . . . . . . . . . . . . . . . . . . 107 6.2.2 Examples of shape feature extraction . . . . . . . . . . . . . . . . . . . 109 6.3 Object modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6.3.1 A place model that describes multiple landmark objects . . . . . . . . 112 6.3.2 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.3.3 Incorporation of non-conic features with a pairwise conic model . . . . 114 6.4 Place learning and recognition system . . . . . . . . . . . . . . . . . . . . . . 121 6.4.1 HCRF-based recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.5 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.5.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.5.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 7 Application to Elevator Button Recognition 136 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 7.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 7.1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7.1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7.2 Object modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 7.2.1 Geometric model for multiple button objects . . . . . . . . . . . . . . 140 7.2.2 Pairwise conic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 7.3 Learning and recognition system . . . . . . . . . . . . . . . . . . . . . . . . . 141 7.3.1 Button object learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 7.3.2 CRF-based recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 7.4 Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.4.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.4.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 151 7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 8 Concluding remarks 159 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 8.2 Further work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 References 161 Summary (in Korean) 16

Efficient acquisition, representation and rendering of light fields

In this thesis we discuss the representation of three-dimensional scenes using image data (image-based rendering), and more precisely the so-called light field approach. We start with an up-to-date survey on previous work in this young field of research. Then we propose a light field representation based on image data and additional per-pixel depth values. This enables us to reconstruct arbitrary views of the scene in an efficient way and with high quality. Furtermore, we can use the same representation to determine optimal reference views during the acquisition of a light field. We further present the so-called free form parameterization, which allows for a relatively free placement of reference views. Finally, we demonstrate a prototype of the Lumi-Shelf system, which acquires, transmits, and renders the light field of a dynamic scene at multiple frames per second.Diese Doktorarbeit beschäftigt sich mit der Repräsentierung dreidimensionaler Szenen durch Bilddaten (engl. image-based rendering, deutsch bildbasierte Bildsynthese), speziell mit dem Ansatz des sog. Lichtfelds. Nach einem aktuellen Überblick über bisherige Arbeiten in diesem jungen Forschungsgebiet stellen wir eine Datenrepräsentation vor, die auf Bilddaten mit zusätzlichen Tiefenwerten basiert. Damit sind wir in der Lage, beliebige Ansichten der Szene effizient und in hoher Qualität zu rekonstruieren sowie die optimalen Referenz-Ansichten bei der Akquisition eines Lichtfelds zu bestimmen. Weiterhin präsentieren wir die sog. Freiform-Parametrisierung, die eine relativ freie Anordnung der Referenz-Ansichten erlaubt. Abschließend demonstrieren wir einen Prototyp des Lumishelf-Systems, welches die Aufnahme, Übertragung und Darstellung des Lichtfeldes einer dynamischen Szene mit mehreren Bildern pro Sekunde ermöglicht

Efficient acquisition, representation and rendering of light fields

In this thesis we discuss the representation of three-dimensional scenes using image data (image-based rendering), and more precisely the so-called light field approach. We start with an up-to-date survey on previous work in this young field of research. Then we propose a light field representation based on image data and additional per-pixel depth values. This enables us to reconstruct arbitrary views of the scene in an efficient way and with high quality. Furtermore, we can use the same representation to determine optimal reference views during the acquisition of a light field. We further present the so-called free form parameterization, which allows for a relatively free placement of reference views. Finally, we demonstrate a prototype of the Lumi-Shelf system, which acquires, transmits, and renders the light field of a dynamic scene at multiple frames per second.Diese Doktorarbeit beschäftigt sich mit der Repräsentierung dreidimensionaler Szenen durch Bilddaten (engl. image-based rendering, deutsch bildbasierte Bildsynthese), speziell mit dem Ansatz des sog. Lichtfelds. Nach einem aktuellen Überblick über bisherige Arbeiten in diesem jungen Forschungsgebiet stellen wir eine Datenrepräsentation vor, die auf Bilddaten mit zusätzlichen Tiefenwerten basiert. Damit sind wir in der Lage, beliebige Ansichten der Szene effizient und in hoher Qualität zu rekonstruieren sowie die optimalen Referenz-Ansichten bei der Akquisition eines Lichtfelds zu bestimmen. Weiterhin präsentieren wir die sog. Freiform-Parametrisierung, die eine relativ freie Anordnung der Referenz-Ansichten erlaubt. Abschließend demonstrieren wir einen Prototyp des Lumishelf-Systems, welches die Aufnahme, Übertragung und Darstellung des Lichtfeldes einer dynamischen Szene mit mehreren Bildern pro Sekunde ermöglicht

Keep Your Eyes above the Ball: Investigation of Virtual Reality (VR) Assistive Gaming for Age-Related Macular Degeneration (AMD) Visual Training

Humans are beyond all visual beings since most of the outside information is gathered through the visual system. When the aging process starts, visual functional damages become more and more common and the risk of developing visual impairment is higher. Age-related macular degeneration (AMD) is one of the main afflictions that leads to severe damage to the optical system due to the aging process. The ones affected lose the ability to use the central part of vision, essential for accurate visual information processing. Even if less accurate, peripheral vision remains unaffected, hence medical experts have developed training procedures to train patients to use peripheral vision instead to navigate their environment and continue their daily lives. This type of training is called eccentric viewing. However, there are several shortcomings in current approaches, such as not being engaging or individualizable enough nor cost and time-effective. The main scope of this dissertation was to find out if more engaging and individualizable methods can be used for peripheral training of AMD patients. The current work used virtual reality (VR) gaming to deliver AMD training; the first time such an approach was used for eccentric viewing training. In combination with eye-tracking, real-time individualized assistance was also achieved. Thanks to an integrated eye-tracker in the headset, concentric gaze-contingent stimuli were used to redirect the eyes toward an eccentric location. The concentric feature allowed participants to choose freely and individually their peripheral focus point. One study investigated the feasibility a VR system for individualized visual training of ophthalmic patients, two studies investigated two types of peripheral stimuli (three spatial cues and two optical distortions) and the last study was a case study looking into the feasibility of such an approach for a patient with late AMD. Changes in gaze directionality were observed in all the last three studies for one specific spatial cue, a concentric ring. In accordance with the literature, the gaze was directed spontaneously toward the most effective peripheral position. The last study additionally proved gaming feasible for future testing of the elderly AMD population. The current work opened the road to more individualized and engaging interventions for eccentric viewing training for late AMD

Automated flight planning for roof inspection using a face-based approach

The rapid proliferation of consumer small unmanned aerial systems (sUASs) has expanded ownership to include amateurs and professionals alike. These platforms in combination with numerous open source and proprietary applications tailored to gather aerial imagery and generate 3D point clouds and meshes from aerial imagery, have made 3D modeling available to anyone who can afford an entry-level sUAS. These flight plans force the sensor to remain at greater distances from their targets, resulting in varying spatial resolution of sloped surfaces. The work described here explains the development of a variety of 3D automated flight plans to provide vantage points not achievable by constant-altitude, nadir-looking imagery. Specifically, the issue of roof inspection is addressed in detail. This work generates an automated flight plan that positions the sUAS and orients its sensor such that the focal plane array is parallel to the roof plane based on a priori knowledge of the roof\u27s geometry, greatly reducing single- or two-point perspective. This a priori knowledge can come from a variety sources including databases, a site survey, or data extracted from an existing point cloud. Still images or video from orthogonal flight plans can be used for visual inspection, or the generation of dense point clouds and meshes. These products are compared to those generated from nadir imagery. This novel flight planning approach permits the aircraft to fly the orthogonal flight plans from start to finish without intervention from the remote pilot. This work is scalable to similar sUAS-based tasks including aerial-based thermography of buildings and infrastructure

The characterisation and simulation of 3D vision sensors for measurement optimisation

The use of 3D Vision is becoming increasingly common in a range of industrial applications including part identification, reverse engineering, quality control and inspection. To facilitate this increased usage, especially in autonomous applications such as free-form assembly and robotic metrology, the capability to deploy a sensor to the optimum pose for a measurement task is essential to reduce cycle times and increase measurement quality. Doing so requires knowledge of the 3D sensor capabilities on a material specific basis, as the optical properties of a surface, object shape, pose and even the measurement itself have severe implications for the data quality. This need is not reflected in the current state of sensor haracterisation standards which commonly utilise optically compliant artefacts and therefore can not inform the user of a 3D sensor the realistic expected performance on non-ideal objects.This thesis presents a method of scoring candidate viewpoints for their ability to perform geometric measurements on an object of arbitrary surface finish. This is achieved by first defining a technology independent, empirical sensor characterisation method which implements a novel variant of the commonly used point density point cloud quality metric, which is normalised to isolate the effect of surface finish on sensor performance, as well as the more conventional assessment of point standard deviation. The characterisation method generates a set of performance maps for a sensor per material which are a function of distance and surface orientation. A sensor simulation incorporates these performance maps to estimate the statistical properties of a point cloud on objects with arbitrary shape and surface finish, providing the sensor has been characterised on the material in question.A framework for scoring measurement specific candidate viewpoints is presented in the context of the geometric inspection of four artefacts with different surface finish but identical geometry. Views are scored on their ability to perform each measurement based on a novel view score metric, which incorporates the expected point density, noise and occlusion of measurement dependent model features. The simulation is able to score the views reliably on all four surface finishes tested, which range from ideal matt white to highly polished aluminium. In 93% of measurements, a set of optimal or nearly optimal views is correctly selected.</div