Calibration and Reconstruction in Non-Central Axial Catadioptric Systems

Tese de doutoramento em Engenharia Electrotécnica e de Computadores, no ramo de Automação e Robótica, apresentada ao Departamento de Engenharia Eletrotécnica e de Computadores da Faculdade de Ciências e Tecnologia da Universidade de CoimbraEsta tese de doutoramento estuda sistemas de visão axiais catadióptricos nãocentrais, ou seja, sistemas com um espelho de simetria axial e uma câmara pinhole com o centro ótico pertencente ao eixo do espelho. São propostos métodos originais para calibração e reconstrução 3D usando a imagem de pontos e retas. Por “calibração” entende-se a reconstrução da geometria do sistema de visão, em termos da forma do espelho e da posição e orientação relativa camera/espelho. Para além disso, também se pretende estimar a pose da câmara em relação ao sistema de coordenadas do mundo, ou seja, a estimação dos parâmetros extrínsecos. Assume-se que a câmara pinhole está calibrada internamente a priori. Os algoritmos baseiam-se na utilização da imagem de um padrão de calibração planar, por exemplo, um padrão em xadrez. São propostos cinco algoritmos distintos. Um método estima a posição do eixo do espelho na imagem (de modo a determinar a orientação relativa câmara/ espelho) usando a invariância do cross-ratio. Outro método estima os parâmetros extrínsecos e a distância câma-ra/espelho, dado o conhecimento da forma do espelho. Baseia-se no estabelecimento de uma relação linear 3D/1D entre pontos do mundo e elementos da imagem, e na utilização do algoritmo Direct-Linear-Transformation (DLT) de modo a determinar um subconjunto dos parâmetros do sistema. Os parâmetros restantes são estimados usando procedimentos de otimização não-linear, numa variável de cada vez. Como uma extensão ao método anterior, também é proposta a estimação da forma do espelho como parte do processo de calibração. Este método utiliza a imagem de pontos e retas. Aproveita o facto de que todos os pontos num círculo da imagem centrado na origem possuem raios de retroprojeção que se intersetam num único ponto, formando um sistema de projeção central. Também é proposto um algoritmo para o caso particular de sistemas catadióptricos com espelhos esféricos, onde a calibração é alcançada através do ajuste de curvas quárticas às imagens de retas de um padrão de calibração. É derivada uma solução analítica, que é seguidamente refinada através de um procedimento de otimização não-linear. v Finalmente, considerando o caso de um sistema axial catadióptrico completamente calibrado, é feita a reconstrução da posição 3D de uma reta através de uma única imagem dessa mesma reta (que é possível devido ao facto de o sistema ser não-central). A reta é reconstruída a partir de 3 ou mais pontos na imagem, conhecendo o rácio da distância entre 3 pontos na reta (o que é uma assunção admissível em, por exemplo, ambientes estruturados com objetos arquitetónicos repetitivos, como janelas ou ladrilhos). É usada a invariância do cross-ratio de modo a restringir a localização da reta e, seguidamente, é feita a reconstrução a partir de um conjunto de pontos na imagem através de otimização não-linear. São apresentadas experiências com imagens reais e simuladas de modo a avaliar a precisão e robustez dos métodos.This PhD thesis focuses on non-central axial catadioptric vision systems, i.e. systems with an axial symmetrical mirror and a pinhole camera with its optical center located on the mirror axis. We propose novel methods to achieve calibration and 3D reconstruction from the image of points and lines. By “calibration” we mean the reconstruction of the vision system geometry, in terms of mirror shape and mirror/camera relative position and orientation. We also aim at the estimation of the pose of the camera w.r.t. the world coordinates frame, i.e. the estimation of the extrinsic parameters. We assume that the pinhole camera is internally calibrated a priori. The algorithms rely on the image of a planar calibration pattern, e.g. a checkerboard. We propose five distinct algorithms. One method aims at estimating the position of the mirror axis in the image (to determine camera/mirror relative orientation) using the cross-ratio as an invariant. Another method estimates the extrinsic parameters and camera/mirror distance given the knowledge of the mirror shape. It relies on establishing a 3D/1D linear relation between world points and image features, and using the Direct- Linear-Transformation (DLT) algorithm to obtain a subset of the system parameters. The remaining parameters are estimated using non-linear optimization, on a single variable at a time. As an extension to the previous method, we propose the estimation of the mirror shape as part of the calibration process. This method requires the image of points and lines. It uses the fact that all points in any image circle centered at the origin have backprojection rays that intersect at a single point, effectively becoming a central projection system. We also propose an algorithm for the particular case of catadioptric systems with spherical mirrors, where the calibration is achieved by fitting quartic curves to the images of lines in a calibration pattern. An analytical solution is derived, which is later refined by a non-linear optimization procedure. Finally, we consider the case of a fully calibrated non-central axial catadioptric system, and aim at the reconstruction of the 3D position of a line from a single vii image of that line (which is possible because the system is non-central). The line is reconstructed from 3 or more image points, given the knowledge of the distance ratio of 3 points in the line (a fair assumption in, for example, structured environments with repetitive architectural features, like windows or tiles). We use cross-ratio as an invariant to constrain the line localization and then perform the reconstruction from a set of image points through non-linear optimization. Experiments with simulated and real images are performed to evaluate the accuracy and robustness of the methods.FCT - PROTEC SFRH/BD/50281/200

Noncentral catadioptric systems with quadric mirrors : geometry and calibration

Tese de doutoramento em Engenharia Electrotécnica (Informática) apresentada à Faculdade de Ciências e Tecnologia da Universidade de CoimbraNesta dissertação de doutoramento estudamos e analisamos a geometria dos sistema catadióptricos não-centrais compostos por uma câmara pinhole ou ortográfica e um espelho curvo, cuja forma é uma quádrica não degenerada, incluindo elipsóides, que podem ser esferas, hiperbolóides e parabolóides. A geometria destes sistemas de visão é parameterizada, analisando o fenómeno de formação da imagem, e é composta pelos parâmetros intrínsecos da câmara, os parâmetros da superfície do espelho e a posição e orientação da câmara em relação ao espelho e ao sistema de referência do mundo. A formação da imagem é estudada numa perspectiva puramente geométrica, focando principalmente o modelo de projecção e a calibração do sistema de visão. As principais contribuições deste trabalho incluem a demonstração de que num sistema catadióptrico não-central com um câmara em perspectiva e uma quádrica não degenerada, o ponto de reflexão na superfície do espelho (projectando na imagem qualquer ponto 3D do mundo) pertence a uma curva quártica que é dada pela intersecção de duas superfícies quádricas. O correspondente modelo de projecção é também desenvolvido e é expresso através de uma equação não linear implícita, dependente de um único parâmetro. Relativamente `a calibração destes sistemas de visão, foi desenvolvido um método de calibração, assumindo o conhecimento dos parâmetros intrínsecos da câmara em perspectiva e de um conjunto de pontos 3D expressos em coordenadas locais (estrutura 3D do mundo). Informação acerca do contorno aparente do espelho é também usada para melhorar a precisão da estimação. Um outro método de calibração é proposto, assumindo uma calibração prévia do sistema no sentido de um modelo geral de câmara (correspondências entre pontos na imagem e raios incidentes no espaço). Adicionalmente, a posição e orientação (pose) da câmara em relação ao espelho e ao sistema de referência do mundo são estimadas usando métricas algébricas e equações lineares (escritas para um método de calibração que também é apresentado). Considera-se a câmara como pré-calibrada. São desenvolvidas e apresentadas experiências com simulações extensivas e também com imagens reais de forma a testar a robustez e precisão dos métodos apresentados. As principais conclusões apontam para o facto de estes sistemas de visão serem altamente não lineares e a sua calibração ser possível com boa precisão, embora difícil de alcançar com precisão muito elevada, especialmente se o sistema de visão tem como objectivo aplicações direccionadas para a precisão. Apesar disso, pode observar-se que a informação da estrutura do mundo pode ser complementada com informação adicional, tal como o contorno aparente da quádrica, de forma a melhorar a qualidade dos resultados de calibração. Na verdade, o uso do contorno aparente do espelho pode, por si, melhorar drasticamente a precisão da estimação.In this PhD thesis we study and analyze the geometry of noncentral catadioptric systems composed by a pinhole or orthographic camera and a non-ruled quadric shaped mirror, that is to say an ellipsoid, which can be a sphere, a hyperboloid or a paraboloid surface. The geometry of these vision systems is parameterized by analyzing the image formation and is composed by the intrinsic parameters of the camera, the parameters of the mirror surface and the poses of the camera in relation to the mirror and to the world reference frames. Image formation is studied in a purely geometrical way, focusing mainly on the projection model and on the calibration of the vision system. The main contributions include the proof that in a noncentral catadioptric system with a perspective camera and a non degenerate quadric the reflection point on the surface (projecting any given 3D world point to the image) is on the quartic curve that is the intersection of two quadrics. The projection model related to the previous definition of the reflection point is also derived and is expressed as an implicit non linear function on a single unknown. In what concerns the calibration of these vision systems, we developed a calibration method assuming the knowledge of the intrinsic parameters of the perspective camera and of some 3D points in a local reference frame (structure) . Information about the apparent contour is also used to enhance the accuracy of the estimation. Another calibration method is proposed, assuming a previous calibration of the system in the sense of a general camera model (correspondences between image points and incident lines in space). Additionally, the camera-mirror and camera-world poses are estimated using algebraic metrics and linear equations (derived for a calibration method that is also presented). The camera is considered to be pre-calibrated. Experiments with extensive simulations and also using real images are performed to test the robustness and accuracy of the methods presented. The main conclusions are that these vision systems are highly non linear and that their calibration is possible with good accuracy but difficult to achieve with very high accuracy, specially if the vision system is aimed at being used for accuracy-driven applications. Nevertheless it is observed that structure of the world can be complemented with some additional information as the quadric apparent contour in order to improve the quality of the calibration results. Actually, the use of the apparent contour can dramatically improve the accuracy of the estimation

Enhancing 3D Visual Odometry with Single-Camera Stereo Omnidirectional Systems

We explore low-cost solutions for efficiently improving the 3D pose estimation problem of a single camera moving in an unfamiliar environment. The visual odometry (VO) task -- as it is called when using computer vision to estimate egomotion -- is of particular interest to mobile robots as well as humans with visual impairments. The payload capacity of small robots like micro-aerial vehicles (drones) requires the use of portable perception equipment, which is constrained by size, weight, energy consumption, and processing power. Using a single camera as the passive sensor for the VO task satisfies these requirements, and it motivates the proposed solutions presented in this thesis. To deliver the portability goal with a single off-the-shelf camera, we have taken two approaches: The first one, and the most extensively studied here, revolves around an unorthodox camera-mirrors configuration (catadioptrics) achieving a stereo omnidirectional system (SOS). The second approach relies on expanding the visual features from the scene into higher dimensionalities to track the pose of a conventional camera in a photogrammetric fashion. The first goal has many interdependent challenges, which we address as part of this thesis: SOS design, projection model, adequate calibration procedure, and application to VO. We show several practical advantages for the single-camera SOS due to its complete 360-degree stereo views, that other conventional 3D sensors lack due to their limited field of view. Since our omnidirectional stereo (omnistereo) views are captured by a single camera, a truly instantaneous pair of panoramic images is possible for 3D perception tasks. Finally, we address the VO problem as a direct multichannel tracking approach, which increases the pose estimation accuracy of the baseline method (i.e., using only grayscale or color information) under the photometric error minimization as the heart of the “direct” tracking algorithm. Currently, this solution has been tested on standard monocular cameras, but it could also be applied to an SOS. We believe the challenges that we attempted to solve have not been considered previously with the level of detail needed for successfully performing VO with a single camera as the ultimate goal in both real-life and simulated scenes

Interpreting Sphere Images Using the Double-Contact Theorem

An occluding contour of a sphere is projected to a conic in the perspective image, and such a conic is called a sphere image. Recently, it has been discovered that each sphere image is tangent to the image of the absolute conic at two double-contact image points. The double-contact theorem describes the properties of three conics which all have double contact with another conic. This theorem provides a linear algorithm to find the another conic if these three conics are given. In this paper, the double-contact theorem is employed to interpret the properties among three sphere images and the image of the absolute conic. The image of the absolute conic can be determined from three sphere images using the double-contact theorem. Therefore, a linear calibration method using three sphere images is obtained. Only three sphere images are required, and all five intrinsic parameters are recovered linearly without making assumptions, such as, zero-skew or unitary aspect ratio. Extensive experiments on simulated and real data were performed and shown that our calibration method is an order of magnitude faster than previous optimized methods and a little faster than former linear methods while maintaining comparable accuracy.http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000235772300073&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=8e1609b174ce4e31116a60747a720701Computer Science, Artificial IntelligenceComputer Science, Theory & MethodsSCI(E)CPCI-S(ISTP)

Vision Sensors and Edge Detection

Vision Sensors and Edge Detection book reflects a selection of recent developments within the area of vision sensors and edge detection. There are two sections in this book. The first section presents vision sensors with applications to panoramic vision sensors, wireless vision sensors, and automated vision sensor inspection, and the second one shows image processing techniques, such as, image measurements, image transformations, filtering, and parallel computing

Metric and appearance based visual SLAM for mobile robots

Simultaneous Localization and Mapping (SLAM) maintains autonomy for mobile robots and it has been studied extensively during the last two decades. It is the process of building the map of an unknown environment and determining the location of the robot using this map concurrently. Different kinds of sensors such as Global Positioning System (GPS), Inertial Measurement Unit (IMU), laser range finder and sonar are used for data acquisition in SLAM. In recent years, passive visual sensors are utilized in visual SLAM (vSLAM) problem because of their increasing ubiquity. This thesis is concerned with the metric and appearance-based vSLAM problems for mobile robots. From the point of view of metric-based vSLAM, a performance improvement technique is developed. Template matching based video stabilization and Harris corner detector are integrated. Extracting Harris corner features from stabilized video consistently increases the accuracy of the localization. Data coming from a video camera and odometry are fused in an Extended Kalman Filter (EKF) to determine the pose of the robot and build the map of the environment. Simulation results validate the performance improvement obtained by the proposed technique. Moreover, a visual perception system is proposed for appearance-based vSLAM and used for under vehicle classification. The proposed system consists of three main parts: monitoring, detection and classification. In the first part a new catadioptric camera system, where a perspective camera points downwards to a convex mirror mounted to the body of a mobile robot, is designed. Thanks to the catadioptric mirror the scenes against the camera optical axis direction can be viewed. In the second part speeded up robust features (SURF) are used to detect the hidden objects that are under vehicles. Fast appearance based mapping algorithm (FAB-MAP) is then exploited for the classification of the means of transportations in the third part. Experimental results show the feasibility of the proposed system. The proposed solution is implemented using a non-holonomic mobile robot. In the implementations the bottom of the tables in the laboratory are considered as the under vehicles. A database that includes di erent under vehicle images is used. All the algorithms are implemented in Microsoft Visual C++ and OpenCV 2.4.4

Rotation Free Active Vision

International audience— Incremental Structure from Motion (SfM) algorithms require, in general, precise knowledge of the camera linear and angular velocities in the camera frame for estimating the 3D structure of the scene. Since an accurate measurement of the camera own motion may be a non-trivial task in several robotics applications (for instance when the camera is onboard a UAV), we propose in this paper an active SfM scheme fully independent from the camera angular velocity. This is achieved by considering, as visual features, some rotational invariants obtained from the projection of the perceived 3D points onto a virtual unitary sphere (unified camera model). This feature set is then exploited for designing a rotation-free active SfM algorithm able to optimize online the direction of the camera linear velocity for improving the convergence of the structure estimation task. As case study, we apply our framework to the depth estimation of a set of 3D points and discuss several simulations and experimental results for illustrating the approach

The Maunakea Spectroscopic Explorer Book 2018

(Abridged) This is the Maunakea Spectroscopic Explorer 2018 book. It is intended as a concise reference guide to all aspects of the scientific and technical design of MSE, for the international astronomy and engineering communities, and related agencies. The current version is a status report of MSE's science goals and their practical implementation, following the System Conceptual Design Review, held in January 2018. MSE is a planned 10-m class, wide-field, optical and near-infrared facility, designed to enable transformative science, while filling a critical missing gap in the emerging international network of large-scale astronomical facilities. MSE is completely dedicated to multi-object spectroscopy of samples of between thousands and millions of astrophysical objects. It will lead the world in this arena, due to its unique design capabilities: it will boast a large (11.25 m) aperture and wide (1.52 sq. degree) field of view; it will have the capabilities to observe at a wide range of spectral resolutions, from R2500 to R40,000, with massive multiplexing (4332 spectra per exposure, with all spectral resolutions available at all times), and an on-target observing efficiency of more than 80%. MSE will unveil the composition and dynamics of the faint Universe and is designed to excel at precision studies of faint astrophysical phenomena. It will also provide critical follow-up for multi-wavelength imaging surveys, such as those of the Large Synoptic Survey Telescope, Gaia, Euclid, the Wide Field Infrared Survey Telescope, the Square Kilometre Array, and the Next Generation Very Large Array.Comment: 5 chapters, 160 pages, 107 figure