Exploiting line metric reconstruction from non-central circular panoramas

In certain non-central imaging systems, straight lines are projected via a non-planar surface encapsulating the 4 degrees of freedom of the 3D line. Consequently the geometry of the 3D line can be recovered from a minimum of four image points. However, with classical non-central catadioptric systems there is not enough effective baseline for a practical implementation of the method. In this paper we propose a multi-camera system configuration resembling the circular panoramic model which results in a particular non-central projection allowing the stitching of a non-central panorama. From a single panorama we obtain well-conditioned 3D reconstruction of lines, which are specially interesting in texture-less scenarios. No previous information about the direction or arrangement of the lines in the scene is assumed. The proposed method is evaluated on both synthetic and real images

Fitting line projections in non-central catadioptric cameras with revolution symmetry

Line-images in non-central cameras contain much richer information of the original 3D line than line projections in central cameras. The projection surface of a 3D line in most catadioptric non-central cameras is a ruled surface, encapsulating the complete information of the 3D line. The resulting line-image is a curve which contains the 4 degrees of freedom of the 3D line. That means a qualitative advantage with respect to the central case, although extracting this curve is quite difficult. In this paper, we focus on the analytical description of the line-images in non-central catadioptric systems with symmetry of revolution. As a direct application we present a method for automatic line-image extraction for conical and spherical calibrated catadioptric cameras. For designing this method we have analytically solved the metric distance from point to line-image for non-central catadioptric systems. We also propose a distance we call effective baseline measuring the quality of the reconstruction of a 3D line from the minimum number of rays. This measure is used to evaluate the different random attempts of a robust scheme allowing to reduce the number of trials in the process. The proposal is tested and evaluated in simulations and with both synthetic and real images

Calibration and Reconstruction in Non-Central Axial Catadioptric Systems

Tese de doutoramento em Engenharia Electrotécnica e de Computadores, no ramo de Automação e Robótica, apresentada ao Departamento de Engenharia Eletrotécnica e de Computadores da Faculdade de Ciências e Tecnologia da Universidade de CoimbraEsta tese de doutoramento estuda sistemas de visão axiais catadióptricos nãocentrais, ou seja, sistemas com um espelho de simetria axial e uma câmara pinhole com o centro ótico pertencente ao eixo do espelho. São propostos métodos originais para calibração e reconstrução 3D usando a imagem de pontos e retas. Por “calibração” entende-se a reconstrução da geometria do sistema de visão, em termos da forma do espelho e da posição e orientação relativa camera/espelho. Para além disso, também se pretende estimar a pose da câmara em relação ao sistema de coordenadas do mundo, ou seja, a estimação dos parâmetros extrínsecos. Assume-se que a câmara pinhole está calibrada internamente a priori. Os algoritmos baseiam-se na utilização da imagem de um padrão de calibração planar, por exemplo, um padrão em xadrez. São propostos cinco algoritmos distintos. Um método estima a posição do eixo do espelho na imagem (de modo a determinar a orientação relativa câmara/ espelho) usando a invariância do cross-ratio. Outro método estima os parâmetros extrínsecos e a distância câma-ra/espelho, dado o conhecimento da forma do espelho. Baseia-se no estabelecimento de uma relação linear 3D/1D entre pontos do mundo e elementos da imagem, e na utilização do algoritmo Direct-Linear-Transformation (DLT) de modo a determinar um subconjunto dos parâmetros do sistema. Os parâmetros restantes são estimados usando procedimentos de otimização não-linear, numa variável de cada vez. Como uma extensão ao método anterior, também é proposta a estimação da forma do espelho como parte do processo de calibração. Este método utiliza a imagem de pontos e retas. Aproveita o facto de que todos os pontos num círculo da imagem centrado na origem possuem raios de retroprojeção que se intersetam num único ponto, formando um sistema de projeção central. Também é proposto um algoritmo para o caso particular de sistemas catadióptricos com espelhos esféricos, onde a calibração é alcançada através do ajuste de curvas quárticas às imagens de retas de um padrão de calibração. É derivada uma solução analítica, que é seguidamente refinada através de um procedimento de otimização não-linear. v Finalmente, considerando o caso de um sistema axial catadióptrico completamente calibrado, é feita a reconstrução da posição 3D de uma reta através de uma única imagem dessa mesma reta (que é possível devido ao facto de o sistema ser não-central). A reta é reconstruída a partir de 3 ou mais pontos na imagem, conhecendo o rácio da distância entre 3 pontos na reta (o que é uma assunção admissível em, por exemplo, ambientes estruturados com objetos arquitetónicos repetitivos, como janelas ou ladrilhos). É usada a invariância do cross-ratio de modo a restringir a localização da reta e, seguidamente, é feita a reconstrução a partir de um conjunto de pontos na imagem através de otimização não-linear. São apresentadas experiências com imagens reais e simuladas de modo a avaliar a precisão e robustez dos métodos.This PhD thesis focuses on non-central axial catadioptric vision systems, i.e. systems with an axial symmetrical mirror and a pinhole camera with its optical center located on the mirror axis. We propose novel methods to achieve calibration and 3D reconstruction from the image of points and lines. By “calibration” we mean the reconstruction of the vision system geometry, in terms of mirror shape and mirror/camera relative position and orientation. We also aim at the estimation of the pose of the camera w.r.t. the world coordinates frame, i.e. the estimation of the extrinsic parameters. We assume that the pinhole camera is internally calibrated a priori. The algorithms rely on the image of a planar calibration pattern, e.g. a checkerboard. We propose five distinct algorithms. One method aims at estimating the position of the mirror axis in the image (to determine camera/mirror relative orientation) using the cross-ratio as an invariant. Another method estimates the extrinsic parameters and camera/mirror distance given the knowledge of the mirror shape. It relies on establishing a 3D/1D linear relation between world points and image features, and using the Direct- Linear-Transformation (DLT) algorithm to obtain a subset of the system parameters. The remaining parameters are estimated using non-linear optimization, on a single variable at a time. As an extension to the previous method, we propose the estimation of the mirror shape as part of the calibration process. This method requires the image of points and lines. It uses the fact that all points in any image circle centered at the origin have backprojection rays that intersect at a single point, effectively becoming a central projection system. We also propose an algorithm for the particular case of catadioptric systems with spherical mirrors, where the calibration is achieved by fitting quartic curves to the images of lines in a calibration pattern. An analytical solution is derived, which is later refined by a non-linear optimization procedure. Finally, we consider the case of a fully calibrated non-central axial catadioptric system, and aim at the reconstruction of the 3D position of a line from a single vii image of that line (which is possible because the system is non-central). The line is reconstructed from 3 or more image points, given the knowledge of the distance ratio of 3 points in the line (a fair assumption in, for example, structured environments with repetitive architectural features, like windows or tiles). We use cross-ratio as an invariant to constrain the line localization and then perform the reconstruction from a set of image points through non-linear optimization. Experiments with simulated and real images are performed to evaluate the accuracy and robustness of the methods.FCT - PROTEC SFRH/BD/50281/200

Omnidirectional Stereo Vision for Autonomous Vehicles

Environment perception with cameras is an important requirement for many applications for autonomous vehicles and robots. This work presents a stereoscopic omnidirectional camera system for autonomous vehicles which resolves the problem of a limited field of view and provides a 360° panoramic view of the environment. We present a new projection model for these cameras and show that the camera setup overcomes major drawbacks of traditional perspective cameras in many applications

A Factorization Based Self-Calibration for Radially Symmetric Cameras

The paper proposes a novel approach for planar selfcalibration of radially symmetric cameras. We model these camera images using notions of distortion center and concentric distortion circles around it. The rays corresponding to pixels lying on a single distortion circle form a right circular cone. Each of these cones is associated with two unknowns; optical center and focal length (opening angle). In the central case, we consider all distortion circles to have the same optical center, whereas in the non-central case they have different optical centers lying on the same optical axis. Based on this model we provide a factorization based self-calibration algorithm for planar scenes from dense image matches. Our formulation provides a rich set of constraints to validate the correctness of the distortion center. We also propose possible extensions of this algorithm i

Omnidirectional Stereo Vision for Autonomous Vehicles

Environment perception with cameras is an important requirement for many applications for autonomous vehicles and robots. This work presents a stereoscopic omnidirectional camera system for autonomous vehicles which resolves the problem of a limited field of view and provides a 360° panoramic view of the environment. We present a new projection model for these cameras and show that the camera setup overcomes major drawbacks of traditional perspective cameras in many applications

Learning the surroundings: 3D scene understanding from omnidirectional images

Las redes neuronales se han extendido por todo el mundo, siendo utilizadas en una gran variedad de aplicaciones. Estos métodos son capaces de reconocer música y audio, generar textos completos a partir de ideas simples u obtener información detallada y relevante de imágenes y videos. Las posibilidades que ofrecen las redes neuronales y métodos de aprendizaje profundo son incontables, convirtiéndose en la principal herramienta de investigación y nuevas aplicaciones en nuestra vida diaria. Al mismo tiempo, las imágenes omnidireccionales se están extendiendo dentro de la industria y nuestra sociedad, causando que la visión omnidireccional gane atención. A partir de imágenes 360 capturamos toda la información que rodea a la cámara en una sola toma.La combinación del aprendizaje profundo y la visión omnidireccional ha atraído a muchos investigadores. A partir de una única imagen omnidireccional se obtiene suficiente información del entorno para que una red neuronal comprenda sus alrededores y pueda interactuar con el entorno. Para aplicaciones como navegación y conducción autónoma, el uso de cámaras omnidireccionales proporciona información en torno del robot, person o vehículo, mientras que las cámaras convencionales carecen de esta información contextual debido a su reducido campo de visión. Aunque algunas aplicaciones pueden incluir varias cámaras convencionales para aumentar el campo de visión del sistema, tareas en las que el peso es importante (P.ej. guiado de personas con discapacidad visual o navegación de drones autónomos), un número reducido de dispositivos es altamente deseable.En esta tesis nos centramos en el uso conjunto de cámaras omnidireccionales, aprendizaje profundo, geometría y fotometría. Evaluamos diferentes enfoques para tratar con imágenes omnidireccionales, adaptando métodos a los modelos de proyección omnidireccionales y proponiendo nuevas soluciones para afrontar los retos de este tipo de imágenes. Para la comprensión de entornos interiores, proponemos una nueva red neuronal que obtiene segmentación semántica y mapas de profundidad de forma conjunta a partir de un único panoramaequirectangular. Nuestra red logra, con un nuevo enfoque convolucional, aprovechar la información del entorno proporcionada por la imagen panorámica y explotar la información combinada de semántica y profundidad. En el mismo tema, combinamos aprendizaje profundo y soluciones geométricas para recuperar el diseño estructural, junto con su escala, de entornos de interior a partir de un único panorama no central. Esta combinación de métodos proporciona una implementación rápida, debido a la red neuronal, y resultados precisos, gracias a lassoluciones geométricas. Además, también proponemos varios enfoques para la adaptación de redes neuronales a la distorsión de modelos de proyección omnidireccionales para la navegación y la adaptación del dominio soluciones previas. En términos generales, esta tesis busca encontrar soluciones novedosas e innovadoras para aprovechar las ventajas de las cámaras omnidireccionales y superar los desafíos que plantean.Neural networks have become widespread all around the world and are used for many different applications. These new methods are able to recognize music and audio, generate full texts from simple ideas and obtain detailed and relevant information from images and videos. The possibilities of neural networks and deep learning methods are uncountable, becoming the main tool for research and new applications in our daily-life. At the same time, omnidirectional and 360 images are also becoming widespread in industry and in consumer society, causing omnidirectional computer vision to gain attention. From 360 images, we capture all the information surrounding the camera in a single shot. The combination of deep learning methods and omnidirectional computer vision have attracted many researchers to this new field. From a single omnidirectional image, we obtain enough information of the environment to make a neural network understand its surroundings and interact with the environment. For applications such as navigation and autonomous driving, the use of omnidirectional cameras provide information all around the robot, person or vehicle, while conventional perspective cameras lack this context information due to their narrow field of view. Even if some applications can include several conventional cameras to increase the system's field of view, tasks where weight is more important (i.e. guidance of visually impaired people or navigation of autonomous drones), the less cameras we need to include, the better. In this thesis, we focus in the joint use of omnidirectional cameras, deep learning, geometry and photometric methods. We evaluate different approaches to handle omnidirectional images, adapting previous methods to the distortion of omnidirectional projection models and also proposing new solutions to tackle the challenges of this kind of images. For indoor scene understanding, we propose a novel neural network that jointly obtains semantic segmentation and depth maps from single equirectangular panoramas. Our network manages, with a new convolutional approach, to leverage the context information provided by the panoramic image and exploit the combined information of semantics and depth. In the same topic, we combine deep learning and geometric solvers to recover the scaled structural layout of indoor environments from single non-central panoramas. This combination provides a fast implementation, thanks to the learning approach, and accurate result, due to the geometric solvers. Additionally, we also propose several approaches of network adaptation to the distortion of omnidirectional projection models for outdoor navigation and domain adaptation of previous solutions. All in all, this thesis looks for finding novel and innovative solutions to take advantage of omnidirectional cameras while overcoming the challenges they pose.<br /

3D Scene Geometry Estimation from 360∘^\circ Imagery: A Survey

This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under the omnidirectional optics. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360$^\circ$, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this paper by pointing out current and future trends.Comment: Published in ACM Computing Survey

Image Geometry

International audienc

Terahertz Technology and Its Applications

The Terahertz frequency range (0.1 – 10)THz has demonstrated to provide many opportunities in prominent research fields such as high-speed communications, biomedicine, sensing, and imaging. This spectral range, lying between electronics and photonics, has been historically known as “terahertz gap” because of the lack of experimental as well as fabrication technologies. However, many efforts are now being carried out worldwide in order improve technology working at this frequency range. This book represents a mechanism to highlight some of the work being done within this range of the electromagnetic spectrum. The topics covered include non-destructive testing, teraherz imaging and sensing, among others