EXPERIMENTAL ASSESSMENT OF TECHNIQUES FOR FISHEYE CAMERA CALIBRATION

Fisheye lens cameras enable to increase the Field of View (FOV), and consequently they have been largely used in several applications like robotics. The use of this type of cameras in close-range Photogrammetry for high accuracy applications, requires rigorous calibration. The main aim of this work is to present the calibration results of a Fuji Finepix S3PRO camera with Samyang 8mm fisheye lens using rigorous mathematical models. Mathematical models based on Perspective, Stereo-graphic, Equi-distant, Orthogonal and Equi-solid-angle projections were implemented and used in the experiments. The fisheye lenses are generally designed following one of the last four models, and Bower-Samyang 8mm lens is based on Stereo-graphic projection. These models were used in combination with symmetric radial, decentering and affinity distortion models. Experiments were performed to verify which set of IOPs (Interior Orientation Parameters) presented better results to describe the camera inner geometry. Collinearity mathematical model, which is based on perspective projection, presented the less accurate results, which was expected because fisheye lenses are not designed following the perspective projection. Stereo-graphic, Equi-distant, Orthogonal and Equi-solid-angle projections presented similar results even considering that Bower-Samyang fisheye lens was built based on Stereo-graphic projection. The experimental results also demonstrated a small correlation between IOPs and EOPs (Exterior Orientation Parameters) for Bower-Samyang lens

Panorama imaging for image-to-physical registration of narrow drill holes inside spongy bones

Image-to-physical registration based on volumetric data like computed tomography on the one side and intraoperative endoscopic images on the other side is an important method for various surgical applications. In this contribution, we present methods to generate panoramic views from endoscopic recordings for image-to-physical registration of narrow drill holes inside spongy bone. One core application is the registration of drill poses inside the mastoid during minimally invasive cochlear implantations. Besides the development of image processing software for registration, investigations are performed on a miniaturized optical system, achieving 360° radial imaging with one shot by extending a conventional, small, rigid, rod lens endoscope. A reflective cone geometry is used to deflect radially incoming light rays into the endoscope optics. Therefore, a cone mirror is mounted in front of a conventional 0° endoscope. Furthermore, panoramic images of inner drill hole surfaces in artificial bone material are created. Prior to drilling, cone beam computed tomography data is acquired from this artificial bone and simulated endoscopic views are generated from this data. A qualitative and quantitative image comparison of resulting views in terms of image-to-image registration is performed. First results show that downsizing of panoramic optics to a diameter of 3mm is possible. Conventional rigid rod lens endoscopes can be extended to produce suitable panoramic one-shot image data. Using unrolling and stitching methods, images of the inner drill hole surface similar to computed tomography image data of the same surface were created. Registration is performed on ten perturbations of the search space and results in target registration errors of (0:487 ± 0:438)mm at the entry point and (0:957 ± 0:948)mm at the exit as well as an angular error of (1:763 ± 1:536)°. The results show suitability of this image data for image-to-image registration. Analysis of the error components in different directions reveals a strong influence of the pattern structure, meaning higher diversity results into smaller errors. © 2017 SPIE

Noncentral catadioptric systems with quadric mirrors : geometry and calibration

Tese de doutoramento em Engenharia Electrotécnica (Informática) apresentada à Faculdade de Ciências e Tecnologia da Universidade de CoimbraNesta dissertação de doutoramento estudamos e analisamos a geometria dos sistema catadióptricos não-centrais compostos por uma câmara pinhole ou ortográfica e um espelho curvo, cuja forma é uma quádrica não degenerada, incluindo elipsóides, que podem ser esferas, hiperbolóides e parabolóides. A geometria destes sistemas de visão é parameterizada, analisando o fenómeno de formação da imagem, e é composta pelos parâmetros intrínsecos da câmara, os parâmetros da superfície do espelho e a posição e orientação da câmara em relação ao espelho e ao sistema de referência do mundo. A formação da imagem é estudada numa perspectiva puramente geométrica, focando principalmente o modelo de projecção e a calibração do sistema de visão. As principais contribuições deste trabalho incluem a demonstração de que num sistema catadióptrico não-central com um câmara em perspectiva e uma quádrica não degenerada, o ponto de reflexão na superfície do espelho (projectando na imagem qualquer ponto 3D do mundo) pertence a uma curva quártica que é dada pela intersecção de duas superfícies quádricas. O correspondente modelo de projecção é também desenvolvido e é expresso através de uma equação não linear implícita, dependente de um único parâmetro. Relativamente `a calibração destes sistemas de visão, foi desenvolvido um método de calibração, assumindo o conhecimento dos parâmetros intrínsecos da câmara em perspectiva e de um conjunto de pontos 3D expressos em coordenadas locais (estrutura 3D do mundo). Informação acerca do contorno aparente do espelho é também usada para melhorar a precisão da estimação. Um outro método de calibração é proposto, assumindo uma calibração prévia do sistema no sentido de um modelo geral de câmara (correspondências entre pontos na imagem e raios incidentes no espaço). Adicionalmente, a posição e orientação (pose) da câmara em relação ao espelho e ao sistema de referência do mundo são estimadas usando métricas algébricas e equações lineares (escritas para um método de calibração que também é apresentado). Considera-se a câmara como pré-calibrada. São desenvolvidas e apresentadas experiências com simulações extensivas e também com imagens reais de forma a testar a robustez e precisão dos métodos apresentados. As principais conclusões apontam para o facto de estes sistemas de visão serem altamente não lineares e a sua calibração ser possível com boa precisão, embora difícil de alcançar com precisão muito elevada, especialmente se o sistema de visão tem como objectivo aplicações direccionadas para a precisão. Apesar disso, pode observar-se que a informação da estrutura do mundo pode ser complementada com informação adicional, tal como o contorno aparente da quádrica, de forma a melhorar a qualidade dos resultados de calibração. Na verdade, o uso do contorno aparente do espelho pode, por si, melhorar drasticamente a precisão da estimação.In this PhD thesis we study and analyze the geometry of noncentral catadioptric systems composed by a pinhole or orthographic camera and a non-ruled quadric shaped mirror, that is to say an ellipsoid, which can be a sphere, a hyperboloid or a paraboloid surface. The geometry of these vision systems is parameterized by analyzing the image formation and is composed by the intrinsic parameters of the camera, the parameters of the mirror surface and the poses of the camera in relation to the mirror and to the world reference frames. Image formation is studied in a purely geometrical way, focusing mainly on the projection model and on the calibration of the vision system. The main contributions include the proof that in a noncentral catadioptric system with a perspective camera and a non degenerate quadric the reflection point on the surface (projecting any given 3D world point to the image) is on the quartic curve that is the intersection of two quadrics. The projection model related to the previous definition of the reflection point is also derived and is expressed as an implicit non linear function on a single unknown. In what concerns the calibration of these vision systems, we developed a calibration method assuming the knowledge of the intrinsic parameters of the perspective camera and of some 3D points in a local reference frame (structure) . Information about the apparent contour is also used to enhance the accuracy of the estimation. Another calibration method is proposed, assuming a previous calibration of the system in the sense of a general camera model (correspondences between image points and incident lines in space). Additionally, the camera-mirror and camera-world poses are estimated using algebraic metrics and linear equations (derived for a calibration method that is also presented). The camera is considered to be pre-calibrated. Experiments with extensive simulations and also using real images are performed to test the robustness and accuracy of the methods presented. The main conclusions are that these vision systems are highly non linear and that their calibration is possible with good accuracy but difficult to achieve with very high accuracy, specially if the vision system is aimed at being used for accuracy-driven applications. Nevertheless it is observed that structure of the world can be complemented with some additional information as the quadric apparent contour in order to improve the quality of the calibration results. Actually, the use of the apparent contour can dramatically improve the accuracy of the estimation

CONTRIBUTION A LA STEREOVISION OMNIDIRECTIONNELLE ET AU TRAITEMENT DES IMAGES CATADIOPTRIQUES : APPLICATION AUX SYSTEMES AUTONOMES

Computer vision and digital image processing are two disciplines aiming to endow computers with a sense of perception and image analysis, similar to that of humans. Artificial visual perception can be greatly enhanced when a large field of view is available. This thesis deals with the use of omnidirectional cameras as a mean of expanding the field of view of computer vision systems. The visual perception of depth (3D) by means of omnistereo configurations, and special processing algorithms adapted to catadioptric images, are the main subjects studied in this thesis. Firstly a survey on 3D omnidirectional vision systems is conducted. It highlights the main approaches for obtaining depth information, and provides valuable indications for the choice of the configuration according to the application requirements. Then the design of an omnistereo sensor is addressed, we present a new configuration of the proposed sensor formed by a unique catadioptric camera, dedicated to robotic applications. An experimental investigation of depth estimation accuracy was conducted to validate the new configuration.Digital images acquired by catadioptric cameras present various special geometrical proprieties, such as non-uniform resolution and severe radial distortions. The application of conventional algorithms to process such images is limited in terms of performance. For that, new algorithms adapted to the spherical geometry of catadioptric images have been developed.Gathered omnidirectional computer vision techniques were finally used in two real applications. The first concerns the integration of catadioptric cameras to a mobile robot. The second focuses on the design of a solar tracker, based on a catadioptric camera.The results confirm that the adoption of such sensors for autonomous systems offer more performance and flexibility in regards to conventional sensors.La vision par ordinateur est une discipline qui vise doter les ordinateurs d’un sens de perception et d’analyse d'image semblable à celui de l’homme. La perception visuelle artificielle peut être grandement améliorée quand un grand champ de vision est disponible. Cette thèse traite de l'utilisation des caméras omnidirectionnelles comme un moyen d'élargir le champ de vision des systèmes de vision artificielle. La perception visuelle de la profondeur (3D) par le biais de configurations omnistéréo, et les algorithmes de traitement adaptés aux images catadioptriques, sont les principaux sujets étudiés.Tout d'abord une étude des systèmes de vision omnidirectionnelle 3D est menée. Elle met en évidence les principales approches pour obtenir l’information sur la profondeur et fournit des indications précieuses sur le choix de la configuration en fonction des besoins de l'application. Ensuite, la conception d'un capteur omnistéréo est adressée ; nous présentons une nouvelle configuration du capteur proposé basé une caméra catadioptrique unique, et dédié à la robotique mobile. Des expérimentations sur la précision d’estimation de la profondeur ont été menées pour valider la nouvelle configuration. Les images catadioptriques présentent diverses propriétés géométriques particulières, telles que la résolution non-uniforme et de fortes distorsions radiales. L’application des algorithmes de traitement classiques à ce type d’images se trouve limité en termes de performances. Dans ce sens, de nouveaux algorithmes adaptés à la géométrie sphérique de ces images ont été développés.Les techniques de vision omnidirectionnelle artificielle recueillies ont été finalement exploitées dans deux applications réelles. La première concerne l’intégration des caméras catadioptriques à un robot mobile. La seconde porte sur la conception d’un suiveur solaire, à base d’une caméra catadioptrique.Les résultats obtenus confirment que l’adoption de tels capteurs pour les systèmes autonomes offre plus de performances et de flexibilité en regards aux capteurs classiques

Intraoperative endoskopische Registrierung von engen Bohrungen im Mastoid

Eine Stichkanalbohrung durch den Mastoid gewährt einen minimalinvasiven Zugang zur Cochlea, verläuft jedoch nahe wichtiger Nervenbahnen im menschlichen Schädel. Gegenüber dem konventionellen Auffräsen von Hand geht dem Chirurgen dabei die unmittelbare Sicht auf Risikostrukturen im Operationsgebiet verloren - der Verlauf der Bohrtrajektorie ist für ihn nicht zu erkennen. Die wichtigen Geschmacks- und Gesichtsnerven werden daher einem erhöhten Verletzungsrisiko ausgesetzt und das Vertrauen von Chirurgen und Patienten in den Eingriff kann beeinträchtigt werden. Eine intraoperative Kontrolle der Bohrtrajektorie ist deshalb wichtig, um einen sicheren Eingriff zu ermöglichen und Vertrauen zu bewahren. Bekannte Methoden zur intraoperativen Lageprüfung, die auf teuren, gesundheitsschädlichen oder im Operationssaal standardmäßig nicht vorhandenen technischen Einrichtungen basieren, werden in dieser Arbeit durch einen endoskopischen Ansatz ersetzt. Die schwammartige Mastoidknochenstruktur, die ein dreidimensionales Muster bietet, wird als Feld aus Merkmalen für eine Registrierung verwendet. Dafür werden in der minimalinvasiven Bohrung intraoperativ 2D-Endoskopaufnahmen erstellt, die mit präoperativen 3D-CT-Daten verglichen werden. Konkret handelt es sich um eine 2D-3D Bild-zu-Patient-Registrierung durch 2D-2D Bild-zu-Bild-Vergleiche. Einzelne Endoskopaufnahmen werden dazu zu einem Panoramabild zusammengefügt. Vergleichbare 2D-Bilder werden aus den 3D-CT-Daten in vielen verschiedenen Bohrkanallagen extrahiert. Je nach Lage unterscheidet sich die Struktur in den CT-Bildern, sodass über den Vergleich des Musters im Endoskopbild (realer Bohrkanal) zu den CT-Bildern (simulierter Bohrkanal) die größte Übereinstimmung und damit die Lage des Bohrkanals bestimmt werden kann. Für eine Vielfalt an sichtbaren Merkmalen ist ein Großteil der Bohrkanaloberfläche aufzuzeichnen. Verschiedene Endoskoptypen führen, in Kombination mit Stitching-Techniken, zu Panoramaaufnahmen der Mastoidstruktur an der Bohrkanaloberfläche. Kommerzielle Optiken werden auf ihre Eignung hin untersucht und eine miniaturisierte Rundblickoptik entwickelt. Die Stitching-Methoden werden speziell an die endoskopischen Bilddaten der verschiedenen Optiken angepasst. Das Vorgehen zur Registrierung wird sowohl simulativ als auch in aufeinander aufbauenden Mastoidphantomen untersucht. Erreichbare Genauigkeiten werden ausgewertet, während die beteiligten Parameter Suchraumdichte, Bohrkanaldurchmesser, Größe des endoskopierten Bereichs und die Auflösung der CT-Daten variiert werden, um deren Einflüsse zu ermitteln. Endoskopische Aufnahmen in Humanpräparaten zeigen den Transfer in die Praxis und die Integration in den Verlauf einer minimalinvasiven Cochleaimplantation. Eine Registrierung ermöglicht sowohl die Kontrolle der Bohrkanallage als auch eine intraoperative Korrektur und kann so eine vom gewünschten Pfad abweichende Bohrung berichtigen.A branch canal drilling through the mastoid provides a minimally invasive access to the cochlea, but passes nearby important nerve tracts in the human skull. Compared to conventional milling by hand, the surgeon loses the immediate view of high-risk structures in the operating area - the course of the drilling trajectory is not visible to him. The important taste and facial nerves are therefore exposed to an increased risk of injury and the confidence of surgeons and patients in the surgical intervention may be impaired. Intraoperative control of the drilling trajectory is therefore important to enable a safe intervention and to maintain confidence. Known methods for intraoperative pose verification, which are based on expensive, harmful or by default non-existent technical equipment in the operating room, are replaced by an endoscopic approach in this thesis. The sponge-like mastoid bone structure, providing a three-dimensional pattern, is used as a field of features for registration. For this purpose, 2D endoscopic images are recorded intraoperatively in the minimally invasive drill hole and compared to preoperative 3D CT data. In concrete terms, this is an 2D-3D image-to-patient registration using 2D-2D image-to-image comparisons. Individual endoscopic images are combined to form a panoramic image. Comparable 2D images are extracted from the 3D CT data in many different drill hole poses. Depending on the pose, the pattern in the CT images differs, so that by comparing the pattern in the endoscope image (real drill hole) to the CT images (simulated drill hole), the best match and thus the position of the drill hole can be determined. For a variety of visible features, a large portion of the drill hole surface must be recorded. Different types of endoscopes in combination with stitching techniques lead to panoramic images of the mastoid structure at the drill hole surface. Commercial optics are investigated for their suitability and a miniaturized panoramic optics is developed. The stitching methods are specially tailored to the endoscopic image data of the different optics. The procedure for the registration is evaluated in simulations as well as in mastoid phantoms that build on one another. Achievable accuracies are evaluated while the parameters involved -- search space density, drill hole diameter, size of the area recorded via endoscope and the resolution of the CT data -- are varied to determine their influences. Endoscopic recordings in human specimens show the transfer into practice and the integration into the procedure of a minimally invasive cochlear implantation. A registration allows for both, the verification of the drill hole pose as well as an intraoperative correction, and can thus correct a drill hole that deviates from the desired pat

Konforme geometrische Algebra und deren Anwendungen auf stochastische Optimierungsprobleme im Bereich 3D-Vision

In the present work, the modeling capabilities of conformal geometric algebra (CGA) are harnessed to approach typical problems from the research field of 3D-vision. This increasingly popular methodology is then extended in a new fashion by the integration of a least squares technique into the framework of CGA. Specifically, choosing the linear Gauss-Helmert model as the basis, the most general variant of least squares adjustment can be brought into operation. The result is a new versatile parameter estimation, termed GH-method, that reconciles two different mathematical areas, that is algebra and stochastics, under the umbrella of geometry. The main concern of the thesis is to show up the advantages inhering with this combination. Monocular pose estimation, from the subject 3D-vision, is the applicational focus of this thesis; given a picture of a scene, position and orientation of the image capturing vision system with respect to an external coordinate system define the pose. The developed parameter estimation technique is applied to different variants of this problem. Parameters are encoded by the algebra elements, called multivectors. They can be geometric objects as a circle, geometric operators as a rotation or likewise the pose. In the conducted pose experiments, observations are image pixels with associated uncertainties. The high accuracy achieved throughout all experiments confirms the competitiveness of the proposed estimation technique. Central to this work is also the consideration of omnidirectional vision using a paracatadioptric imaging sensor. It is demonstrated that CGA provides the ideal framework to model the related image formation. Two variants of the perspective pose estimation problem are adapted to the omnidirectional case. A new formalization of the epipolar geometry of two images in terms of CGA is developed, from which new insights into the structures behind the essential and the fundamental matrix, respectively, are drawn. Renowned standard approaches are shown to implicitly make use of CGA. Finally, an invocation of the GH-method for estimating epipoles is presented. Experimental results substantiate the goodness of this approach. Next to the detailed elucidations on parameter estimation, this text also gives a comprehensive introduction to geometric algebra, its tensor representation, the conformal space and the respective conformal geometric algebra. A valuable contribution is especially the analytic investigation into the geometric capabilities of CGA.Die vorliegende Arbeit ist motiviert durch die im Forschungszweig Computer Vision (CV) der Informatik typisch auftretenden geometrischen Problemstellungen auf der Grundlage von digitalen Bildaufnahmen. Hierzu zählt die Berechnung einer optimal durch eine Menge von Bildpunkten verlaufende Kurve, die Bestimmung der Epipolargeometrie, das Schätzen der Pose eines Objektes oder die 3D-Rekonstruktion. Diese Klasse von Problemen lässt sich durch den Einsatz der geometrischen Algebra (GA) – so werden unter geometrischen Aspekten besonders interessante Clifford Algebren bezeichnet – in überaus prägnanter und geschlossener Form modellieren. Dieser mit wachsender Akzeptanz verfolgte Ansatz, der beständig durch den Lehrstuhl „Kognitive Systeme“ der Universität Kiel weiterentwickelt wird, ist zentraler Bestandteile der Dissertation. Speziell wird die „konforme geometrische Algebra“ (CGA), die auf einer nicht-linearen Einbettung des euklidischen 3D-Raumes in einen fünfdimensionalen projektiven konformen Raum beruht, eingesetzt. Die Elemente dieser Algebra erlauben die Repräsentation geometrischer Basisentitäten, im wesentlichen Punkte, Linien, Kreise, Kugeln und Ebenen. Eine Vielzahl von Operationen ist möglich; besonders interessant sind die Transformationen der enthaltenen konformen Gruppe sowie die Möglichkeit algebraisch mit Unterräumen zu rechnen, d.h. diese zu vergrößern, zu schneiden oder Inzidenzen abzufragen. Den zweiten wichtigen Bestandteil der Arbeit stellt ein für die oben genannten Problemstellungen typisches stochastischen Verfahren dar – die Ausgleichsrechnung nach der Methode der kleinsten Quadrate. Deren allgemeinste Form erwächst aus der Verwendung des aus der Geodäsie bekannten linearen Gauß-Helmert (GH) Modells. Der resultierende GH-Schätzer zeigt alle Optimalitätseigenschaften wie minimale Varianz und Erwartungstreue. Eine der geometrischen Algebra inhärente Tensordarstellung stellt eine geeignete numerische Schnittstelle zwischen CGA und der GH-Schätzmethode zur Verfügung. Aufgrund der Bilinearität des Algebraprodukts lässt sich so ebenfalls das Konzept der Fehlerfortpflanzung, ein wichtiges Instrument der Ausgleichsrechnung, mit hoher Genauigkeit auf die Operationen der Algebra ausdehnen. Im Ergebnis entsteht ein neues universelles Parameterschätzverfahren zur Bestimmung der des jeweiligen Problems zugrundeliegenden Variablen. Ziel der vorliegenden Arbeit ist es auch, die aus der Verbindung von Algebra und Stochastik entstehenden Vorteile anhand von typischen CV-Anwendungen herauszustellen. Den Schwerpunkt hierfür bildet die Schätzung der Pose (Position und Orientierung eines Objekts bezüglich eines objektfremden Koordinatensystems), z.B. die eines Roboters anhand eines vom Roboter aufgenommenen Kamerabildes. Es wird ebenfalls gezeigt, dass CGA den optimalen Rahmen zur Modellierung omnidirektionaler Bildgebungsverfahren bietet, falls diese auf einem katadioptrischen System mit parabolischem Spiegel beruhen. Als omnidirektionale Anwendungen werden Posenschätzung sowie die Bestimmung der Epipolargeometrie präsentiert. Die erreichte Güte der GH-Parameterschätzung in den einzelnen Anwendungen wird jeweils durch experimentell gewonnene Resultate untermauert. Neben den umfangreichen Ausführungen zur Parameterschätzung liefert diese Arbeit auch eine detaillierte Einführung und Herleitung der geometrischen Algebra. Besonderes Augenmerk ist auch auf die analytische Darlegung der konformen geometrischen Algebra zu richten