3D alignment for interactive evolutionary design

3D model alignment (‘Pose Normalization’ in the literature) is investigated as part of wider research into guided evolutionary Computer-Aided Design. CAD technology in development will combine human interaction and geometric optimization, within an evolutionary design system. Evolving shapes will be influenced by simple pre-set geometric fuzzy-constraints – internal voids and external bounding geometry created by users. To compare evolving candidate shapes with these pre-set constraints they must first be aligned (rotated, scaled, and co-located). A shortlist of five promising alignment techniques is described. Benchmark data generated using standard CAD functions (centre of gravity, principle axes etc.) will be presented at the conference

Retrieval of 3-Dimensional Rigid and Non-Rigid Objects

Η παρούσα διδακτορική διατριβή εστιάζει στο πρόβλημα της ανάκτησης 3Δ αντικειμένων από μεγάλες βάσεις δεδομένων σε σχεδόν πραγματικό χρόνο. Για την αντιμετώπιση του προβλήματος αυτού, η έρευνα επικεντρώνεται σε τρία βασικά υποπροβλήματα του χώρου: (α) κανονικοποίηση θέσης άκαμπτων 3Δ μοντέλων με εφαρμογές στην ανάκτηση 3Δ αντικειμένων, (β) περιγραφή εύκαμπτων 3Δ αντικειμένων και (γ) αναζήτηση από βάσεις δεδομένων 3Δ αντικειμένων βασιζόμενη σε 2Δ εικόνες-ερώτησης. Σχετικά με το πρώτο υποπρόβλημα, την κανονικοποίηση θέσης 3Δ μοντέλων, παρουσιάζονται τρεις νέες μέθοδοι οι οποίες βασίζονται στις εξής αρχές: (α) Τριδιάστατη Ανακλαστική Συμμετρία Αντικειμένου (ROSy) και (β, γ) Διδιάστατη Ανακλαστική Συμμετρία Αντικειμένου υπολογιζόμενη επί Πανοραμικών Προβολών (SymPan και SymPan+). Όσον αφορά το δεύτερο υποπρόβλημα, αναπτύχθηκε μια μέθοδος ανάκτησης εύκαμπτων 3Δ αντικειμένων, η οποία συνδυάζει τις ιδιότητες της σύμμορφης γεωμετρίας και της τοπολογικής πληροφορίας βασιζόμενης σε γράφους, με ενιαίο τρόπο (ConTopo++). Επιπλέον, προτείνεται μια στρατηγική συνταιριασμού συμβολοσειρών, για τη σύγκριση των γράφων που αναπαριστούν 3Δ αντικείμενα. Σχετικά με το τρίτο υποπρόβλημα, παρουσιάζεται μια μέθοδος ανάκτησης 3Δ αντικειμένων, βασιζόμενη σε 2Δ εικόνες-ερώτησης, οι οποίες αντιπροσωπεύουν προβολές πραγματικών 3Δ αντικειμένων. Τα πλήρη 3Δ αντικείμενα της βάσης δεδομένων περιγράφονται από ένα σύνολο πανοραμικών προβολών και ένα μοντέλο Bag-of-Visual-Words δημιουργείται χρησιμοποιώντας τα χαρακτηριστικά SIFT που προέρχονται από αυτά. Οι μεθοδολογίες που αναπτύχθηκαν και περιγράφονται στην παρούσα διατριβή αξιολογούνται όσον αφορά την ακρίβεια ανάκτησης και παρουσιάζονται κάνοντας χρήση ποσοτικών και ποιοτικών μέτρων μέσω μιας εκτεταμένης και συνεκτικής αξιολόγησης σε σχέση με μεθόδους τρέχουσας τεχνολογικής στάθμης επάνω σε τυποποιημένες βάσεις δεδομένων.This dissertation focuses on the problem of 3D object retrieval from large datasets in a near realtime manner. In order to address this task we focus on three major subproblems of the field: (i) pose normalization of rigid 3D models with applications to 3D object retrieval, (ii) non-rigid 3D object description and (iii) search over rigid 3D object datasets based on 2D image queries. Regarding the first of the three subproblems, 3D model pose normalization, three novel pose normalization methods are presented, based on: (i) 3D Reflective Object Symmetry (ROSy) and (ii, iii) 2D Reflective Object Symmetry computed on Panoramic Views (SymPan and SymPan+). Considering the second subproblem, a non-rigid 3D object retrieval methodology, based on the properties of conformal geometry and graph-based topological information (ConTopo++) has been developed. Furthermore, a string matching strategy for the comparison of graphs that describe 3D objects, is proposed. Regarding the third subproblem a 3D object retrieval method, based on 2D range image queries that represent partial views of real 3D objects, is presented. The complete 3D objects of the database are described by a set of panoramic views and a Bag-of-Visual-Words model is built using SIFT features extracted from them. The methodologies developed and described in this dissertation are evaluated in terms of retrieval accuracy and demonstrated using both quantitative and qualitative measures via an extensive consistent evaluation against state-of-the-art methods on standard datasets

Geometric guides for interactive evolutionary design

This thesis describes the addition of novel Geometric Guides to a generative Computer-Aided Design (CAD) application that supports early-stage concept generation. The application generates and evolves abstract 3D shapes, used to inspire the form of new product concepts. It was previously a conventional Interactive Evolutionary system where users selected shapes from evolving populations. However, design industry users wanted more control over the shapes, for example by allowing the system to influence the proportions of evolving forms. The solution researched, developed, integrated and tested is a more cooperative human-machine system combining classic user interaction with innovative geometric analysis. In the literature review, different types of Interactive Evolutionary Computation (IEC), Pose Normalisation (PN), Shape Comparison, and Minimum-Volume Bounding Box approaches are compared, with some of these technologies identified as applicable for this research. Using its Application Programming Interface, add-ins for the Siemens NX CAD system have been developed and integrated with an existing Interactive Evolutionary CAD system. These add-ins allow users to create a Geometric Guide (GG) at the start of a shape exploration session. Before evolving shapes can be compared with the GG, they must be aligned and scaled (known as Pose Normalisation in the literature). Computationally-efficient PN has been achieved using geometric functions such as Bounding Box for translation and scaling, and Principle Axes for the orientation. A shape comparison algorithm has been developed that is based on the principle of non-intersecting volumes. This algorithm is also implemented with standard, readily available geometric functions, is conceptually simple, accessible to other researchers and also offers appropriate efficacy. Objective geometric testing showed that the PN and Shape Comparison methods developed are suitable for this guiding application and can be efficiently adapted to enhance an Interactive Evolutionary Design system. System performance with different population sizes was examined to indicate how best to use the new guiding capabilities to assist users in evolutionary shape searching. This was backed up by participant testing research into two user interaction strategies. A Large Background Population (LBP) approach where the GG is used to select a sub-set of shapes to show to the user was shown to be the most effective. The inclusion of Geometric Guides has taken the research from the existing aesthetic focused tool to a system capable of application to a wider range of engineering design problems. This system supports earlier design processes and ideation in conceptual design and allows a designer to experiment with ideas freely to interactively explore populations of evolving solutions. The design approach has been further improved, and expanded beyond the previous quite limited scope of form exploration

Visual Odometry and Sparse Scene Reconstruction for UAVs with a Multi-Fisheye Camera System

Autonomously operating UAVs demand a fast localization for navigation, to actively explore unknown areas and to create maps. For pose estimation, many UAV systems make use of a combination of GPS receivers and inertial sensor units (IMU). However, GPS signal coverage may go down occasionally, especially in the close vicinity of objects, and precise IMUs are too heavy to be carried by lightweight UAVs. This and the high cost of high quality IMU motivate the use of inexpensive vision based sensors for localization using visual odometry or visual SLAM (simultaneous localization and mapping) techniques. The first contribution of this thesis is a more general approach to bundle adjustment with an extended version of the projective coplanarity equation which enables us to make use of omnidirectional multi-camera systems which may consist of fisheye cameras that can capture a large field of view with one shot. We use ray directions as observations instead of image points which is why our approach does not rely on a specific projection model assuming a central projection. In addition, our approach allows the integration and estimation of points at infinity, which classical bundle adjustments are not capable of. We show that the integration of far or infinitely far points stabilizes the estimation of the rotation angles of the camera poses. In its second contribution, we employ this approach to bundle adjustment in a highly integrated system for incremental pose estimation and mapping on light-weight UAVs. Based on the image sequences of a multi-camera system our system makes use of tracked feature points to incrementally build a sparse map and incrementally refines this map using the iSAM2 algorithm. Our system is able to optionally integrate GPS information on the level of carrier phase observations even in underconstrained situations, e.g. if only two satellites are visible, for georeferenced pose estimation. This way, we are able to use all available information in underconstrained GPS situations to keep the mapped 3D model accurate and georeferenced. In its third contribution, we present an approach for re-using existing methods for dense stereo matching with fisheye cameras, which has the advantage that highly optimized existing methods can be applied as a black-box without modifications even with cameras that have field of view of more than 180 deg. We provide a detailed accuracy analysis of the obtained dense stereo results. The accuracy analysis shows the growing uncertainty of observed image points of fisheye cameras due to increasing blur towards the image border. Core of the contribution is a rigorous variance component estimation which allows to estimate the variance of the observed disparities at an image point as a function of the distance of that point to the principal point. We show that this improved stochastic model provides a more realistic prediction of the uncertainty of the triangulated 3D points.Autonom operierende UAVs benötigen eine schnelle Lokalisierung zur Navigation, zur Exploration unbekannter Umgebungen und zur Kartierung. Zur Posenbestimmung verwenden viele UAV-Systeme eine Kombination aus GPS-Empfängern und Inertial-Messeinheiten (IMU). Die Verfügbarkeit von GPS-Signalen ist jedoch nicht überall gewährleistet, insbesondere in der Nähe abschattender Objekte, und präzise IMUs sind für leichtgewichtige UAVs zu schwer. Auch die hohen Kosten qualitativ hochwertiger IMUs motivieren den Einsatz von kostengünstigen bildgebenden Sensoren zur Lokalisierung mittels visueller Odometrie oder SLAM-Techniken zur simultanen Lokalisierung und Kartierung. Im ersten wissenschaftlichen Beitrag dieser Arbeit entwickeln wir einen allgemeineren Ansatz für die Bündelausgleichung mit einem erweiterten Modell für die projektive Kollinearitätsgleichung, sodass auch omnidirektionale Multikamerasysteme verwendet werden können, welche beispielsweise bestehend aus Fisheyekameras mit einer Aufnahme einen großen Sichtbereich abdecken. Durch die Integration von Strahlrichtungen als Beobachtungen ist unser Ansatz nicht von einem kameraspezifischen Abbildungsmodell abhängig solange dieses der Zentralprojektion folgt. Zudem erlaubt unser Ansatz die Integration und Schätzung von unendlich fernen Punkten, was bei klassischen Bündelausgleichungen nicht möglich ist. Wir zeigen, dass durch die Integration weit entfernter und unendlich ferner Punkte die Schätzung der Rotationswinkel der Kameraposen stabilisiert werden kann. Im zweiten Beitrag verwenden wir diesen entwickelten Ansatz zur Bündelausgleichung für ein System zur inkrementellen Posenschätzung und dünnbesetzten Kartierung auf einem leichtgewichtigen UAV. Basierend auf den Bildsequenzen eines Mulitkamerasystems baut unser System mittels verfolgter markanter Bildpunkte inkrementell eine dünnbesetzte Karte auf und verfeinert diese inkrementell mittels des iSAM2-Algorithmus. Unser System ist in der Lage optional auch GPS Informationen auf dem Level von GPS-Trägerphasen zu integrieren, wodurch sogar in unterbestimmten Situation - beispielsweise bei nur zwei verfügbaren Satelliten - diese Informationen zur georeferenzierten Posenschätzung verwendet werden können. Im dritten Beitrag stellen wir einen Ansatz zur Verwendung existierender Methoden für dichtes Stereomatching mit Fisheyekameras vor, sodass hoch optimierte existierende Methoden als Black Box ohne Modifzierungen sogar mit Kameras mit einem Gesichtsfeld von mehr als 180 Grad verwendet werden können. Wir stellen eine detaillierte Genauigkeitsanalyse basierend auf dem Ergebnis des dichten Stereomatchings dar. Die Genauigkeitsanalyse zeigt, wie stark die Genauigkeit beobachteter Bildpunkte bei Fisheyekameras zum Bildrand aufgrund von zunehmender Unschärfe abnimmt. Das Kernstück dieses Beitrags ist eine Varianzkomponentenschätzung, welche die Schätzung der Varianz der beobachteten Disparitäten an einem Bildpunkt als Funktion von der Distanz dieses Punktes zum Hauptpunkt des Bildes ermöglicht. Wir zeigen, dass dieses verbesserte stochastische Modell eine realistischere Prädiktion der Genauigkeiten der 3D Punkte ermöglicht

Enhancing 3D Visual Odometry with Single-Camera Stereo Omnidirectional Systems

We explore low-cost solutions for efficiently improving the 3D pose estimation problem of a single camera moving in an unfamiliar environment. The visual odometry (VO) task -- as it is called when using computer vision to estimate egomotion -- is of particular interest to mobile robots as well as humans with visual impairments. The payload capacity of small robots like micro-aerial vehicles (drones) requires the use of portable perception equipment, which is constrained by size, weight, energy consumption, and processing power. Using a single camera as the passive sensor for the VO task satisfies these requirements, and it motivates the proposed solutions presented in this thesis. To deliver the portability goal with a single off-the-shelf camera, we have taken two approaches: The first one, and the most extensively studied here, revolves around an unorthodox camera-mirrors configuration (catadioptrics) achieving a stereo omnidirectional system (SOS). The second approach relies on expanding the visual features from the scene into higher dimensionalities to track the pose of a conventional camera in a photogrammetric fashion. The first goal has many interdependent challenges, which we address as part of this thesis: SOS design, projection model, adequate calibration procedure, and application to VO. We show several practical advantages for the single-camera SOS due to its complete 360-degree stereo views, that other conventional 3D sensors lack due to their limited field of view. Since our omnidirectional stereo (omnistereo) views are captured by a single camera, a truly instantaneous pair of panoramic images is possible for 3D perception tasks. Finally, we address the VO problem as a direct multichannel tracking approach, which increases the pose estimation accuracy of the baseline method (i.e., using only grayscale or color information) under the photometric error minimization as the heart of the “direct” tracking algorithm. Currently, this solution has been tested on standard monocular cameras, but it could also be applied to an SOS. We believe the challenges that we attempted to solve have not been considered previously with the level of detail needed for successfully performing VO with a single camera as the ultimate goal in both real-life and simulated scenes

Augmented Reality and Artificial Intelligence in Image-Guided and Robot-Assisted Interventions

In minimally invasive orthopedic procedures, the surgeon places wires, screws, and surgical implants through the muscles and bony structures under image guidance. These interventions require alignment of the pre- and intra-operative patient data, the intra-operative scanner, surgical instruments, and the patient. Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. State of the art approaches often support the surgeon by using external navigation systems or ill-conditioned image-based registration methods that both have certain drawbacks. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. Consequently, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. This dissertation investigates the applications of AR, artificial intelligence, and robotics in interventional medicine. Our solutions were applied in a broad spectrum of problems for various tasks, namely improving imaging and acquisition, image computing and analytics for registration and image understanding, and enhancing the interventional visualization. The benefits of these approaches were also discovered in robot-assisted interventions. We revealed how exemplary workflows are redefined via AR by taking full advantage of head-mounted displays when entirely co-registered with the imaging systems and the environment at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The system's awareness of the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. We also leveraged the principles governing image formation and combined it with deep learning and RGBD sensing to fuse images and reconstruct interventional data. We hope that our holistic approaches towards improving the interface of surgery and enhancing the usability of interventional imaging, not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications

Low-rank Based Algorithms for Rectification, Repetition Detection and De-noising in Urban Images

In this thesis, we aim to solve the problem of automatic image rectification and repeated patterns detection on 2D urban images, using novel low-rank based techniques. Repeated patterns (such as windows, tiles, balconies and doors) are prominent and significant features in urban scenes. Detection of the periodic structures is useful in many applications such as photorealistic 3D reconstruction, 2D-to-3D alignment, facade parsing, city modeling, classification, navigation, visualization in 3D map environments, shape completion, cinematography and 3D games. However both of the image rectification and repeated patterns detection problems are challenging due to scene occlusions, varying illumination, pose variation and sensor noise. Therefore, detection of these repeated patterns becomes very important for city scene analysis. Given a 2D image of urban scene, we automatically rectify a facade image and extract facade textures first. Based on the rectified facade texture, we exploit novel algorithms that extract repeated patterns by using Kronecker product based modeling that is based on a solid theoretical foundation. We have tested our algorithms in a large set of images, which includes building facades from Paris, Hong Kong and New York

TwinTex: Geometry-aware Texture Generation for Abstracted 3D Architectural Models

Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex, the first automatic texture mapping framework to generate a photo-realistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort. Project page: https://vcc.tech/research/2023/TwinTex.Comment: Accepted to SIGGRAPH ASIA 202