Probabilistic RGB-D Odometry based on Points, Lines and Planes Under Depth Uncertainty

This work proposes a robust visual odometry method for structured environments that combines point features with line and plane segments, extracted through an RGB-D camera. Noisy depth maps are processed by a probabilistic depth fusion framework based on Mixtures of Gaussians to denoise and derive the depth uncertainty, which is then propagated throughout the visual odometry pipeline. Probabilistic 3D plane and line fitting solutions are used to model the uncertainties of the feature parameters and pose is estimated by combining the three types of primitives based on their uncertainties. Performance evaluation on RGB-D sequences collected in this work and two public RGB-D datasets: TUM and ICL-NUIM show the benefit of using the proposed depth fusion framework and combining the three feature-types, particularly in scenes with low-textured surfaces, dynamic objects and missing depth measurements.Comment: Major update: more results, depth filter released as opensource, 34 page

Refractive Structure-From-Motion Through a Flat Refractive Interface

Recovering 3D scene geometry from underwater images involves the Refractive Structure-from-Motion (RSfM) problem, where the image distortions caused by light refraction at the interface between different propagation media invalidates the single view point assumption. Direct use of the pinhole camera model in RSfM leads to inaccurate camera pose estimation and consequently drift. RSfM methods have been thoroughly studied for the case of a thick glass interface that assumes two refractive interfaces between the camera and the viewed scene. On the other hand, when the camera lens is in direct contact with the water, there is only one refractive interface. By explicitly considering a refractive interface, we develop a succinct derivation of the refractive fundamental matrix in the form of the generalised epipolar constraint for an axial camera. We use the refractive fundamental matrix to refine initial pose estimates obtained by assuming the pinhole model. This strategy allows us to robustly estimate underwater camera poses, where other methods suffer from poor noise-sensitivity. We also formulate a new four view constraint enforcing camera pose consistency along a video which leads us to a novel RSfM framework. For validation we use synthetic data to show the numerical properties of our method and we provide results on real data to demonstrate performance within laboratory settings and for applications in endoscopy

A surgical system for automatic registration, stiffness mapping and dynamic image overlay

In this paper we develop a surgical system using the da Vinci research kit (dVRK) that is capable of autonomously searching for tumors and dynamically displaying the tumor location using augmented reality. Such a system has the potential to quickly reveal the location and shape of tumors and visually overlay that information to reduce the cognitive overload of the surgeon. We believe that our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and the shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on phantom prostate models with embedded stiff inclusions.Comment: International Symposium on Medical Robotics (ISMR 2018

Faster than the blink of an eye

Arboreal snakes such as the amazon tree boa (Corallus hortulanus) are able to cantilever large sections of their body for very long periods of time with seemingly minimal muscular effort. From this cantilevered position they exert quick strikes as well as compensate for any movement of the object from which they cantilever. The mechanisms of muscle coordination required for the dynamic switch between resting and strike have been hypothesized for terrestrial puff adders (Bitis arietans) to result from the vast and unique musculo-tendon arrangement of the snake's epaxial muscles put under pre-strike tension, i.e. a spring-like mechanism where the snake is able to store a large amount of energy in tendons which can then be later quickly released. Furthermore, while muscle activity during gap crossing or extension activities has been described for an arboreal species, it is not clear how the stationary snake's muscles compensate for perturbations of the anchoring object, e.g. as happens in the wild with a branch swaying in the breeze. Using a self-built high-speed 3D tracking system along with a novel method for collecting chronic multi-electrode bipolar electromyography (EMG) information, my data is not only unsupportive of the elastic energy-storing strike hypothesis but provides insights to the muscle coordination required for stabilization in a moving, as well as stationary, environment

Minimal Solutions to Geometric Problems with Multiple Cameras or Multiple Sensor Modalities

Tese de doutoramento em Engenharia Electrotécnica e de Computadores, no ramo de Especialização em Automação e Robótica, apresentada ao Departamento de Engenharia Electrotécnica e de Computadores da Universidade de CoimbraThis thesis addresses minimal problems that involve multiple cameras or a combination of cameras with other sensors, particularly focusing on four cases: extrinsic calibration between a camera and a laser rangefinder (LRF); full calibration of an ultrasound array (US) with a camera; full calibration of a camera within a calibrated network; relative pose between axial systems. The first problem (LRF-Camera) is highly important in the context of mobile robotics in order to fuse the information of an LRF and a Camera in localization maps. The second problem (US-Camera) is becoming increasingly relevant in the context of medical imaging to perform guided intervention and 3D reconstruction with US probes. Both these problems use a planar calibration target to obtain a minimal solution from 3 and 4 correspondences respectively. They are formulated as the registration between planes detected by the camera and lines detected by either the LRF or the US. The third problem (Camera-Network) is concerned with two application scenarios: addition of a new camera to a calibrated network, and tracking of a hand-held camera within the field of view of a calibrated network. The last problem (Axial System) has its main application in motion estimation of stereo camera pairs. Both these problems introduce a 5-dimensional linear subspace to model line incidence relations of an axial system, of which a pair of calibrated cameras is a particular example. In the Camera-Network problem a generalized fundamental matrix is derived to obtain a 11-correspondence minimal solution. In the Axial System problem a generalized essential matrix is derived to obtain a 10-correspondence non-minimal solution. Although it should be possible to solve this last problem with as few as 6 correspondences, the proposed solution is the closest to minimal in the literature. Additionally this thesis addresses the use of the RANSAC framework in the context of the problems mentioned above. While RANSAC is the most widely used method in computer vision for robust estimation when minimal solutions are available, it cannot be applied directly to some of the problems discussed here. A new framework -- multi-RANSAC -- is presented as an adaptation of RANSAC to problems with multiple sampling datasets. Problems with multiple cameras or multiple sensors often fall in this category and thus this new framework can greatly improve their results. Its applicability is demonstrated in both the US-Camera and the Camera-Network problems.Esta tese aborda os problemas mínimos no contexto de visão por computador, isto é, problemas com o mesmo número de restrições e de parâmetros desconhecidos, para os quais existe um conjunto finito e discreto de soluções. A tese foca-se em particular nos seguintes problemas: calibração extrínseca entre uma câmara e um sensor laser rangefinder (LRF); calibração completa de uma sonda ultrasom (US) com uma câmara; calibração completa de uma câmara dentro de uma rede calibrada; estimação de pose relativa entre sistema axiais. O primeiro problema (LRF-Camera) é extremamente importante no contexto de robótica móvel para fundir a informação de um sensor LRF e uma câmara em mapas de localização. O segundo problema (US-Camera) está-se a tornar cada vez mais relevante no contexto de imagiologia médica para realizar intervenções guiadas e reconstrução 3D com sondas ecográficas. Ambos os problemas usam um alvo de calibração planar para obter uma solução mínima usando 3 e 4 correspondências respectivamente, e são formulados como o registo 3D entre planos detectados pela câmara e linhas detectadas pelo LRF ou US. O terceiro problema (Camera-Network) tem duas aplicações em mente: a introdução de uma nova câmara numa rede calibrada, e o seguimento de uma câmara guiada manualmente dentro do campo de visão de uma rede calibrada. O último problema (Axial System) tem a sua maior aplicação na estimação de pose relativa entre pares de câmaras estéreo. Em ambos os problemas é introduzido um subespaço linear em 5 dimensões que modela as relações de incidência de linhas num sistema axial, do qual as câmaras estéreo são um caso particular. No problema Camera- Network é introduzida uma generalização da matriz fundamental que permite obter uma solução mínima com 11 correspondências. No problema Axial System é introduzida uma generalização da matrix essencial que permite obter uma solução não mínima com 10 correspondências. Apesar de ser possível, em teoria, resolver este último problema com apenas 6 correspondências, a solução apresentada nesta tese usa um menor número de correspondências que as alternativas existentes. Adicionalmente esta tese aborda o uso de RANSAC no contexto dos problemas anteriormente descritos. O RANSAC é o estimador robusto mais utilizado em visão por computador quando existem soluções mínimas para um determinado problema, no entanto não pode ser aplicado directamente em algumas das aplicações aqui descritas. Um novo método é proposto – multiset-RANSAC – que adapta o RANSAC para situações que envolvem a amostragem de múltiplos conjuntos de dados. Os problemas com múltiplas câmaras ou múltiplos sensores encontram-se mutas vezes nesta categoria, tornando o multiset-RANSAC numa ferramenta que pode melhorar bastante os resultados em alguns dos problemas focados nesta tese. A utilidade deste método é demonstrada nos problemas US-Camera e Camera-Network

Feature Based Calibration of a Network of Kinect Sensors

The availability of affordable depth sensors in conjunction with common RGB cameras, such as the Microsoft Kinect, can provide robots with a complete and instantaneous representation of the current surrounding environment. However, in the problem of calibrating multiple camera systems, traditional methods bear some drawbacks, such as requiring human intervention. In this thesis, we propose an automatic and reliable calibration framework that can easily estimate the extrinsic parameters of a Kinect sensor network. Our framework includes feature extraction, Random Sample Consensus and camera pose estimation from high accuracy correspondences. We also implement a robustness analysis of position estimation algorithms. The result shows that our system could provide precise data under certain amount noise. Keywords Kinect, Multiple Camera Calibration, Feature Points Extraction, Correspondence, RANSA

Development of an augmented reality guided computer assisted orthopaedic surgery system

Previously held under moratorium from 1st December 2016 until 1st December 2021.This body of work documents the developed of a proof of concept augmented reality guided computer assisted orthopaedic surgery system – ARgCAOS. After initial investigation a visible-spectrum single camera tool-mounted tracking system based upon fiducial planar markers was implemented. The use of visible-spectrum cameras, as opposed to the infra-red cameras typically used by surgical tracking systems, allowed the captured image to be streamed to a display in an intelligible fashion. The tracking information defined the location of physical objects relative to the camera. Therefore, this information allowed virtual models to be overlaid onto the camera image. This produced a convincing augmented experience, whereby the virtual objects appeared to be within the physical world, moving with both the camera and markers as expected of physical objects. Analysis of the first generation system identified both accuracy and graphical inadequacies, prompting the development of a second generation system. This too was based upon a tool-mounted fiducial marker system, and improved performance to near-millimetre probing accuracy. A resection system was incorporated into the system, and utilising the tracking information controlled resection was performed, producing sub-millimetre accuracies. Several complications resulted from the tool-mounted approach. Therefore, a third generation system was developed. This final generation deployed a stereoscopic visible-spectrum camera system affixed to a head-mounted display worn by the user. The system allowed the augmentation of the natural view of the user, providing convincing and immersive three dimensional augmented guidance, with probing and resection accuracies of 0.55±0.04 and 0.34±0.04 mm, respectively.This body of work documents the developed of a proof of concept augmented reality guided computer assisted orthopaedic surgery system – ARgCAOS. After initial investigation a visible-spectrum single camera tool-mounted tracking system based upon fiducial planar markers was implemented. The use of visible-spectrum cameras, as opposed to the infra-red cameras typically used by surgical tracking systems, allowed the captured image to be streamed to a display in an intelligible fashion. The tracking information defined the location of physical objects relative to the camera. Therefore, this information allowed virtual models to be overlaid onto the camera image. This produced a convincing augmented experience, whereby the virtual objects appeared to be within the physical world, moving with both the camera and markers as expected of physical objects. Analysis of the first generation system identified both accuracy and graphical inadequacies, prompting the development of a second generation system. This too was based upon a tool-mounted fiducial marker system, and improved performance to near-millimetre probing accuracy. A resection system was incorporated into the system, and utilising the tracking information controlled resection was performed, producing sub-millimetre accuracies. Several complications resulted from the tool-mounted approach. Therefore, a third generation system was developed. This final generation deployed a stereoscopic visible-spectrum camera system affixed to a head-mounted display worn by the user. The system allowed the augmentation of the natural view of the user, providing convincing and immersive three dimensional augmented guidance, with probing and resection accuracies of 0.55±0.04 and 0.34±0.04 mm, respectively