Tracking 3D de objetos deformáveis em tempo real

Dissertação de mestrado em Engenharia InformáticaO tracking 3D de objetos é um tópico que tem sido amplamente estudado há vários anos. Apesar de já existirem diversas soluções robustas para tracking de objetos rígidos, quando se trata de objetos de formáveis o problema aumenta de complexidade. Nos últimos anos, tem-se assistido a um aumento da utilização de técnicas de Machine/Deep Learning para resolver problemas da área da visão por computa dor, incluindo o tracking 3D de objetos. Por outro lado, têm surgido diversos dispositivos de baixo custo (do género da Kinect) que permitem obter imagens RGB-D, as quais, além da informação de cor, con têm informação de profundidade. Nesta dissertação pretendeu-se estudar, desenvolver e implementar abordagens de tracking 3D de objetos deformáveis que recorram a técnicas de Machine/Deep Learning e tenham como input imagens RGB-D. Foi realizada uma abordagem de Deep Learning para o tracking do objeto deformável onde é utilizada uma arquitetura U-NET. Os testes realizados têm por base dois datasets e obteve-se resultados satisfatórios (IOU>0.8) em ambos os datasets. Para além disso, o tracking 3D é realizado em tempo real.3D object tracking is a topic that has been widely studied for several years. Although there are already several robust solutions for tracking rigid objects, when it comes to deformable objects the problem increases in complexity. In recent years, there has been an increase in the use of Machine / Deep Learning techniques to solve problems in the area of computer vision, including 3D object tracking. On the other hand, several low-cost devices (like Kinect) have appeared that allow obtaining RGB-D images, which, in addition to color information, contain depth information. In this dissertation we intend to study, develop and implement 3D tracking approaches for deformable objects that use Machine / Deep Learning techniques and have RGB-D images as input. An approach of Deep Learning was performed to track the deformable object where a U-NET architecture is used. The tests performed are based on two sets of data and satisfactory results were obtained (IOU> 0.8) in both sets of data. In addition, 3D tracking should be performed in real time

Utilization and experimental evaluation of occlusion aware kernel correlation filter tracker using RGB-D

Unlike deep-learning which requires large training datasets, correlation filter-based trackers like Kernelized Correlation Filter (KCF) uses implicit properties of tracked images (circulant matrices) for training in real-time. Despite their practical application in tracking, a need for a better understanding of the fundamentals associated with KCF in terms of theoretically, mathematically, and experimentally exists. This thesis first details the workings prototype of the tracker and investigates its effectiveness in real-time applications and supporting visualizations. We further address some of the drawbacks of the tracker in cases of occlusions, scale changes, object rotation, out-of-view and model drift with our novel RGB-D Kernel Correlation tracker. We also study the use of particle filter to improve trackers\u27 accuracy. Our results are experimentally evaluated using a) standard dataset and b) real-time using Microsoft Kinect V2 sensor. We believe this work will set the basis for better understanding the effectiveness of kernel-based correlation filter trackers and to further define some of its possible advantages in tracking

Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces