Técnicas de coste reducido para el posicionamiento del paciente en radioterapia percutánea utilizando un sistema de imágenes ópticas

Patient positioning is an important part of radiation therapy which is one of the main solutions for the treatment of malignant tissue in the human body. Currently, the most common patient positioning methods expose healthy tissue of the patient's body to extra dangerous radiations. Other non-invasive positioning methods are either not very accurate or are very costly for an average hospital. In this thesis, we explore the possibility of developing a system comprised of affordable hardware and advanced computer vision algorithms that facilitates patient positioning. Our algorithms are based on the usage of affordable RGB-D sensors, image features, ArUco planar markers, and other geometry registration methods. Furthermore, we take advantage of consumer-level computing hardware to make our systems widely accessible. More specifically, we avoid the usage of approaches that need to take advantage of dedicated GPU hardware for general-purpose computing since they are more costly. In different publications, we explore the usage of the mentioned tools to increase the accuracy of reconstruction/localization of the patient in its pose. We also take into account the visualization of the patient's target position with respect to their current position in order to assist the person who performs patient positioning. Furthermore, we make usage of augmented reality in conjunction with a real-time 3D tracking algorithm for better interaction between the program and the operator. We also solve more fundamental problems about ArUco markers that could be used in the future to improve our systems. These include highquality multi-camera calibration and mapping using ArUco markers plus detection of these markers in event cameras which are very useful in the presence of fast camera movement. In the end, we conclude that it is possible to increase the accuracy of 3D reconstruction and localization by combining current computer vision algorithms with fiducial planar markers with RGB-D sensors. This is reflected in the low amount of error we have achieved in our experiments for patient positioning, pushing forward the state of the art for this application.En el tratamiento de tumores malignos en el cuerpo, el posicionamiento del paciente en las sesiones de radioterapia es una cuestión crucial. Actualmente, los métodos más comunes de posicionamiento del paciente exponen tejido sano del mismo a radiaciones peligrosas debido a que no es posible asegurar que la posición del paciente siempre sea la misma que la que tuvo cuando se planificó la zona a radiar. Los métodos que se usan actualmente, o no son precisos o tienen costes que los hacen inasequibles para ser usados en hospitales con financiación limitada. En esta Tesis hemos analizado la posibilidad de desarrollar un sistema compuesto por hardware de bajo coste y métodos avanzados de visión por ordenador que ayuden a que el posicionamiento del paciente sea el mismo en las diferentes sesiones de radioterapia, con respecto a su pose cuando fue se planificó la zona a radiar. La solución propuesta como resultado de la Tesis se basa en el uso de sensores RGB-D, características extraídas de la imagen, marcadores cuadrados denominados ArUco y métodos de registro de la geometría en la imagen. Además, en la solución propuesta, se aprovecha la existencia de hardware convencional de bajo coste para hacer nuestro sistema ampliamente accesible. Más específicamente, evitamos el uso de enfoques que necesitan aprovechar GPU, de mayores costes, para computación de propósito general. Se han obtenido diferentes publicaciones para conseguir el objetivo final. Las mismas describen métodos para aumentar la precisión de la reconstrucción y la localización del paciente en su pose, teniendo en cuenta la visualización de la posición ideal del paciente con respecto a su posición actual, para ayudar al profesional que realiza la colocación del paciente. También se han propuesto métodos de realidad aumentada junto con algoritmos para seguimiento 3D en tiempo real para conseguir una mejor interacción entre el sistema ideado y el profesional que debe realizar esa labor. De forma añadida, también se han propuesto soluciones para problemas fundamentales relacionados con el uso de marcadores cuadrados que han sido utilizados para conseguir el objetivo de la Tesis. Las soluciones propuestas pueden ser empleadas en el futuro para mejorar otros sistemas. Los problemas citados incluyen la calibración y el mapeo multicámara de alta calidad utilizando los marcadores y la detección de estos marcadores en cámaras de eventos, que son muy útiles en presencia de movimientos rápidos de la cámara. Al final, concluimos que es posible aumentar la precisión de la reconstrucción y localización en 3D combinando los actuales algoritmos de visión por ordenador, que usan marcadores cuadrados de referencia, con sensores RGB-D. Los resultados obtenidos con respecto al error que el sistema obtiene al reproducir el posicionamiento del paciente suponen un importante avance en el estado del arte de este tópico

RGBD Camera Pose Estimation Techniques, Slip Detection, and Occluded Object Search Strategies for Deformable Linear Object Features in Autonomous Robotic Space Task Execution

This thesis studies Robotic handling of Deformable Linear Objects (DLO). Many habitats used for space exploration include panels with multiple wires and connections which can be easily reconfigured by humans but very difficult to be handled autonomously by robotic systems due to the flexible nature of the wires. In some situations, the wires can come loose and get separated from their connections resulting in malfunctioning of some onboard systems. This thesis develops methods for autonomous handling of flexible wires (deformable linear objects) involving the unplugging and re-plugging or stowing of one end of the wire from a connection point. An anomaly situation may arise when the end of a gripped DLO slips away from the robotic end effector into the environment while being maneuvered, entering the object into anunknown state. The objective of the research presented herein was to use purely visual sensing to detect this DLO slip locating the loose connector end, estimating its pose, and autonomously developing a motion plan for retrieval and delivery of the connector end to its originally intended destination. Three pose estimation methods are implemented: employing fiducial markers, RGBD image processing, and machine learning algorithms to generate the pose of the end of the DLO being manipulated. Experiments are performed using two cooperating robotic arms that show identification rates of 48.1%, 100.0%, and 77.8% and arm retrieval grasp rates of 48.1%, 74.1%, and 64.0% respectively among 27 trials. The identification rate varied based on the level of occlusion of the DLO end within the workspace. Slip detection is accomplished by comparing this estimated position’s distance to the manipulating arm’s end effector against a threshold quantifying a slip, producing a success rate of 77.2% from 18 slip trials. In the event that the loose connector settles out of the camera’s view, a spiral search pattern was designed to maneuver the secondary camera for further workspace inspection, with a search identification rate of 91.7% in 36 trials. The effectiveness of the overall system as a solution for anomaly detection and resolution is exhibited through three demonstrations with varying environmental configurations

A pilot study for the digital replacement of a distorted dentition acquired by Cone Beam Computed Tomography (CBCT)

Abstract Introduction: Cone beam CT (CBCT) is becoming a routine imaging modality designed for the maxillofacial region. Imaging patients with intra-oral metallic objects cause streak artefacts. Artefacts impair any virtual model by obliterating the teeth. This is a major obstacle for occlusal registration and the fabrication of orthognathic wafers to guide the surgical correction of dentofacial deformities. Aims and Objectives: To develop a method of replacing the inaccurate CBCT images of the dentition with an accurate representation and test the feasibility of the technique in the clinical environment. Materials and Method: Impressions of the teeth are acquired and acrylic baseplates constructed on dental casts incorporating radiopaque registration markers. The appliances are fitted and a preoperative CBCT is performed. Impressions are taken of the dentition with the devices in situ and subsequent dental models produced. The models are scanned to produce a virtual model. Both images of the patient and the model are imported into a virtual reality software program and aligned on the virtual markers. This allows the alignment of the dentition without relying on the teeth for superimposition. The occlusal surfaces of the dentition can be replaced with the occlusal image of the model. Results: The absolute mean distance of the mesh between the markers in the skulls was in the region of 0.09mm ± 0.03mm; the replacement dentition had an absolute mean distance of about 0.24mm ± 0.09mm. In patients the absolute mean distance between markers increased to 0.14mm ± 0.03mm. It was not possible to establish the discrepancies in the patient’s dentition, since the original image of the dentition is inherently inaccurate. Conclusion: It is possible to replace the CBCT virtual dentition of cadaveric skulls with an accurate representation to create a composite skull. The feasibility study was successful in the clinical arena. This could be a significant advancement in the accuracy of surgical prediction planning, with the ultimate goal of fabrication of a physical orthognathic wafer using reverse engineering

Real-Time Interactive 3D Reconstruction of Indoor Environments With High Accuracy

3D registration of depth images has been studied extensively in the past. With emergence of low cost RGB-D cameras, many applications have emerged. Yet, the quality of alignment has much to improve. It remains a challenge to create a 3D model of the environment with high accuracy which could be used for engineering applications. Moreover, challenging scenarios where features are scarce, handling large areas as scene, enabling the user to freely roam around the scene for image acquisition in a practical manner, guiding the user in taking better images and accomplishing all of this with low cost RGB-D camera in real time is a subject which is yet to be improved. In this thesis, we address all this challenges by designing and implementing a mobile system that rely on markers. Our system detects and matches markers in real-time with very low CPU load. Moreover, this mobile 3D reconstruction system is interactive which enables a user not only to move freely while doing 3D reconstruction, but also makes the user aware of current status of 3D reconstruction in real-time

Automatic Calibration of a Two-Axis Rotary Table for 3D Scanning Purposes

Rotary tables are often used to speed up the acquisition time during the 3D scanning of complex geometries. In order to avoid manual registration of the point clouds acquired with different orientations, automatic algorithms to compensate the rotation were developed. Alternatively, a proper calibration of the rotary axis with respect to the camera system is needed. Several methods are available in the literature, but they only consider a single-axis calibration. In this paper, a method for the simultaneous calibration of both axes of the table is proposed. A checkerboard is attached to the table, and several images with different poses are acquired. An optimization algorithm is then setup to determine the orientation and the locations of the two axes. A metric to assess the calibration quality was also defined by computing the average mean reprojection error. This metric is used to investigate the optimal number and distribution of the calibration poses, demonstrating that the optimum calibration results are achieved when a wider dispersion of the calibration poses is adopted

Continuous 3D Reconstruction of Plants with Multispectral Information

Phenotyping is the process of identifying desirables traits of plants. These traits do not depend only on the plant genome but also on the environment. Imaging techniques can be applied on this field to help relieving the bottleneck due to manual gathering technique. Climate changes represent a challenge to satisfy the increasing demand of food. Phenotyping can relieve this problem. In this work I developed a robust pipeline to build a 3D multispectral model of as a basis for phenotyping

Learning and Searching Methods for Robust, Real-Time Visual Odometry.

Accurate position estimation provides a critical foundation for mobile robot perception and control. While well-studied, it remains difficult to provide timely, precise, and robust position estimates for applications that operate in uncontrolled environments, such as robotic exploration and autonomous driving. Continuous, high-rate egomotion estimation is possible using cameras and Visual Odometry (VO), which tracks the movement of sparse scene content known as image keypoints or features. However, high update rates, often 30~Hz or greater, leave little computation time per frame, while variability in scene content stresses robustness. Due to these challenges, implementing an accurate and robust visual odometry system remains difficult. This thesis investigates fundamental improvements throughout all stages of a visual odometry system, and has three primary contributions: The first contribution is a machine learning method for feature detector design. This method considers end-to-end motion estimation accuracy during learning. Consequently, accuracy and robustness are improved across multiple challenging datasets in comparison to state of the art alternatives. The second contribution is a proposed feature descriptor, TailoredBRIEF, that builds upon recent advances in the field in fast, low-memory descriptor extraction and matching. TailoredBRIEF is an in-situ descriptor learning method that improves feature matching accuracy by efficiently customizing descriptor structures on a per-feature basis. Further, a common asymmetry in vision system design between reference and query images is described and exploited, enabling approaches that would otherwise exceed runtime constraints. The final contribution is a new algorithm for visual motion estimation: Perspective Alignment Search~(PAS). Many vision systems depend on the unique appearance of features during matching, despite a large quantity of non-unique features in otherwise barren environments. A search-based method, PAS, is proposed to employ features that lack unique appearance through descriptorless matching. This method simplifies visual odometry pipelines, defining one method that subsumes feature matching, outlier rejection, and motion estimation. Throughout this work, evaluations of the proposed methods and systems are carried out on ground-truth datasets, often generated with custom experimental platforms in challenging environments. Particular focus is placed on preserving runtimes compatible with real-time operation, as is necessary for deployment in the field.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113365/1/chardson_1.pd

Augmented reality for computer assisted orthopaedic surgery

In recent years, computer-assistance and robotics have established their presence in operating theatres and found success in orthopaedic procedures. Benefits of computer assisted orthopaedic surgery (CAOS) have been thoroughly explored in research, finding improvements in clinical outcomes, through increased control and precision over surgical actions. However, human-computer interaction in CAOS remains an evolving field, through emerging display technologies including augmented reality (AR) – a fused view of the real environment with virtual, computer-generated holograms. Interactions between clinicians and patient-specific data generated during CAOS are limited to basic 2D interactions on touchscreen monitors, potentially creating clutter and cognitive challenges in surgery. Work described in this thesis sought to explore the benefits of AR in CAOS through: an integration between commercially available AR and CAOS systems, creating a novel AR-centric surgical workflow to support various tasks of computer-assisted knee arthroplasty, and three pre–clinical studies exploring the impact of the new AR workflow on both existing and newly proposed quantitative and qualitative performance metrics. Early research focused on cloning the (2D) user-interface of an existing CAOS system onto a virtual AR screen and investigating any resulting impacts on usability and performance. An infrared-based registration system is also presented, describing a protocol for calibrating commercial AR headsets with optical trackers, calculating a spatial transformation between surgical and holographic coordinate frames. The main contribution of this thesis is a novel AR workflow designed to support computer-assisted patellofemoral arthroplasty. The reported workflow provided 3D in-situ holographic guidance for CAOS tasks including patient registration, pre-operative planning, and assisted-cutting. Pre-clinical experimental validation on a commercial system (NAVIO®, Smith & Nephew) for these contributions demonstrates encouraging early-stage results showing successful deployment of AR to CAOS systems, and promising indications that AR can enhance the clinician’s interactions in the future. The thesis concludes with a summary of achievements, corresponding limitations and future research opportunities.Open Acces

Contribuciones a la estimación de la pose de la cámara en aplicaciones industriales de realidad aumentada

Augmented Reality (AR) aims to complement the visual perception of the user environment superimposing virtual elements. The main challenge of this technology is to combine the virtual and real world in a precise and natural way. To carry out this goal, estimating the user position and orientation in both worlds at all times is a crucial task. Currently, there are numerous techniques and algorithms developed for camera pose estimation. However, the use of synthetic square markers has become the fastest, most robust and simplest solution in these cases. In this scope, a big number of marker detection systems have been developed. Nevertheless, most of them presents some limitations, (1) their unattractive and non-customizable visual appearance prevent their use in industrial products and (2) the detection rate is drastically reduced in presence of noise, blurring and occlusions. In this doctoral dissertation the above-mentioned limitations are addressed. In first place, a comparison has been made between the different marker detection systems currently available in the literature, emphasizing the limitations of each. Secondly, a novel approach to design, detect and track customized markers capable of easily adapting to the visual limitations of commercial products has been developed. In third place, a method that combines the detection of black and white square markers with keypoints and contours has been implemented to estimate the camera position in AR applications. The main motivation of this work is to offer a versatile alternative (based on contours and keypoints) in cases where, due to noise, blurring or occlusions, it is not possible to identify markers in the images. Finally, a method for reconstruction and semantic segmentation of 3D objects using square markers in photogrammetry processes has been presented.La Realidad Aumentada (AR) tiene como objetivo complementar la percepción visual del entorno circunstante al usuario mediante la superposición de elementos virtuales. El principal reto de dicha tecnología se basa en fusionar, de forma precisa y natural, el mundo virtual con el mundo real. Para llevar a cabo dicha tarea, es de vital importancia conocer en todo momento tanto la posición, así como la orientación del usuario en ambos mundos. Actualmente, existen un gran número de técnicas de estimación de pose. No obstante, el uso de marcadores sintéticos cuadrados se ha convertido en la solución más rápida, robusta y sencilla utilizada en estos casos. En este ámbito de estudio, existen un gran número de sistemas de detección de marcadores ampliamente extendidos. Sin embargo, su uso presenta ciertas limitaciones, (1) su aspecto visual, poco atractivo y nada customizable impiden su uso en ciertos productos industriales en donde la personalización comercial es un aspecto crucial y (2) la tasa de detección se ve duramente decrementada ante la presencia de ruido, desenfoques y oclusiones Esta tesis doctoral se ocupa de las limitaciones anteriormente mencionadas. En primer lugar, se ha realizado una comparativa entre los diferentes sistemas de detección de marcadores actualmente en uso, enfatizando las limitaciones de cada uno. En segundo lugar, se ha desarrollado un novedoso enfoque para diseñar, detectar y trackear marcadores personalizados capaces de adaptarse fácilmente a las limitaciones visuales de productos comerciales. En tercer lugar, se ha implementado un método que combina la detección de marcadores cuadrados blancos y negros con keypoints y contornos, para estimar de la posición de la cámara en aplicaciones AR. La principal motivación de este trabajo se basa en ofrecer una alternativa versátil (basada en contornos y keypoints) en aquellos casos donde, por motivos de ruido, desenfoques u oclusiones no sea posible identificar marcadores en las imágenes. Por último, se ha desarrollado un método de reconstrucción y segmentación semántica de objetos 3D utilizando marcadores cuadrados en procesos de fotogrametría