Appearance Modeling of Living Human Tissues

This is the peer reviewed version of the following article: Nunes, A.L.P., Maciel, A., Meyer, G.W., John, N.W., Baranoski, G.V.G., & Walter, M. (2019). Appearance Modeling of Living Human Tissues, Computer Graphics Forum, which has been published in final form at https://doi.org/10.1111/cgf.13604. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-ArchivingThe visual fidelity of realistic renderings in Computer Graphics depends fundamentally upon how we model the appearance of objects resulting from the interaction between light and matter reaching the eye. In this paper, we survey the research addressing appearance modeling of living human tissue. Among the many classes of natural materials already researched in Computer Graphics, living human tissues such as blood and skin have recently seen an increase in attention from graphics research. There is already an incipient but substantial body of literature on this topic, but we also lack a structured review as presented here. We introduce a classification for the approaches using the four types of human tissues as classifiers. We show a growing trend of solutions that use first principles from Physics and Biology as fundamental knowledge upon which the models are built. The organic quality of visual results provided by these Biophysical approaches is mainly determined by the optical properties of biophysical components interacting with light. Beyond just picture making, these models can be used in predictive simulations, with the potential for impact in many other areas

Tracking and Mapping in Medical Computer Vision: A Review

As computer vision algorithms are becoming more capable, their applications in clinical systems will become more pervasive. These applications include diagnostics such as colonoscopy and bronchoscopy, guiding biopsies and minimally invasive interventions and surgery, automating instrument motion and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing and applying algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. We then review datasets provided in the field and the clinical needs therein. Then, we delve in depth into the algorithmic side, and summarize recent developments, which should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. Finally, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications in the field. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.Comment: 31 pages, 17 figure

Scene Reconstruction Beyond Structure-from-Motion and Multi-View Stereo

Image-based 3D reconstruction has become a robust technology for recovering accurate and realistic models of real-world objects and scenes. A common pipeline for 3D reconstruction is to first apply Structure-from-Motion (SfM), which recovers relative poses for the input images and sparse geometry for the scene, and then apply Multi-view Stereo (MVS), which estimates a dense depthmap for each image. While this two-stage process is quite effective in many 3D modeling scenarios, there are limits to what can be reconstructed. This dissertation focuses on three particular scenarios where the SfM+MVS pipeline fails and introduces new approaches to accomplish each reconstruction task. First, I introduce a novel method to recover dense surface reconstructions of endoscopic video. In this setting, SfM can generally provide sparse surface structure, but the lack of surface texture as well as complex, changing illumination often causes MVS to fail. To overcome these difficulties, I introduce a method that utilizes SfM both to guide surface reflectance estimation and to regularize shading-based depth reconstruction. I also introduce models of reflectance and illumination that improve the final result. Second, I introduce an approach for augmenting 3D reconstructions from large-scale Internet photo-collections by recovering the 3D position of transient objects --- specifically, people --- in the input imagery. Since no two images can be assumed to capture the same person in the same location, the typical triangulation constraints enjoyed by SfM and MVS cannot be directly applied. I introduce an alternative method to approximately triangulate people who stood in similar locations, aided by a height distribution prior and visibility constraints provided by SfM. The scale of the scene, gravity direction, and per-person ground-surface normals are also recovered. Finally, I introduce the concept of using crowd-sourced imagery to create living 3D reconstructions --- visualizations of real places that include dynamic representations of transient objects. A key difficulty here is that SfM+MVS pipelines often poorly reconstruct ground surfaces given Internet images. To address this, I introduce a volumetric reconstruction approach that leverages scene scale and person placements. Crowd simulation is then employed to add virtual pedestrians to the space and bring the reconstruction "to life."Doctor of Philosoph

3D Textured Surface Reconstruction from Endoscopic Video

Endoscopy enables high-resolution visualization of tissue texture and is a critical step in many clinical workflows, including diagnosis of infections, tumors or diseases and treatment planning for cancers. This includes my target problems of radiation treatment planning in the nasopharynx and pre-cancerous polyps screening and treatment in colonoscopy. However, an endoscopic video does not provide its information in 3D space, making it difficult to use for tumor localization, and it is inefficient to review. In addition, when there are incomplete camera observations of the organ surface, full surface coverage cannot be guaranteed in an endoscopic procedure, and unsurveyed regions can hardly be noticed in a continuous first-person perspective. This dissertation introduces a new imaging approach that we call endoscopography: an endoscopic video is reconstructed into a full 3D textured surface, which we call an endoscopogram. In this dissertation, I present two endoscopography techniques. One method is a combination of a frame-by-frame algorithmic 3D reconstruction method and a groupwise deformable surface registration method. My contribution is the innovative combination of the two methods that improves the temporal consistency of the frame-by-frame 3D reconstruction algorithm and eliminates the manual intervention that was needed in the deformable surface registration method. The combined method reconstructs an endoscopogram in an offline manner, and the information contained in the tissue texture in the endoscopogram can be transferred to a 3D image such as CT through a surface-to-surface registration. Then, through an interactive tool, the physician can draw directly on the endoscopogram surface to specify a tumor, which then can be automatically transferred to CT slices to aid tumor localization. The second method is a novel deep-learning-driven dense SLAM (simultaneous localization and mapping) system, called RNN-SLAM, that in real time can produce an endoscopogram with display of the unsurveyed regions. In particular, my contribution is the deep learning system in the RNN-SLAM, called RNN-DP. RNN-DP is a novel multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses.Doctor of Philosoph

Development of a face recognition system and its intelligent lighting compensation method for dark-field application

A face recognition system which uses 3D lighting estimation and optimal lighting compensation for dark-field application is proposed. To develop the proposed system, which can realize people identification in a near scene dark-field environment, a light-emitting diode (LED) overhead light, eight LED wall lights, a visible light binocular camera, and a control circuit are used. First, 68 facial landmarks are detected and their coordinates in both image as well as camera coordinate systems are computed. Second, a 3D morphable model (3DMM) is developed after considering facial shadows, and a transformation matrix between the 3DMM and camera coordinate systems is estimated. Third, to assess lighting uniformity, 30 evaluation points are selected from the face. Sequencing computations of LED radiation intensity, ray reflection luminance, camera response, and face lighting uniformity are then carried out. Ray occlusion is processed using a simplified 3D face model. Fourth, an optimal lighting compensation is realized: the overhead light is used for flood lighting, and the wall lights are employed as meticulous lighting. A genetic algorithm then is used to identify the optimal lighting of the wall lights. Finally, an Eigenface method is used for face recognition. The results show that our system and method can improve face recognition accuracy by >10% compared to traditional recognition methods

Image-Based Scene Analysis for Computer-Assisted Laparoscopic Surgery

This thesis is concerned on image-based scene analysis for computer-assisted laparoscopic surgery. The focus lies on how to extract different types of information from laparoscopic video data. Methods for semantic analysis can be used to determine what instruments and organs are currently visible and where they are located. Quantitative analysis provides numerical information on the size and distances of structures. Workflow analysis uses information from previously seen images to estimate the progression of surgery. To demonstrate that the proposed methods function in real-world scenarios, multiple evaluations on actual laparoscopic image data recorded from surgeries were performed. The proposed methods for semantic and quantitative analysis were successfully evaluated in live phantom and animal studies and also used during a live gastric bypass on a human patient

Modelagem de aparência baseada em biofísica para tecidos do fígado humano

A representação gráfica de tecidos humanos é uma importante demanda para aplicações de áreas como ensino, entretenimento e treinamento médico. Frequentemente, a simulação de tais materiais envolve considerar características dinâmicas vinculadas as suas funções no corpo humano e que influenciam diretamente também em sua aparência. O fígado humano, apesar de um órgão interno, portanto, de difícil acesso, possui diferentes modelos de representação apresentados na literatura da Computação Gráfica (CG). Entretanto, tais modelos desconsideram as influências das propriedades ópticas dos elementos biofísicos que compõem os tecidos hepáticos, fornecendo assim, aproximações cuja parametrização controla apenas um estado específico do material orgânico, em geral, avaliando visualmente o resultado. O presente trabalho apresenta a modelagem dos tecidos do fígado humano através da descrição dos elementos biofísicos que compõem suas camadas estruturais: o parênquima e a cápsula de Glisson. Além disso, tal modelo implementa a interação luz-matéria em termos de eventos como a absorção, dispersão, reflexão e transmissão de luz, como processos biológicos que produzem a coloração específica do material, ou seja, sua resposta espectral. A abordagem matemática do modelo é definida como numérica e estocástica, para a qual é apresentada uma solução para garantir sua convergência. Reunindo recentes descrições sobre a estrutura dos tecidos hepáticos e sua interação com a luz apresentadas na literatura biomédica, o modelo desenvolvido representa a primeira solução baseada em biofísica para um órgão interno do corpo humano. Os resultados de imagens geradas através do modelo são apresentados junto a fotografias de tecidos análogos, assim como, curvas de respostas espectrais e espaciais disponíveis na literatura biomédica são comparadas com as produzidas pelo modelo desenvolvido, evidenciando a capacidade deste na representação gráfica do tecido hepático

Konzeption und Entwicklung eines trinokularen Endoskops zur robusten Oberflächenerfassung in der minimalinvasiven Chirurgie

Die minimalinvasive Chirurgie ist eine besonders anspruchsvolle Aufgabe für den Chirurgen, da die Operation ausschließlich über Endoskope und stangenartige, filigrane Instrumente erfolgt. Computerassistierte Stereo-Endoskopiesysteme erleichtern die Tiefenwahrnehmung und unterstützen bei verschiedensten Anwendungen wie z.B. der Resektion eines Nierentumors durch Augmented Reality. Eine wesentliche Aufgabe ist die robuste dreidimensionale Erfassung der beobachteten Oberfläche der Organe. Aufgrund starker Reflexionen durch die endoskopische Lichtquelle, homogener Texturen und weicher, sich bewegender Geometrien ist eine zuverlässige Oberflächenerfassung sehr herausfordernd und stellt noch ein ungelöstes Problem dar. In dieser Arbeit wird deshalb ein neuartiges miniaturisiertes Dreikamerasystem als Demonstrator für ein trinokulares Endoskop sowie ein Algorithmus zur Dreibildauswertung mit semi-globaler Optimierung entwickelt. Durch synthetische und reale Messdaten werden theoretische Überlegungen anhand von drei Hypothesen geprüft. Im Vergleich zu einer stereoskopischen Auswertung wird untersucht, ob eine Dreibildauswertung robustere Ergebnisse liefert, kleinere Referenz- und Suchfenster ermöglicht und eine rechenzeitaufwendige semi-globale Optimierung ersetzt. Es stellt sich heraus, dass die ersten beiden Annahmen grundsätzlich zutreffen, eine semi-globale Optimierung aber nur bedingt ersetzt werden kann. Weiterhin werden die Fehlereinflüsse durch Reflexionen näher spezifiziert und durch gekreuzte Polarisationsfilter sehr effektiv unterdrückt. Das vorgestellte Dreikamera-Endoskop und angepasste Auswerteverfahren tragen wesentlich zur Verbesserung der computerassistierten Endoskopie bei und bringen die Forschungen in diesem Gebiet einen Schritt voran.Minimally invasive surgery is a quite challenging task to the surgeon due to operation through an endoscope and sensitive telescopic instruments exclusively. Computer assisted stereo endoscopic systems eases depth perception and supports several tasks such as dissection of a renal tumour by augmented reality. An essential procedure is robust surface reconstruction of the observed organs. Due to strong reflections from the endoscopic light source, homogeneous textures and weak deforming geometries robust surface reconstruction becomes quite challenging and is not solved successfully yet. Therefore, in this work a novel miniaturised three camera endoscope is introduced and an algorithm for three image analysis and semi-global optimisation is implemented. Synthetic and real experimental measurements are conducted to evaluate theoretical assumptions and review three hypotheses. In contrast to stereo analysis, it is examined whether three image analysis leads to more robust results, allows for smaller matching window sizes and replaces a time-consuming semiglobal matching algorithm. The investigations show that the first two assumptions can generally be confirmed, but the semi-global matching is necessary in some cases. Additionally, errors by reflections are examined in more detail and are suppressed efficiently by crossed polarising filters. The novel three camera endoscope and customized image analysis algorithm gives a great benefit to computer assisted endoscopy and brings research a step closer to more reliable assistant systems