Accurate geometry reconstruction of vascular structures using implicit splines

3-D visualization of blood vessel from standard medical datasets (e.g. CT or MRI) play an important role in many clinical situations, including the diagnosis of vessel stenosis, virtual angioscopy, vascular surgery planning and computer aided vascular surgery. However, unlike other human organs, the vasculature system is a very complex network of vessel, which makes it a very challenging task to perform its 3-D visualization. Conventional techniques of medical volume data visualization are in general not well-suited for the above-mentioned tasks. This problem can be solved by reconstructing vascular geometry. Although various methods have been proposed for reconstructing vascular structures, most of these approaches are model-based, and are usually too ideal to correctly represent the actual variation presented by the cross-sections of a vascular structure. In addition, the underlying shape is usually expressed as polygonal meshes or in parametric forms, which is very inconvenient for implementing ramification of branching. As a result, the reconstructed geometries are not suitable for computer aided diagnosis and computer guided minimally invasive vascular surgery. In this research, we develop a set of techniques associated with the geometry reconstruction of vasculatures, including segmentation, modelling, reconstruction, exploration and rendering of vascular structures. The reconstructed geometry can not only help to greatly enhance the visual quality of 3-D vascular structures, but also provide an actual geometric representation of vasculatures, which can provide various benefits. The key findings of this research are as follows: 1. A localized hybrid level-set method of segmentation has been developed to extract the vascular structures from 3-D medical datasets. 2. A skeleton-based implicit modelling technique has been proposed and applied to the reconstruction of vasculatures, which can achieve an accurate geometric reconstruction of the vascular structures as implicit surfaces in an analytical form. 3. An accelerating technique using modern GPU (Graphics Processing Unit) is devised and applied to rendering the implicitly represented vasculatures. 4. The implicitly modelled vasculature is investigated for the application of virtual angioscopy

Neural Radiance Fields: Past, Present, and Future

The various aspects like modeling and interpreting 3D environments and surroundings have enticed humans to progress their research in 3D Computer Vision, Computer Graphics, and Machine Learning. An attempt made by Mildenhall et al in their paper about NeRFs (Neural Radiance Fields) led to a boom in Computer Graphics, Robotics, Computer Vision, and the possible scope of High-Resolution Low Storage Augmented Reality and Virtual Reality-based 3D models have gained traction from res with more than 1000 preprints related to NeRFs published. This paper serves as a bridge for people starting to study these fields by building on the basics of Mathematics, Geometry, Computer Vision, and Computer Graphics to the difficulties encountered in Implicit Representations at the intersection of all these disciplines. This survey provides the history of rendering, Implicit Learning, and NeRFs, the progression of research on NeRFs, and the potential applications and implications of NeRFs in today's world. In doing so, this survey categorizes all the NeRF-related research in terms of the datasets used, objective functions, applications solved, and evaluation criteria for these applications.Comment: 413 pages, 9 figures, 277 citation

Advances in Simultaneous Localization and Mapping in Confined Underwater Environments Using Sonar and Optical Imaging.

This thesis reports on the incorporation of surface information into a probabilistic simultaneous localization and mapping (SLAM) framework used on an autonomous underwater vehicle (AUV) designed for underwater inspection. AUVs operating in cluttered underwater environments, such as ship hulls or dams, are commonly equipped with Doppler-based sensors, which---in addition to navigation---provide a sparse representation of the environment in the form of a three-dimensional (3D) point cloud. The goal of this thesis is to develop perceptual algorithms that take full advantage of these sparse observations for correcting navigational drift and building a model of the environment. In particular, we focus on three objectives. First, we introduce a novel representation of this 3D point cloud as collections of planar features arranged in a factor graph. This factor graph representation probabalistically infers the spatial arrangement of each planar segment and can effectively model smooth surfaces (such as a ship hull). Second, we show how this technique can produce 3D models that serve as input to our pipeline that produces the first-ever 3D photomosaics using a two-dimensional (2D) imaging sonar. Finally, we propose a model-assisted bundle adjustment (BA) framework that allows for robust registration between surfaces observed from a Doppler sensor and visual features detected from optical images. Throughout this thesis, we show methods that produce 3D photomosaics using a combination of triangular meshes (derived from our SLAM framework or given a-priori), optical images, and sonar images. Overall, the contributions of this thesis greatly increase the accuracy, reliability, and utility of in-water ship hull inspection with AUVs despite the challenges they face in underwater environments. We provide results using the Hovering Autonomous Underwater Vehicle (HAUV) for autonomous ship hull inspection, which serves as the primary testbed for the algorithms presented in this thesis. The sensor payload of the HAUV consists primarily of: a Doppler velocity log (DVL) for underwater navigation and ranging, monocular and stereo cameras, and---for some applications---an imaging sonar.PhDElectrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120750/1/paulozog_1.pd

Variational surface reconstruction

The demand for capturing 3D models of real world objects or scenes has steadily increased in the past. Today, there are numerous developments that indicate an even greater importance in the future: Computer generated special effects are extensively used and highly benefit from such data, 3D printing is starting to become more affordable, and the ability to conveniently include 3D content in websites has quite matured. Thus, 3D reconstruction has been and still is one of the most important research topics in the area of computer vision. Here, the reconstruction of a 3D model from a number of colour images with given camera poses is one of the most common tasks known as multi-view stereo. We contribute to the two main stages that arise in popular strategies for solving this problem: The estimation of depth maps from multiple views and the integration of multiple depth maps into a single watertight surface. Subsequently, we relax the constraint that the camera poses have to be known and present a novel pipeline for 3D reconstruction from image sequences that solely relies on dense ideas. It proves to be an interesting alternative to popular sparse approaches and leads to competitive results. When relying on sparse features, this only allows to estimate an oriented point cloud instead of a surface. To this end, we finally propose a general higher order framework for the surface reconstruction from oriented points.In den letzten Jahrzehnten ist die Nachfrage nach digitalen 3D Modellen von Objekten und Szenen ständig gestiegen und vieles spricht dafür, dass sich dies auch in Zukunft fortsetzt: Computergenerierte Spezialeffekte werden immer flächendeckender eingesetzt, der Druck von dreidimensionalen Gegenständen macht große Fortschritte, und die Darstellung dreidimensionaler Modelle im Webbrowser wird immer ausgereifter. Deshalb ist die 3D Rekonstruktion eines der wichtigsten Forschungsthemen im Bereich des maschinellen Sehens. Die Rekonstruktion von einem 3D Modell aus mehreren Bildern mit gegebenen Kameramatritzen ist hier eine der häufigsten Problemstellungen, bekannt als multi-view stereo. Wir leisten einen Beitrag zu den zwei wichtigen Schritten, die in multi-view stereo Ansätzen angewandt werden: Die Schätzung von Tiefenkarten aus mehreren Bildern und die Fusion von mehreren Tiefenkarten zu einem einzigen 3D Modell. Anschließend lockern wir die Voraussetzung, dass die Kameramatritzen bekannt sein müssen und präsentieren ein neues Verfahren zur 3D Rekonstruktion aus Bildsequenzen, das vollständig auf dichten Ansätzen beruht. Dies erweist sich als interessante Alternative zu populären Methoden, die mit einzelnen Merkmalen arbeiten. Verfahren, die auf einzelnen Merkmalen beruhen, erlauben die Schätzung von orientierten Punktwolken. Daher entwickeln wir zum Schluss ein allgemeines Rahmenwerk für die Berechnung von wasserdichten Oberflächen aus orientierten Punktwolken

Mra görüntülerinden damar ayrıştırılması ve damar yüzeyinin oluşturulması

Tez (Yüksek Lisans) -- İstanbul Teknik Üniversitesi, Bilişim Ensititüsü, 2012Thesis (M.Sc.) -- İstanbul Technical University, Institute of Informatics, 2012Kalp-damar hastalıklarının özellikle batılı ülkelerde ölüm sebeplerinin başında gelmesi, damar yapısının 3 boyutlu olarak oluşturulmasını çok önemli kılmaktadır. Hastaya özel oluşturulan damar yüzeyi, görselleştirilerek teşhis veya cerrahi planlama amaçlı kullanılabilir. Bunun yanında, oluşturulan yüzey temel alınarak yapılacak olan hesaplamalı sıvı dinamiği(HSD) simülasyonları ile hastalık oluşma riski yüksek olan bölgeler tespit edilebilir ve böylece gelecekte bazı hastalıkların herhangi bir belirti göstermeden önce engellenmesi mümkün olabilir. Hastaya özel HSD simülasyonlarının çok önemli iki uygulaması ateroskleroz ve serebral anevrizma yırtılması riskinin belirlenmesidir. Aterosklerotik plakların, kan akışının düzgün olmadığı ve damar duvarı üzerindeki kayma gerilmesinin düşük olduğu bölgelerde oluşma riskinin daha fazla olduğu bilinmektedir. Anevrizma yırtılması riski için de benzer hemodinamik etkiler rol oynamaktadır. Kan akışının hastanın içinde ölçülmesi güvenilir veya rahat bir işlem olmadığından, akışın hastaya özel damar yapısı üzerinde HSD simülasyonu yapılarak ölçülmesi gerekmektedir. Damar yapısının 3 boyutlu olarak oluşturulması problemi iki ana adımdan oluşur. İlk önce, hacim verisindeki bütün vokseller damara ait veya değil olarak etiketlenir. Bu adıma damar ayrıştırılması adı verilir. Daha sonra, bu ayrıştırılmış veri kullanılarak damar yapısı poligonal meş şeklinde 3 boyutlu olarak oluşturulur. Eğer meş sadece görselleştirme amaçlı değil, ayrıca simülasyonlar için de kullanılacaksa meşin yüksek kalitede olması gerekir. Yani, meşi oluşturan çokgenlerin açı ve büyüklükleri nümerik simülasyona uygun olmalıdır. Ne yazık ki, 3 boyutlu verilerden damar ayrıştırılmasının manuel olarak yapılması uzun ve zahmetli bir işlemdir. Ayrıca, ayrıştırılmış veriden yüksek kaliteli meş oluşturmak da kolay değildir. Bu nedenle, hem otomatik ve yarı-otomatik damar ayrıştırılması, hem de ayrıştırılmış veriden poligonal meş şeklinde yüzey oluşturulması sorunları bilimsel yazında çokça incelenmiştir ve birçok farklı yaklaşım bulunmaktadır. Bu iki adım için ayrı ayrı yapılan çalışmaların sayısına kıyasla, iki adımı birleştirmek ile ilgili fazla çalışma bulunmamaktadır ve mevcut çalışmalar genelde tek bir anatomik bölgeye odaklıdır. Biz bu çalışmada MRA verisinden damar ayrıştırılması ve damar yüzeyinin oluşturulmasına odaklanarak, bütün anatomik bölgelerde çalışan, kalın, dar, sağlıklı ve hastalıklı her türlü damarı ayrıştırabilen ve mümkün olduğunca az kullanıcı müdahalesine gerek duyan birleşik bir model tasarlamayı hedefliyoruz. MRA verisini seçmemizin nedeni MRA'nın, CTA ve DSA gibi anjiyografi tekniklerine kıyasla hasta için daha az risk taşıması ve görüntüdeki en parlak yapıların damar olduğu varsayımının genelde geçerli olmasıdır. Bu varsayım, damar ayrıştırılması adımını kolaylaştıracaktır. Damar ayrıştırılması adımında yerel geometri bilgisini kullanarak evrimleşen bir level-set yaklaşımı kullanılmıştır. Daha açık olarak söylersek, damar yüzeyi, 3 boyutlu bir hiperyüzeyin sıfır level-seti olarak ifade edilmiş ve yerel multi-scale Hessian ve ortalama eğrilik bilgisinden türetilen bir evrim fonksiyonun rehberliği ile evrimleştirilmiştir. Hessian bilgisi ve damarların görüntüdeki en parlak yapı olduğu varsayımı kullanılarak, parlak ve boruya benzeyen bir yapı içerisinde bulunan noktalar tespit edilebilir. Ortalama eğrilik de eğriliği fazla olan damarlarda ayrıştırmanın devam etmesi için itici bir rol oynar çünkü bu gibi kıvrımlı damarlarda boruya benzerlik düşük olduğundan Hessian bilgisi yetersiz kalabilmektedir. Evrim fonksiyonu, Hessian bilgisinden türetilen bir ölçü ile ortalama eğriliği, kullanıcı tarafından belirleyen katsayılar yardımıyla dengeler. Evrim süreci, bir kısmi türevli diferansiyel denklem için başlangıç değer probleminin çözümü olarak modellenmiştir. Başlangıç yüzeyi, damar içerisinde olduğu bilinen noktaların etrafında küreler oluşturulmak suretiyle seçilebilir. Noktaların otomatik seçimi için de evrim fonksiyonunda kullandığımız Hessian ölçüsü kullanılabilir. Bu ölçünün en yüksek olduğu noktaların bir damar içerisinde olma olasılığı çok yüksektir. Tabii ki, bazı görüntülerdeki sorunlar, bu otomatik seçilimi etkileyebilmektedir ve böyle durumlarda kullanıcı müdahalesi gerekmektedir. Başlangıç yüzeyi belirlendikten sonra yüzey, evrim fonksiyonu rehberliği altında, yakınsama sağlanıncaya kadar evrimleşir. Damar ayrıştırılma adımının sonuçları, gerçek MRA verileri üzerinde görsel olarak ve sentetik olarak oluşturulmuş ve gürültü eklenmiş veriler üzerinde sayısal olarak değerlendirilmiştir. Sonuçlara bakıldığında Hessian bilgisinin, ortalama eğriliğe baskınlığı artırıldığında ayrıştırılan bölgenin damar olma olasılığının daha yüksek olduğu ancak özellikle yüksek eğrilikli damarların ayrıştırılmadığı görülmüştür. Tersine olarak, ortalama eğriliğin baskınlığı artırıldığında daha fazla damar ayrıştırılmakta ancak aşırı ayrıştırma durumunun ortaya çıkma şansı da artmaktadır. Ortalama eğrilik, noktanın damar içinde olup olmadığı ile ilgili bir bilgi taşımadığından bu beklenen bir durumdur. Diğer bir önemli gözlem de kullanıcı tarafından belirlenen katsayıların seçiminin ayrıştırılmaya etkisinin, gerçek görüntülerde sentetik görüntülere kıyasla çok daha yüksek olduğudur. Bunun iki nedeni vardır: Birincisi, sentetik görüntülerdeki tek bozulma Gaussian gürültü iken gerçek görüntülerde birçok farklı sorun olabilmesidir. İkinci neden, sentetik görüntülerde eğriliği yüksek olan damar olmamasıdır. Daha önce de bahsettiğimiz gibi, damarın eğriliği yüksek olduğunda Hessian ölçüsü yüksek değerler vermemekte ve ayrıştırmanın devam etmesi için eğrilikten destek gelmesi gerekmektedir. Böylece, önerdiğimiz metodun en önemli sorunu, kullanıcı tarafından belirlenen katsayıların doğru seçilimidir. Üzerinde çalıştığımız veriler için iyi sonuçlar veren katsayılar önermiş olsak da bu katsayıların genelleştirilebilir olduğunu söyleyemeyiz. Tıbbi görüntülerde birçok farklı artifact olması ve damarların çok farklı geometrik şekillerde bulunabilmesi, her görüntü için kullanıcı müdahalesi olmadan iyi sonuç verecek bir metod bulmayı çok zor hale getirmektedir. Çalışmamızın ikinci kısmı, birinci kısmın sonucunu, yani damar yüzeyini kapalı olarak ifade eden 3 boyutlu bir hacmin ayrık örneklemesini, kullanarak yüzeyi çokgenler yardımıyla ifade etmektir. Kapalı ifade edilmiş yüzeyleri çokgenleştirmek için kullanılan standart yöntem üçgenler kullanarak bu işlemi yapan Marching Cubes'dur. Bu yöntem çok hızlı ve isabetli olmakla beraber düşük kaliteli üçgenlerden (çok dar açılı veya çok küçük) oluşan oldukça çıkıntılı yüzeyler oluşturmaktadır. Bu nedenle, sonuçta elde ettiğimiz yüzeyi nümerik simülasyonlarda kullanmak istiyorsak, Marching Cubes uygun bir yöntem değildir. Dolayısıyla, yüksek kaliteli üçgenlerden oluşan daha pürüzsüz yüzeyler oluşturduğu bilinen bir advancing front metodunu tercih ettik. Advancing front metodlarının temel dezavantajı yavaş olmalarıdır ancak nümerik simülasyonlar için kullanılacak yüzeylerin oluşturulma aşamasının gerçek zamanlı olması gerekmediğinden bu soruna tahammül edilebilir. Yöntemde, yüzeyin pürüzsüzlüğü ile verilen veriye bağlılığını dengeleyen, kullanıcı tarafından tanımlanan katsayılar bulunmaktadır. Bu katsayılar, oluşturulan yüzeyin nasıl bir uygulamada kullanılacağına ve girdi olarak alınan verinin ne kadar gürültülü olduğu gözönünde bulundurularak seçilmelidir. Ayrıca, yüzey girdi verisine ne kadar bağlı olursa yöntemin çalışma süresinin de o kadar uzayacağı hesaba katılmalıdır. Son olarak, yüzeyin daha da pürüzsüzleştirilmesi için Taubin pürüzsüzleştirmesi kullanılmıştır. Bu yöntem, Laplacian veya Gaussian gibi klasik pürüzsüzleştirme yöntemlerinin aksine, doğru kullanıldığında, yüzeyde daralma oluşturmamaktadır. Sonuçlar, Marching Cubes yöntemiyle elde edilen sonuçlarla görsel olarak karşılaştırılmış ve önerilen yöntemin Marching Cubes'a kıyasla çok daha yüksek kalitede yüzeyler oluşturduğu ve kıyaslanabilir isabette olduğu görülmüştür. Çalışmamızın bütünü gözönüne alındığında üzerinde durulması gereken en önemli nokta, her adımda istenilen bazı özelliklerin elde edilmesi için istenen başka özelliklerden vazgeçilmesi gerektiğidir. Damar ayrıştırılması kısmında isabetlilik ile kullanıcı müdahalesine gereksinim, hesaplama pahası (computation cost) ve genellik, damar yüzeyi oluşturulması kısmında ise isabetlilik ile pürüzsüzlük ve hesaplama pahası dengelenmelidir. Bu dengeleme işlemini yaparken, elde edilecek olan yüzeyin kullanılacağı özel tıbbi uygulamanın gereksinimleri dikkate alınmalıdır. Birkaç örnek vermek gerekirse, tehşis için yeterli olan bir isabet oranı, beyin ameliyatı planlaması için yeterli olmayabilir. Damar yüzeyinin pürüzsüz olması nümerik simülasyonlar için gerekli olmakla beraber tehşis veya ameliyat planlaması için istenmeyen bir durum olabilir. Sonuç yüzeyindeki çokgen sayısı ameliyat planlaması veya tehşis için bir sorun oluşturmayacak ancak nümerik simülasyonun hızını ciddi şekilde etkileyecektir. İdeal olarak gelecekte ulaşılmak istenilen nokta, verilen herhangi bir tıbbi süreç için herhangi bir tıbbi görüntüyü hiçbir kullanıcı müdahalesi veya düzeltmesine gerek olmaksızın anlamlı şekilde işleyecek bir programa sahip olunmasıdır. Ancak şu an bu noktadan uzak olunduğundan, bir algoritmanın her durumda işe yaramasını beklememek gerekir. Dolayısıyla, tıbbi uygulamalar üzerinde çalışan bilgisayar bilimcilerin, hekimlerle yakın bir işbirliği içerisinde çalışmaları ve böylece geliştirdikleri yöntemin kullanılacağı tıbbi uygulamanın gereksinimlerinden haberdar olmaları büyük önem arz etmektedir.3D surface reconstruction of vascular structures plays a very important role in the medical field as vascular problems can be lethal and are in fact among the leading causes of death. The reconstructed patient specific vessel structure can be visualized and used for diagnostic and surgical planing purposes. Furthermore, analysing the vascular structure and detecting risk areas using CFD simulations can help in making therapeutic decisions and might, in the future, make it possible to prevent some diseases before they show any symptoms. Two very important applications of patient specific CFD simulations are the evaluation of the risk of atherosclerosis and cerebral aneurysm rupture. Atherosclerotic plagues are known to occur more frequently in regions where the flow is not laminar and wall sheer stress is low. Hemodynamic factors are also thought to play a role in the risk of aneurysm rupture. As measuring blood flow in vivo is not reliable or convenient, the only way to assess the risk is by numerical simulation on patient specific 3D vessel structure. The vessel surface reconstruction problem consists of two main steps. First, all the voxels in the volume data are labeled as vessel or non-vessel. This step is called vessel segmentation. Then, using the segmentation, the vessel surface is reconstructed as a polygonal mesh. If the mesh is to be used for numerical simulations instead of only visualization, then the mesh should be high quality which means that it should contain polygons of similar sizes and angles. Regrettably, manual segmentation of 3D data is a lengthy and cumbersome process and reconstruction of a high quality surface from the segmented data is not trivial. Therefore, both the problem of automatic or semi-automatic vessel segmentation from medical images and the problem of reconstructing a high quality surface from segmented volume data have been extensively studied in literature and there are many different types of approaches. Compared to the amount of research that study vessel segmentation and surface reconstruction seperately, the number of studies that aim to combine these methods for the purpose of patient specific numerical simulations is relatively low and they generally consider only a specific anatomical region. In this work, we focus on vessel segmentation and reconstruction from MRA data and we aim to design a unified model that works on all anatomical regions, can detect both large and small vessels and uses minimal user interaction. For the segmentation step, a level-set evolution scheme based on local geometric information is used. More specifically, the vessel surface is represented as the zero level-set of a 3D hypersurface and the hypersurface is evolved under guidance of an evolution function derived from multi-scale Hessian analysis and mean curvature. The evolution is modeled as an initial value problem resulting from a PDE. The surface can be initialized automatically using the Hessian analysis to detect bright tubular structures. The results are analysed visually on real MRA data and quantitatively on a synthetically created dataset deformed with various noise intensities. It is seen from the results that the proposed method is promising. The second step of our work is to construct a polygonal representation of the vessel surface using the result of the first part which is an implicit representation of the vessel surface as a discrete sampling of a 3D volume. The standart classical method for polygonizing implicit surfaces is Marching Cubes which constructs a surface with triangles. Although Marching Cubes is fast and guaranteed to be accurate, it generates a very rough surface and can contain low quality triangles(highly acute-angled triangles). Thus, it is not suitable if we want to run numerical simulations using the resulting surface. Therefore, we use an advancing front method which is known to generate smoother surfaces with better triangle quality. Advancing front methods have the downside of being slow but this is not such a big issue when the surface reconstruction is done offline. Finally, Taubin smoothing is used to further smooth the surface. Compared to simpler smoothing methods like Laplacian and Gaussian smoothing which produce shrinkage, Taubin smoothing does not change the topology of the vessel structure if used correctly. The results obtained from using an advancing front method and Taubin smoothing are compared to the classical Marching Cubes results visually to show that much higher triangle quality is achieved while maintaining acceptable accuracy.Yüksek LisansM.Sc

Vascular Tree Structure: Fast Curvature Regularization and Validation

This work addresses the challenging problem of accurate vessel structure analysis in high resolution 3D biomedical images. Typical segmentation methods fail on recent micro-CT data sets resolving near-capillary vessels due to limitations of standard first-order regularization models. While regularization is needed to address noise and partial volume issues in the data, we argue that extraction of thin tubular structures requires higher-order curvature-based regularization. There are no standard segmentation methods regularizing surface curvature in 3D that could be applied to large 3D volumes. However, we observe that standard measures for vessels structure are more concerned with topology, bifurcation angles, and other parameters that can be directly addressed without segmentation. We propose a novel methodology reconstructing tree structure of the vessels using a new centerline curvature regularization technique. Our high-order regularization model is based on a recent curvature estimation method. We developed a Levenberg-Marquardt optimization scheme and an efficient GPU-based implementation of our algorithm. We also propose a validation mechanism based on synthetic vessel images. Our preliminary results on real ultra-resolution micro CT volumes are promising

Differentiable world programs

L'intelligence artificielle (IA) moderne a ouvert de nouvelles perspectives prometteuses pour la création de robots intelligents. En particulier, les architectures d'apprentissage basées sur le gradient (réseaux neuronaux profonds) ont considérablement amélioré la compréhension des scènes 3D en termes de perception, de raisonnement et d'action. Cependant, ces progrès ont affaibli l'attrait de nombreuses techniques ``classiques'' développées au cours des dernières décennies. Nous postulons qu'un mélange de méthodes ``classiques'' et ``apprises'' est la voie la plus prometteuse pour développer des modèles du monde flexibles, interprétables et exploitables : une nécessité pour les agents intelligents incorporés. La question centrale de cette thèse est : ``Quelle est la manière idéale de combiner les techniques classiques avec des architectures d'apprentissage basées sur le gradient pour une compréhension riche du monde 3D ?''. Cette vision ouvre la voie à une multitude d'applications qui ont un impact fondamental sur la façon dont les agents physiques perçoivent et interagissent avec leur environnement. Cette thèse, appelée ``programmes différentiables pour modèler l'environnement'', unifie les efforts de plusieurs domaines étroitement liés mais actuellement disjoints, notamment la robotique, la vision par ordinateur, l'infographie et l'IA. Ma première contribution---gradSLAM--- est un système de localisation et de cartographie simultanées (SLAM) dense et entièrement différentiable. En permettant le calcul du gradient à travers des composants autrement non différentiables tels que l'optimisation non linéaire par moindres carrés, le raycasting, l'odométrie visuelle et la cartographie dense, gradSLAM ouvre de nouvelles voies pour intégrer la reconstruction 3D classique et l'apprentissage profond. Ma deuxième contribution - taskography - propose une sparsification conditionnée par la tâche de grandes scènes 3D encodées sous forme de graphes de scènes 3D. Cela permet aux planificateurs classiques d'égaler (et de surpasser) les planificateurs de pointe basés sur l'apprentissage en concentrant le calcul sur les attributs de la scène pertinents pour la tâche. Ma troisième et dernière contribution---gradSim--- est un simulateur entièrement différentiable qui combine des moteurs physiques et graphiques différentiables pour permettre l'estimation des paramètres physiques et le contrôle visuomoteur, uniquement à partir de vidéos ou d'une image fixe.Modern artificial intelligence (AI) has created exciting new opportunities for building intelligent robots. In particular, gradient-based learning architectures (deep neural networks) have tremendously improved 3D scene understanding in terms of perception, reasoning, and action. However, these advancements have undermined many ``classical'' techniques developed over the last few decades. We postulate that a blend of ``classical'' and ``learned'' methods is the most promising path to developing flexible, interpretable, and actionable models of the world: a necessity for intelligent embodied agents. ``What is the ideal way to combine classical techniques with gradient-based learning architectures for a rich understanding of the 3D world?'' is the central question in this dissertation. This understanding enables a multitude of applications that fundamentally impact how embodied agents perceive and interact with their environment. This dissertation, dubbed ``differentiable world programs'', unifies efforts from multiple closely-related but currently-disjoint fields including robotics, computer vision, computer graphics, and AI. Our first contribution---gradSLAM---is a fully differentiable dense simultaneous localization and mapping (SLAM) system. By enabling gradient computation through otherwise non-differentiable components such as nonlinear least squares optimization, ray casting, visual odometry, and dense mapping, gradSLAM opens up new avenues for integrating classical 3D reconstruction and deep learning. Our second contribution---taskography---proposes a task-conditioned sparsification of large 3D scenes encoded as 3D scene graphs. This enables classical planners to match (and surpass) state-of-the-art learning-based planners by focusing computation on task-relevant scene attributes. Our third and final contribution---gradSim---is a fully differentiable simulator that composes differentiable physics and graphics engines to enable physical parameter estimation and visuomotor control, solely from videos or a still image

Structure-preserving machine learning for inverse problems

Inverse problems naturally arise in many scientific settings, and the study of these problems has been crucial in the development of important technologies such as medical imaging. In inverse problems, the goal is to estimate an underlying ground truth u∗, typically an image, from corresponding measurements y, where u∗ and y are related by y = N(A(u∗)) (1) for some forward operator A and noise-generating process N (both of which are generally assumed to be known). Variational regularisation is a well-established approach that can be used to approximately solve inverse problems such as Problem (1). In this approach an image is reconstructed from measurements y by solving a minimisation problem such as uˆ = argmin d(A(u),y) +αJ(u). (2) While this approach has proven very successful, it generally requires the parts that make up the optimisation problem to be carefully chosen, and the optimisation problem may require considerable computational effort to solve. There is an active line of research into overcoming these issues using data-driven approaches, which aim to use multiple instances of data to inform a method that can be used on similar data. In this dissertation we investigate ways in which favourable properties of the variational regularisation approach can be combined with a data-driven approach to solving inverse problems. In the first chapter of the dissertation, we propose a bilevel optimisation framework that can be used to optimise sampling patterns and regularisation parameters for variational image reconstruction in accelerated magnetic resonance imaging. We use this framework to learn sampling patterns that result in better image reconstructions than standard random variable density sampling patterns that sample with the same rate. In the second chapter of the dissertation, we study the use of group symmetries in learned reconstruction methods for inverse problems. We show that group invariance of a functional implies that the corresponding proximal operator satisfies a group equivariance property. Applying this idea to model proximal operators as roto-translationally equivariant in an unrolled iterative reconstruction method, we show that reconstruction performance is more robust when tested on images in orientations not seen during training (compared to similar methods that model proximal operators to just be translationally equivariant) and that good methods can be learned with less training data. In the final chapter of the dissertation, we propose a ResNet-styled neural network architecture that is provably nonexpansive. This architecture can be thought of as composing discretisations of gradient flows along learnable convex potentials. Appealing to a classical result on the numerical integration of ODEs, we show that constraining the operator norms of the weight operators is sufficient to give nonexpansiveness, and additional analysis in the case that the numerical integrator is the forward Euler method shows that the neural network is an averaged operator. This guarantees that its fixed point iterations are convergent, and makes it a natural candidate for a learned denoiser in a Plug-and-Play approach to solving inverse problemsCantab Capital Institute for the Mathematics of Informatio

Deep Structured Layers for Instance-Level Optimization in 2D and 3D Vision

The approach we present in this thesis is that of integrating optimization problems as layers in deep neural networks. Optimization-based modeling provides an additional set of tools enabling the design of powerful neural networks for a wide battery of computer vision tasks. This thesis shows formulations and experiments for vision tasks ranging from image reconstruction to 3D reconstruction. We first propose an unrolled optimization method with implicit regularization properties for reconstructing images from noisy camera readings. The method resembles an unrolled majorization minimization framework with convolutional neural networks acting as regularizers. We report state-of-the-art performance in image reconstruction on both noisy and noise-free evaluation setups across many datasets. We further focus on the task of monocular 3D reconstruction of articulated objects using video self-supervision. The proposed method uses a structured layer for accurate object deformation that controls a 3D surface by displacing a small number of learnable handles. While relying on a small set of training data per category for self-supervision, the method obtains state-of-the-art reconstruction accuracy with diverse shapes and viewpoints for multiple articulated objects. We finally address the shortcomings of the previous method that revolve around regressing the camera pose using multiple hypotheses. We propose a method that recovers a 3D shape from a 2D image by relying solely on 3D-2D correspondences regressed from a convolutional neural network. These correspondences are used in conjunction with an optimization problem to estimate per sample the camera pose and deformation. We quantitatively show the effectiveness of the proposed method on self-supervised 3D reconstruction on multiple categories without the need for multiple hypotheses

Computer-aided Visualization of Colonoscopy

Colonoscopy is the most widely used medical technique to examine the human large intestine (colon) and eliminate precancerous or malignant lesions, i.e., polyps. It uses a high-definition camera to examine the inner surface of the colon. Very often, a portion of the colon surface is not visualized during the procedure. Unsurveyed portions of the colon can harbor polyps that then progress to colorectal cancer. Unfortunately, it is hard for the endoscopist to realize there is unsurveyed surface from the video as it is formed. A system to alert endoscopists to missed surface area could thus more fully protect patients from colorectal cancer following colonoscopy. In this dissertation computer-aided visualization techniques were developed in order to solve this problem:1. A novel Simultaneous Localization and Mapping (SLAM) algorithm called RNNSLAM was proposed to address the difficulties of applying a traditional SLAM system on colonic images. I improved a standard SLAM system with a previously proposed Recurrent Neural Network for Depth and Pose Estimation (RNN-DP). The combination of SLAM’s optimization mechanism and RNN-DP’s prior knowledge achieved state-of-the-art performance on colonoscopy, especially addressing the drift problem in both SLAM and RNN-DP. A fusion module was added to this system to generate a dense 3D surface.2. I conducted exploration research on recognizing colonic places that have been visited based on video frames. This technique called image relocalization or retrieval is needed for helping the endoscopist to fully survey the previously unsurveyed regions. A benchmark testing dataset was created for colon image retrieval. Deep neural networks were successfully trained using Structure from Motion results on colonoscopy and achieved promising results.3. To visualize highly-curved portions of a colon or the whole colon, a generalized cylinder deformation algorithm was proposed to semi-flatten the geometry of the colon model for more succinct and global visualization.Doctor of Philosoph