Generalizable automated pixel-level structural segmentation of medical and biological data

Over the years, the rapid expansion in imaging techniques and equipments has driven the demand for more automation in handling large medical and biological data sets. A wealth of approaches have been suggested as optimal solutions for their respective imaging types. These solutions span various image resolutions, modalities and contrast (staining) mechanisms. Few approaches generalise well across multiple image types, contrasts or resolution. This thesis proposes an automated pixel-level framework that addresses 2D, 2D+t and 3D structural segmentation in a more generalizable manner, yet has enough adaptability to address a number of specific image modalities, spanning retinal funduscopy, sequential fluorescein angiography and two-photon microscopy. The pixel-level segmentation scheme involves: i ) constructing a phase-invariant orientation field of the local spatial neighbourhood; ii ) combining local feature maps with intensity-based measures in a structural patch context; iii ) using a complex supervised learning process to interpret the combination of all the elements in the patch in order to reach a classification decision. This has the advantage of transferability from retinal blood vessels in 2D to neural structures in 3D. To process the temporal components in non-standard 2D+t retinal angiography sequences, we first introduce a co-registration procedure: at the pairwise level, we combine projective RANSAC with a quadratic homography transformation to map the coordinate systems between any two frames. At the joint level, we construct a hierarchical approach in order for each individual frame to be registered to the global reference intra- and inter- sequence(s). We then take a non-training approach that searches in both the spatial neighbourhood of each pixel and the filter output across varying scales to locate and link microvascular centrelines to (sub-) pixel accuracy. In essence, this \link while extract" piece-wise segmentation approach combines the local phase-invariant orientation field information with additional local phase estimates to obtain a soft classification of the centreline (sub-) pixel locations. Unlike retinal segmentation problems where vasculature is the main focus, 3D neural segmentation requires additional exibility, allowing a variety of structures of anatomical importance yet with different geometric properties to be differentiated both from the background and against other structures. Notably, cellular structures, such as Purkinje cells, neural dendrites and interneurons, all display certain elongation along their medial axes, yet each class has a characteristic shape captured by an orientation field that distinguishes it from other structures. To take this into consideration, we introduce a 5D orientation mapping to capture these orientation properties. This mapping is incorporated into the local feature map description prior to a learning machine. Extensive performance evaluations and validation of each of the techniques presented in this thesis is carried out. For retinal fundus images, we compute Receiver Operating Characteristic (ROC) curves on existing public databases (DRIVE & STARE) to assess and compare our algorithms with other benchmark methods. For 2D+t retinal angiography sequences, we compute the error metrics ("Centreline Error") of our scheme with other benchmark methods. For microscopic cortical data stacks, we present segmentation results on both surrogate data with known ground-truth and experimental rat cerebellar cortex two-photon microscopic tissue stacks.Open Acces

Digital Twin of Cardiovascular Systems

Patient specific modelling using numerical methods is widely used in understanding diseases and disorders. It produces medical analysis based on the current state of patient’s health. Concurrently, as a parallel development, emerging data driven Artificial Intelligence (AI) has accelerated patient care. It provides medical analysis using algorithms that rely upon knowledge from larger human population data. AI systems are also known to have the capacity to provide a prognosis with overallaccuracy levels that are better than those provided by trained professionals. When these two independent and robust methods are combined, the concept of human digital twins arise. A Digital Twin is a digital replica of any given system or process. They combine knowledge from general data with subject oriented knowledge for past, current and future analyses and predictions. Assumptions made during numerical modelling are compensated using knowledge from general data. For humans, they can provide an accurate current diagnosis as well as possible future outcomes. This allows forprecautions to be taken so as to avoid further degradation of patient’s health.In this thesis, we explore primary forms of human digital twins for the cardiovascular system, that are capable of replicating various aspects of the cardiovascular system using different types of data. Since different types of medical data are available, such as images, videos and waveforms, and the kinds of analysis required may be offline or online in nature, digital twin systems should be uniquely designed to capture each type of data for different kinds of analysis. Therefore, passive, active and semi-active digital twins, as the three primary forms of digital twins, for different kinds of applications are proposed in this thesis. By the virtue of applications and the kind of data involved ineach of these applications, the performance and importance of human digital twins for the cardiovascular system are demonstrated. The idea behind these twins is to allow for the application of the digital twin concept for online analysis, offline analysis or a combination of the two in healthcare. In active digital twins active data, such as signals, is analysed online in real-time; in semi-active digital twin some of the components being analysed are active but the analysis itself is carried out offline; and finally, passive digital twins perform offline analysis of data that involves no active component.For passive digital twin, an automatic workflow to calculate Fractional Flow Reserve (FFR) is proposed and tested on a cohort of 25 patients with acceptable results. For semi-active digital twin, detection of carotid stenosis and its severity using face videos is proposed and tested with satisfactory results from one carotid stenosis patient and a small cohort of healthy adults. Finally, for the active digital twin, an enabling model is proposed using inverse analysis and its application in the detection of Abdominal Aortic Aneurysm (AAA) and its severity, with the help of a virtual patient database. This enabling model detected artificially generated AAA with an accuracy as high as 99.91% and classified its severity with acceptable accuracy of 97.79%. Further, for active digital twin, a truly active model is proposed for continuous cardiovascular state monitoring. It is tested on a small cohort of five patients from a publicly available database for three 10-minute periods, wherein this model satisfactorily replicated and forecasted patients’ cardiovascular state. In addition to the three forms of human digital twins for the cardiovascular system, an additional work on patient prioritisation in pneumonia patients for ITU care using data-driven digital twin is also proposed. The severity indices calculated by these models are assessed using the standard benchmark of Area Under Receiving Operating Characteristic Curve (AUROC). The results indicate that using these models, the ITU and mechanical ventilation can be prioritised correctly to an AUROC value as high as 0.89

Character Recognition

Character recognition is one of the pattern recognition technologies that are most widely used in practical applications. This book presents recent advances that are relevant to character recognition, from technical topics such as image processing, feature extraction or classification, to new applications including human-computer interfaces. The goal of this book is to provide a reference source for academic research and for professionals working in the character recognition field

Label Efficient 3D Scene Understanding

3D scene understanding models are becoming increasingly integrated into modern society. With applications ranging from autonomous driving, Augmented Real- ity, Virtual Reality, robotics and mapping, the demand for well-behaved models is rapidly increasing. A key requirement for training modern 3D models is high- quality manually labelled training data. Collecting training data is often the time and monetary bottleneck, limiting the size of datasets. As modern data-driven neu- ral networks require very large datasets to achieve good generalisation, finding al- ternative strategies to manual labelling is sought after for many industries. In this thesis, we present a comprehensive study on achieving 3D scene under- standing with fewer labels. Specifically, we evaluate 4 approaches: existing data, synthetic data, weakly-supervised and self-supervised. Existing data looks at the potential of using readily available national mapping data as coarse labels for train- ing a building segmentation model. We further introduce an energy-based active contour snake algorithm to improve label quality by utilising co-registered LiDAR data. This is attractive as whilst the models may still require manual labels, these labels already exist. Synthetic data also exploits already existing data which was not originally designed for training neural networks. We demonstrate a pipeline for generating a synthetic Mobile Laser Scanner dataset. We experimentally evalu- ate if such a synthetic dataset can be used to pre-train smaller real-world datasets, increasing the generalisation with less data. A weakly-supervised approach is presented which allows for competitive per- formance on challenging real-world benchmark 3D scene understanding datasets with up to 95% less data. We propose a novel learning approach where the loss function is learnt. Our key insight is that the loss function is a local function and therefore can be trained with less data on a simpler task. Once trained our loss function can be used to train a 3D object detector using only unlabelled scenes. Our method is both flexible and very scalable, even performing well across datasets. Finally, we propose a method which only requires a single geometric represen- tation of each object class as supervision for 3D monocular object detection. We discuss why typical L2-like losses do not work for 3D object detection when us- ing differentiable renderer-based optimisation. We show that the undesirable local- minimas that the L2-like losses fall into can be avoided with the inclusion of a Generative Adversarial Network-like loss. We achieve state-of-the-art performance on the challenging 6DoF LineMOD dataset, without any scene level labels