Deep learning architectures for 2D and 3D scene perception

Scene understanding is a fundamental problem in computer vision tasks, that is being more intensively explored in recent years with the development of deep learning. In this dissertation, we proposed deep learning structures to address challenges in 2D and 3D scene perception. We developed several novel architectures for 3D point cloud understanding at city-scale point by effectively capturing both long-range and short-range information to handle the challenging problem of large variations in object size for city-scale point cloud segmentation. GLSNet++ is a two-branch network for multiscale point cloud segmentation that models this complex problem using both global and local processing streams to capture different levels of contextual and structural 3D point cloud information. We developed PointGrad, a new graph convolution gradient operator for capturing structural relationships, that encoded point-based directional gradients into a high-dimensional multiscale tensor space. Using the Point- Grad operator with graph convolution on scattered irregular point sets captures the salient structural information in the point cloud across spatial and feature scale space, enabling efficient learning. We integrated PointGrad with several deep network architectures for large-scale 3D point cloud semantic segmentation, including indoor scene and object part segmentation. In many real application areas including remote sensing and aerial imaging, the class imbalance is common and sufficient data for rare classes is hard to acquire or has high-cost associated with expert labeling. We developed MDXNet for few-shot and zero-shot learning, which emulates the human visual system by leveraging multi-domain knowledge from general visual primitives with transfer learning for more specialized learning tasks in various application domains. We extended deep learning methods in various domains, including the material domain for predicting carbon nanotube forest attributes and mechanical properties, biomedical domain for cell segmentation.Includes bibliographical references

A non-invasive diagnostic system for early assessment of acute renal transplant rejection.

Early diagnosis of acute renal transplant rejection (ARTR) is of immense importance for appropriate therapeutic treatment administration. Although the current diagnostic technique is based on renal biopsy, it is not preferred due to its invasiveness, recovery time (1-2 weeks), and potential for complications, e.g., bleeding and/or infection. In this thesis, a computer-aided diagnostic (CAD) system for early detection of ARTR from 4D (3D + b-value) diffusion-weighted (DW) MRI data is developed. The CAD process starts from a 3D B-spline-based data alignment (to handle local deviations due to breathing and heart beat) and kidney tissue segmentation with an evolving geometric (level-set-based) deformable model. The latter is guided by a voxel-wise stochastic speed function, which follows from a joint kidney-background Markov-Gibbs random field model accounting for an adaptive kidney shape prior and for on-going visual kidney-background appearances. A cumulative empirical distribution of apparent diffusion coefficient (ADC) at different b-values of the segmented DW-MRI is considered a discriminatory transplant status feature. Finally, a classifier based on deep learning of a non-negative constrained stacked auto-encoder is employed to distinguish between rejected and non-rejected renal transplants. In the “leave-one-subject-out” experiments on 53 subjects, 98% of the subjects were correctly classified (namely, 36 out of 37 rejected transplants and 16 out of 16 nonrejected ones). Additionally, a four-fold cross-validation experiment was performed, and an average accuracy of 96% was obtained. These experimental results hold promise of the proposed CAD system as a reliable non-invasive diagnostic tool

Challenges and solution approaches for an improved assessment of reproductive toxicity – species differences and in silico predictions

Within toxicology, reproductive toxicology is a highly relevant and socially particularly sensitive field. It encompasses all toxicological processes within the reproductive cycle and therefore includes many effects and modes of action. This makes the assessment of reproductive toxicity very challenging despite the established in vivo studies. In addition, the in vivo studies are very demanding both in terms of their conduct and interpretation, and there is scope for decision-making on both aspects. As a result, the interpretation of study results may vary from laboratory to laboratory. For the final classification, the assessment of relevance for men is decisive. The problem here is that relatively little is known about the species differences between men and the usual test animals (rat and rabbit). The rabbit in particular has hardly been researched in molecular biology. The aim of the dissertation was to develop approaches for a better assessment of reproductive toxicity, with two different foci: The first aim was to investigate species differences, focusing on the expression of xenobiotic transporters during ontogeny. Xenobiotic transporters, of the superfamily of ATP-binding cassette transporters (ABC) or solute carriers (SLC), are known to transport exogenous substances in addition to their endogenous substrates and therefore play an important role in the absorption, distribution and excretion of xenobiotics. Species differences in kinetics can in turn have a major impact on toxic effects. In the study, the expression of 20 xenobiotic transporters during ontogeny was investigated at the mRNA level in the liver, kidney and placenta of rats and rabbits and compared with that of men. This revealed major differences in the expression of the transporters between the species. However, further studies on the functionality and activity of the xenobiotic transporters are needed to fully assess the kinetic impact of the observed species differences. Overall, the study provides a valid starting point for further systematic investigations of species differences at the protein level. Furthermore, it provides previously unavailable data on the expression of xenobiotic transporters during ontogeny in rabbits, which is an important step in the molecular biological study of this species. The second part focused on investigating the predictive power of in silico models for reproductive toxicology in relation to pesticides. Both the commercial and the freely available models did not perform adequately in the evaluation. Three reasons could be identified for this: 1. many pesticides are outside the chemical space of the models, 2. different definition/assessment of reproductive toxicity and 3. problems in detecting similarity between molecules. To solve these problems, an extension of the databases on reproductive toxicity in relation to pesticides, respecting a uniform nomenclature, is needed. Furthermore, endpoint-specific models should be developed which, in addition to the usual structure-based fingerprints, use descriptors for, for example, biological activity. Overall, the dissertation shows how essential it is to further research the modes of action of reproductive toxicity. This knowledge is necessary to correctly assess in vivo studies and their relevance to men, as well as to improve the predictive power of in silico models by incorporating this information.Innerhalb der Toxikologie ist die Reproduktionstoxikologie ein hochrelevantes und gesellschaftlich besonders sensibles Fachgebiet. Sie umfasst alle toxikologischen Vorgänge innerhalb des Fortpflanzungszyklus und beinhaltet daher eine große Zahl an Effekten und Wirkmechanismen. Dies macht die Bewertung der Reproduktionstoxizität trotz der etablierten in vivo Studien sehr herausfordernd. Dazu kommt, dass die in vivo Studien sowohl bezogen auf ihre Durchführung als auch Interpretation sehr anspruchsvoll sind und es bei beiden Aspekten Entscheidungsspielräume gibt. Dies kann dazu führen, dass die Interpretation von Studienergebnissen von Labor zu Labor variiert. Für die abschließende Einstufung ist die Bewertung der Relevanz für den Menschen entscheidend. Problematisch dabei ist, dass relativ wenig über die Speziesunterschiede zwischen Menschen und den üblichen Versuchstieren (Ratte und Kaninchen) bekannt ist. Gerade das Kaninchen ist molekularbiologisch kaum erforscht. Ziel der Dissertation war es Lösungsansätze zur besseren Bewertung der Reproduktionstoxizität zu entwickeln, wobei zwei unterschiedlichen Schwerpunkte gesetzt wurden: Das erste Ziel war es, die Speziesunterschiede zu untersuchen, wobei der Schwerpunkt auf der Expression von xenobiotischen Transportern während der Ontogenese lag. Xenobiotische Transporter, der Superfamilie der ATP-bindenden Kassettentransporter (ABC) oder Solute Carrier (SLC), sind dafür bekannt, exogene Substanzen zusätzlich zu ihren endogenen Substraten zu transportieren und spielen daher eine wichtige Rolle bei der Absorption, Distribution und Exkretion von Xenobiotika. Speziesunterschiede in der Kinetik können wiederrum einen großen Einfluss auf die toxische Wirkung haben. In der Studie wurde die Expression von 20 xenobiotischen Transportern während der Ontogenese auf mRNA-Level in Leber, Niere und Plazenta von Ratten und Kaninchen untersucht und mit der des Menschen verglichen. Hierbei zeigten sich große Unterschiede in der Expression der Transporter zwischen den Spezies. Um die kinetischen Auswirkungen der beobachteten Artenunterschiede vollständig beurteilen zu können, sind jedoch weitere Studien zur Funktionalität und Aktivität der Fremdstofftransporter erforderlich. Insgesamt bietet die Studie einen validen Ausgangspunkt für weitere systematische Untersuchungen von Artenunterschieden auf Proteinebene. Darüber hinaus liefert sie bisher nicht verfügbare Daten zur Expression von xenobiotischen Transportern während der Ontogenese im Kaninchen, was einen wichtigen Schritt in der molekularbiologischen Untersuchung dieser Spezies darstellt. Im zweiten Teil lag der Schwerpunkt auf der Untersuchung der Vorhersagekraft von in silico Modellen für Reproduktionstoxikologie in Bezug auf Pestizide. Sowohl die kommerziellen als auch die frei verfügbaren Modelle schnitten bei der Bewertung nicht ausreichend ab. Dafür konnten drei Ursachen ausgemacht werden: 1. Viele Pestizide sind außerhalb des chemischen Raums der Modelle, 2. Unterschiedliche Definition/Beurteilung von Reproduktionstoxizität und 3. Probleme bei der Detektion von Ähnlichkeit zwischen Molekülen. Zur Lösung dieser Probleme ist eine Erweiterung der Datenbanken zur Reproduktionstoxizität in Bezug auf Pestizide, unter Beachtung einer einheitlichen Nomenklatur, nötig. Zudem sollten endpunktspezifische Modelle entwickelt werden, welche zusätzlich zu den üblichen strukturbasierten Fingerprints, Deskriptoren für zum Beispiel biologische Aktivität verwenden. Insgesamt zeigt die Dissertation, wie essenziell es ist, die Wirkmechanismen der Reproduktionstoxizität weiter zu erforschen. Dieses Wissen ist notwendig, um in vivo Studien und deren Relevanz für den Menschen korrekt zu beurteilen, sowie die Vorhersagekraft von in silico Modellen durch Einbeziehung dieser Informationen zu verbessern

Vessel identification in diabetic retinopathy

Diabetic retinopathy is the single largest cause of sight loss and blindness in 18 to 65 year olds. Screening programs for the estimated one to six per- cent of the diabetic population have been demonstrated to be cost and sight saving, howeverthere are insufficient screening resources. Automatic screen-ing systems may help solve this resource short fall. This thesis reports on research into an aspect of automatic grading of diabetic retinopathy; namely the identification of the retinal blood vessels in fundus photographs. It de-velops two vessels segmentation strategies and assess their accuracies. A literature review of retinal vascular segmentation found few results, and indicated a need for further development. The two methods for vessel segmentation were investigated in this thesis are based on mathematical morphology and neural networks. Both methodologies are verified on independently labeled data from two institutions and results are presented that characterisethe trade off betweenthe ability to identify vesseland non-vessels data. These results are based on thirty five images with their retinal vessels labeled. Of these images over half had significant pathology and or image acquisition artifacts. The morphological segmentation used ten images from one dataset for development. The remaining images of this dataset and the entire set of 20 images from the seconddataset were then used to prospectively verify generaliastion. For the neural approach, the imageswere pooled and 26 randomly chosenimageswere usedin training whilst 9 were reserved for prospective validation. Assuming equal importance, or cost, for vessel and non-vessel classifications, the following results were obtained; using mathematical morphology 84% correct classification of vascular and non-vascular pixels was obtained in the first dataset. This increased to 89% correct for the second dataset. Using the pooled data the neural approach achieved 88% correct identification accuracy. The spread of accuracies observed varied. It was highest in the small initial dataset with 16 and 10 percent standard deviation in vascular and non-vascular cases respectively. The lowest variability was observed in the neural classification, with a standard deviation of 5% for both accuracies. The less tangible outcomes of the research raises the issueof the selection and subsequent distribution of the patterns for neural network training. Unfortunately this indication would require further labeling of precisely those cases that were felt to be the most difficult. I.e. the small vessels and border conditions between pathology and the retina. The more concrete, evidence based conclusions,characterise both the neural and the morphological methods over a range of operating points. Many of these operating points are comparable to the few results presented in the literature. The advantage of the author's approach lies in the neural method's consistent as well as accurate vascular classification

Modelling of Harbour and Coastal Structures

As the most heavily populated areas in the world, coastal zones host the majority and some of the most important human settlements, infrastructures and economic activities. Harbour and coastal structures are essential to the above, facilitating the transport of people and goods through ports, and protecting low-lying areas against flooding and erosion. While these structures were previously based on relatively rigid concepts about service life, at present, the design—or the upgrading—of these structures should effectively proof them against future pressures, enhancing their resilience and long-term sustainability. This Special Issue brings together a versatile collection of articles on the modelling of harbour and coastal structures, covering a wide array of topics on the design of such structures through a study of their interactions with waves and coastal morphology, as well as their role in coastal protection and harbour design in present and future climates

Automatic Population of Structured Reports from Narrative Pathology Reports

There are a number of advantages for the use of structured pathology reports: they can ensure the accuracy and completeness of pathology reporting; it is easier for the referring doctors to glean pertinent information from them. The goal of this thesis is to extract pertinent information from free-text pathology reports and automatically populate structured reports for cancer diseases and identify the commonalities and differences in processing principles to obtain maximum accuracy. Three pathology corpora were annotated with entities and relationships between the entities in this study, namely the melanoma corpus, the colorectal cancer corpus and the lymphoma corpus. A supervised machine-learning based-approach, utilising conditional random fields learners, was developed to recognise medical entities from the corpora. By feature engineering, the best feature configurations were attained, which boosted the F-scores significantly from 4.2% to 6.8% on the training sets. Without proper negation and uncertainty detection, the quality of the structured reports will be diminished. The negation and uncertainty detection modules were built to handle this problem. The modules obtained overall F-scores ranging from 76.6% to 91.0% on the test sets. A relation extraction system was presented to extract four relations from the lymphoma corpus. The system achieved very good performance on the training set, with 100% F-score obtained by the rule-based module and 97.2% F-score attained by the support vector machines classifier. Rule-based approaches were used to generate the structured outputs and populate them to predefined templates. The rule-based system attained over 97% F-scores on the training sets. A pipeline system was implemented with an assembly of all the components described above. It achieved promising results in the end-to-end evaluations, with 86.5%, 84.2% and 78.9% F-scores on the melanoma, colorectal cancer and lymphoma test sets respectively