3D exemplar-based image inpainting in electron microscopy

In electron microscopy (EM) a common problem is the non-availability of data, which causes artefacts in reconstructions. In this thesis the goal is to generate artificial data where missing in EM by using exemplar-based inpainting (EBI). We implement an accelerated 3D version tailored to applications in EM, which reduces reconstruction times from days to minutes. We develop intelligent sampling strategies to find optimal data as input for reconstruction methods. Further, we investigate approaches to reduce electron dose and acquisition time. Sparse sampling followed by inpainting is the most promising approach. As common evaluation measures may lead to misinterpretation of results in EM and falsify a subsequent analysis, we propose to use application driven metrics and demonstrate this in a segmentation task. A further application of our technique is the artificial generation of projections in tiltbased EM. EBI is used to generate missing projections, such that the full angular range is covered. Subsequent reconstructions are significantly enhanced in terms of resolution, which facilitates further analysis of samples. In conclusion, EBI proves promising when used as an additional data generation step to tackle the non-availability of data in EM, which is evaluated in selected applications. Enhancing adaptive sampling methods and refining EBI, especially considering the mutual influence, promotes higher throughput in EM using less electron dose while not lessening quality.Ein häufig vorkommendes Problem in der Elektronenmikroskopie (EM) ist die Nichtverfügbarkeit von Daten, was zu Artefakten in Rekonstruktionen führt. In dieser Arbeit ist es das Ziel fehlende Daten in der EM künstlich zu erzeugen, was durch Exemplar-basiertes Inpainting (EBI) realisiert wird. Wir implementieren eine auf EM zugeschnittene beschleunigte 3D Version, welche es ermöglicht, Rekonstruktionszeiten von Tagen auf Minuten zu reduzieren. Wir entwickeln intelligente Abtaststrategien, um optimale Datenpunkte für die Rekonstruktion zu erhalten. Ansätze zur Reduzierung von Elektronendosis und Aufnahmezeit werden untersucht. Unterabtastung gefolgt von Inpainting führt zu den besten Resultaten. Evaluationsmaße zur Beurteilung der Rekonstruktionsqualität helfen in der EM oft nicht und können zu falschen Schlüssen führen, weswegen anwendungsbasierte Metriken die bessere Wahl darstellen. Dies demonstrieren wir anhand eines Beispiels. Die künstliche Erzeugung von Projektionen in der neigungsbasierten Elektronentomographie ist eine weitere Anwendung. EBI wird verwendet um fehlende Projektionen zu generieren. Daraus resultierende Rekonstruktionen weisen eine deutlich erhöhte Auflösung auf. EBI ist ein vielversprechender Ansatz, um nicht verfügbare Daten in der EM zu generieren. Dies wird auf Basis verschiedener Anwendungen gezeigt und evaluiert. Adaptive Aufnahmestrategien und EBI können also zu einem höheren Durchsatz in der EM führen, ohne die Bildqualität merklich zu verschlechtern

Enhancing the Reasoning Capabilities of Natural Language Inference Models with Attention Mechanisms and External Knowledge

Natural Language Inference (NLI) is fundamental to natural language understanding. The task summarises the natural language understanding capabilities within a simple formulation of determining whether a natural language hypothesis can be inferred from a given natural language premise. NLI requires an inference system to address the full complexity of linguistic as well as real-world commonsense knowledge and, hence, the inferencing and reasoning capabilities of an NLI system are utilised in other complex language applications such as summarisation and machine comprehension. Consequently, NLI has received significant recent attention from both academia and industry. Despite extensive research, contemporary neural NLI models face challenges arising from the sole reliance on training data to comprehend all the linguistic and real-world commonsense knowledge. Further, different attention mechanisms, crucial to the success of neural NLI models, present the prospects of better utilisation when employed in combination. In addition, the NLI research field lacks a coherent set of guidelines for the application of one of the most crucial regularisation hyper-parameters in the RNN-based NLI models -- dropout. In this thesis, we present neural models capable of leveraging the attention mechanisms and the models that utilise external knowledge to reason about inference. First, a combined attention model to leverage different attention mechanisms is proposed. Experimentation demonstrates that the proposed model is capable of better modelling the semantics of long and complex sentences. Second, to address the limitation of the sole reliance on the training data, two novel neural frameworks utilising real-world commonsense and domain-specific external knowledge are introduced. Employing the rule-based external knowledge retrieval from the knowledge graphs, the first model takes advantage of the convolutional encoders and factorised bilinear pooling to augment the reasoning capabilities of the state-of-the-art NLI models. Utilising the significant advances in the research of contextual word representations, the second model, addresses the existing crucial challenges of external knowledge retrieval, learning the encoding of the retrieved knowledge and the fusion of the learned encodings to the NLI representations, in unique ways. Experimentation demonstrates the efficacy and superiority of the proposed models over previous state-of-the-art approaches. Third, for the limitation on dropout investigations, formulated on exhaustive evaluation, analysis and validation on the proposed RNN-based NLI models, a coherent set of guidelines is introduced

Non-rigid medical image registration with extended free form deformations: modelling general tissue transitions

Image registration seeks pointwise correspondences between the same or analogous objects in different images. Conventional registration methods generally impose continuity and smoothness throughout the image. However, there are cases in which the deformations may involve discontinuities. In general, the discontinuities can be of different types, depending on the physical properties of the tissue transitions involved and boundary conditions. For instance, in the respiratory motion the lungs slide along the thoracic cage following the tangential direction of their interface. In the normal direction, however, the lungs and the thoracic cage are constrained to be always in contact but they have different material properties producing different compression or expansion rates. In the literature, there is no generic method, which handles different types of discontinuities and considers their directional dependence. The aim of this thesis is to develop a general registration framework that is able to correctly model different types of tissue transitions with a general formalism. This has led to the development of the eXtended Free Form Deformation (XFFD) registration method. XFFD borrows the concept of the interpolation method from the eXtended Finite Element method (XFEM) to incorporate discontinuities by enriching B-spline basis functions, coupled with extra degrees of freedom. XFFD can handle different types of discontinuities and encodes their directional-dependence without any additional constraints. XFFD has been evaluated on digital phantoms, publicly available 3D liver and lung CT images. The experiments show that XFFD improves on previous methods and that it is important to employ the correct model that corresponds to the discontinuity type involved at the tissue transition. The effect of using incorrect models is more evident in the strain, which measures mechanical properties of the tissues

The Value of Seizure Semiology in Epilepsy Surgery: Epileptogenic-Zone Localisation in Presurgical Patients using Machine Learning and Semiology Visualisation Tool

Background Eight million individuals have focal drug resistant epilepsy worldwide. If their epileptogenic focus is identified and resected, they may become seizure-free and experience significant improvements in quality of life. However, seizure-freedom occurs in less than half of surgical resections. Seizure semiology - the signs and symptoms during a seizure - along with brain imaging and electroencephalography (EEG) are amongst the mainstays of seizure localisation. Although there have been advances in algorithmic identification of abnormalities on EEG and imaging, semiological analysis has remained more subjective. The primary objective of this research was to investigate the localising value of clinician-identified semiology, and secondarily to improve personalised prognostication for epilepsy surgery. Methods I data mined retrospective hospital records to link semiology to outcomes. I trained machine learning models to predict temporal lobe epilepsy (TLE) and determine the value of semiology compared to a benchmark of hippocampal sclerosis (HS). Due to the hospital dataset being relatively small, we also collected data from a systematic review of the literature to curate an open-access Semio2Brain database. We built the Semiology-to-Brain Visualisation Tool (SVT) on this database and retrospectively validated SVT in two separate groups of randomly selected patients and individuals with frontal lobe epilepsy. Separately, a systematic review of multimodal prognostic features of epilepsy surgery was undertaken. The concept of a semiological connectome was devised and compared to structural connectivity to investigate probabilistic propagation and semiology generation. Results Although a (non-chronological) list of patients’ semiologies did not improve localisation beyond the initial semiology, the list of semiology added value when combined with an imaging feature. The absolute added value of semiology in a support vector classifier in diagnosing TLE, compared to HS, was 25%. Semiology was however unable to predict postsurgical outcomes. To help future prognostic models, a list of essential multimodal prognostic features for epilepsy surgery were extracted from meta-analyses and a structural causal model proposed. Semio2Brain consists of over 13000 semiological datapoints from 4643 patients across 309 studies and uniquely enabled a Bayesian approach to localisation to mitigate TLE publication bias. SVT performed well in a retrospective validation, matching the best expert clinician’s localisation scores and exceeding them for lateralisation, and showed modest value in localisation in individuals with frontal lobe epilepsy (FLE). There was a significant correlation between the number of connecting fibres between brain regions and the seizure semiologies that can arise from these regions. Conclusions Semiology is valuable in localisation, but multimodal concordance is more valuable and highly prognostic. SVT could be suitable for use in multimodal models to predict the seizure focus

Migration, Mobility and Human Rights at the Eastern Border of the European Union - Space of Freedom and Security

This edited collection of migration papers would like to emphasise the acute need for migration related study and research in Romania. At this time, migration and mobility are studied as minor subjects in Economics, Sociology, Political Sciences and European Studies only (mostly at post-graduate level). We consider that Romanian universities need more ‘migration studies’, while research should cover migration as a whole, migration and mobility being analysed from different points of view – social, economical, legal etc. Romania is part of the European Migration Space not only as a source of labourers for the European labour market, but also as source of quality research for the European scientific arena. Even a country located at the eastern border of the European Union, we consider Romania as part of the European area of freedom, security and justice, and therefore interested in solving correctly all challenges incurred by the complex phenomena of migration and workers’ mobility at the European level. The waves of illegal immigrants arriving continuously on the Spanish, Italian and Maltese shores, and the workers’ flows from the new Member States from Central and Eastern Europe following the 2004 accession, forced the EU officials and the whole Europe to open the debate on the economical and mostly social consequences of labour mobility. This study volume is our contribution to this important scientific debate. Starting with the spring of 2005, the Jean Monnet European Centre of Excellence and the School of High Comparative European Studies (SISEC), both within the West University of Timisoara, have proposed a series of events in order to raise the awareness of the Romanian scientific environment on this very sensitive issues: migration and mobility in the widen European Space. An annual international event to celebrate 9 May - The Europe Day was already a tradition for SISEC (an academic formula launched back in 1995 in order to prepare national experts in European affairs, offering academic post-graduate degrees in High European Studies). With the financial support from the Jean Monnet Programme (DG Education and Culture, European Commission), a first migration panel was organised in the framework of the international colloquium ‘Romania and the European Union in 2007’ held in Timisoara between 6 and 7 of May 2005 (panel Migration, Asylum and Human Rights at the Eastern Border of the European Union). Having in mind the positive welcoming of the migration related subjects during the 2005 colloquium, a second event was organised on 5 May 2006 in the framework of the European Year of Workers’ Mobility: the international colloquium Migration and Mobility: Assets and Challenges for the Enlargement of the European Union. In the same period, the Jean Monnet European Centre of Excellence, SISEC and The British Council in Bucharest have jointly edited two special issues of The Romanian Journal of European Studies, no.4/2005 and 5-6/2006, both dedicated to migration and mobility. Preliminary versions of many of the chapters of this volume were presented at the above mentioned international events. The papers were chosen according to their scientific quality, after an anonymously peer-review selection. The authors debate both theoretical issues and practical results of their research. They are renowned experts at international level, members of the academia, PhD students or experienced practitioners involved in the management of the migration flows at the governmental level. This volume was financed by the Jean Monnet Programme of the Directorate General Education and Culture, European Commission, throughout the Jean Monnet European Centre of Excellence (C03/0110) within the West University of Timisoara, Romania, and is dedicated to the European Year of Workers’ Mobility 2006. Timisoara, December 200

Using compartment models of diffusion MRI to investigate the preterm brain

Preterm birth is the leading cause of neonatal mortality, with survivors experiencing motor, cognitive and other deficits at increased rates. In preterm infancy, the developing brain undergoes folding, myelination, and rapid cellular growth. Diffusion-Weighted Magnetic Resonance Imaging (DW MRI) is an imaging modality that allows noninvasive inference of cellular microstructure in living tissue, and its parameters reflect changes in brain tissue composition. In this thesis, we employ compartment models of DW MRI to investigate the microstructure in preterm-born subjects at different ages. Within infants, we have used the NODDI model to investigate longitudinal changes in neurite density and orientation dispersion within the white matter, cerebral cortex and thalamus, explaining known trends in diffusion tensor parameters with greater specificity. We then used a quantitative T2 sequence to develop and investigate a novel, multi-modal parameter known as the ‘g-ratio’. We have also investigated changing microstructural geometry within the cortex. Immediately after preterm birth, the highly-ordered underlying cellular structure makes diffusion in the cortex almost entirely radial. This undergoes a transition to a disordered and isotropic state over the first weeks of life, which we have used the DIAMOND model to quantify. This radiality decreases at a rate that depends on the cortical lobe. In a cohort of young adults born extremely preterm, we have quantified differences in brain microstructure compared to term-born controls. In preterm subjects, the brain structures are smaller than for controls, leading to increased partial volume in some regions of interest. We introduce a method to infer diffusion parameters in partial volume, even for regions which are smaller than the diffusion resolution. Overall, this thesis utilises and evaluates a variety of compartment models of DW MRI. By developing and applying principled and robust methodology, we present new insights into microstructure within the preterm-born brain

Foundations, Inference, and Deconvolution in Image Restoration

Image restoration is a critical preprocessing step in computer vision, producing images with reduced noise, blur, and pixel defects. This enables precise higher-level reasoning as to the scene content in later stages of the vision pipeline (e.g., object segmentation, detection, recognition, and tracking). Restoration techniques have found extensive usage in a broad range of applications from industry, medicine, astronomy, biology, and photography. The recovery of high-grade results requires models of the image degradation process, giving rise to a class of often heavily underconstrained, inverse problems. A further challenge specific to the problem of blur removal is noise amplification, which may cause strong distortion by ringing artifacts. This dissertation presents new insights and problem solving procedures for three areas of image restoration, namely (1) model foundations, (2) Bayesian inference for high-order Markov random fields (MRFs), and (3) blind image deblurring (deconvolution). As basic research on model foundations, we contribute to reconciling the perceived differences between probabilistic MRFs on the one hand, and deterministic variational models on the other. To do so, we restrict the variational functional to locally supported finite elements (FE) and integrate over the domain. This yields a sum of terms depending locally on FE basis coefficients, and by identifying the latter with pixels, the terms resolve to MRF potential functions. In contrast with previous literature, we place special emphasis on robust regularizers used commonly in contemporary computer vision. Moreover, we draw samples from the derived models to further demonstrate the probabilistic connection. Another focal issue is a class of high-order Field of Experts MRFs which are learned generatively from natural image data and yield best quantitative results under Bayesian estimation. This involves minimizing an integral expression, which has no closed form solution in general. However, the MRF class under study has Gaussian mixture potentials, permitting expansion by indicator variables as a technical measure. As approximate inference method, we study Gibbs sampling in the context of non-blind deblurring and obtain excellent results, yet at the cost of high computing effort. In reaction to this, we turn to the mean field algorithm, and show that it scales quadratically in the clique size for a standard restoration setting with linear degradation model. An empirical study of mean field over several restoration scenarios confirms advantageous properties with regard to both image quality and computational runtime. This dissertation further examines the problem of blind deconvolution, beginning with localized blur from fast moving objects in the scene, or from camera defocus. Forgoing dedicated hardware or user labels, we rely only on the image as input and introduce a latent variable model to explain the non-uniform blur. The inference procedure estimates freely varying kernels and we demonstrate its generality by extensive experiments. We further present a discriminative method for blind removal of camera shake. In particular, we interleave discriminative non-blind deconvolution steps with kernel estimation and leverage the error cancellation effects of the Regression Tree Field model to attain a deblurring process with tightly linked sequential stages