Modeling and Analysis of Subcellular Protein Localization in Hyper-Dimensional Fluorescent Microscopy Images Using Deep Learning Methods

Hyper-dimensional images are informative and become increasingly common in biomedical research. However, the machine learning methods of studying and processing the hyper-dimensional images are underdeveloped. Most of the methods only model the mapping functions between input and output by focusing on the spatial relationship, whereas neglect the temporal and causal relationships. In many cases, the spatial, temporal, and causal relationships are correlated and become a relationship complex. Therefore, only modeling the spatial relationship may result in inaccurate mapping function modeling and lead to undesired output. Despite the importance, there are multiple challenges on modeling the relationship complex, including the model complexity and the data availability. The objective of this dissertation is to comprehensively study the mapping function modeling of the spatial-temporal and the spatial-temporal-causal relationship in hyper-dimensional data with deep learning approaches. The modeling methods are expected to accurately capture the complex relationships in class-level and object-level so that new image processing tools can be developed based on the methods to study the relationships between targets in hyper-dimensional data. In this dissertation, four different cases of relationship complex are studied, including the class-level spatial-temporal-causal relationship and spatial-temporal relationship modeling, and the object-level spatial-temporal-causal relationship and spatial-temporal relationship modeling. The modelings are achieved by deep learning networks that implicitly model the mapping functions with network weight matrix. For spatial-temporal relationship, because the cause factor information is unavailable, discriminative modeling that only relies on available information is studied. For class-level and object-level spatial-temporal-causal relationship, generative modeling is studied with a new deep learning network and three new tools proposed. For spatial-temporal relationship modeling, a state-of-the-art segmentation network has been found to be the best performer over 18 networks. Based on accurate segmentation, we study the object-level temporal dynamics and interactions through dynamics tracking. The multi-object portion tracking (MOPT) method allows object tracking in subcellular level and identifies object events, including object born, dead, split, and fusion. The tracking results is 2.96% higher on consistent tracking accuracy and 35.48% higher on event identification accuracy, compared with the existing state-of-the-art tracking methods. For spatial-temporal-causal relationship modeling, the proposed four-dimensional reslicing generative adversarial network (4DR-GAN) captures the complex relationships between the input and the target proteins. The experimental results on four groups of proteins demonstrate the efficacy of 4DR-GAN compared with the widely used Pix2Pix network. On protein localization prediction (PLP), the predicted localization from 4DR-GAN is more accurate in subcellular localization, temporal consistency, and dynamics. Based on efficient PLP, the digital activation (DA) and digital inactivation (DI) tools allow precise spatial and temporal control on global and local localization manipulation. They allow researchers to study the protein functions and causal relationships by observing the digital manipulation and PLP output response

Medical image registration using unsupervised deep neural network: A scoping literature review

In medicine, image registration is vital in image-guided interventions and other clinical applications. However, it is a difficult subject to be addressed which by the advent of machine learning, there have been considerable progress in algorithmic performance has recently been achieved for medical image registration in this area. The implementation of deep neural networks provides an opportunity for some medical applications such as conducting image registration in less time with high accuracy, playing a key role in countering tumors during the operation. The current study presents a comprehensive scoping review on the state-of-the-art literature of medical image registration studies based on unsupervised deep neural networks is conducted, encompassing all the related studies published in this field to this date. Here, we have tried to summarize the latest developments and applications of unsupervised deep learning-based registration methods in the medical field. Fundamental and main concepts, techniques, statistical analysis from different viewpoints, novelties, and future directions are elaborately discussed and conveyed in the current comprehensive scoping review. Besides, this review hopes to help those active readers, who are riveted by this field, achieve deep insight into this exciting field

Anatomical Modeling of Cerebral Microvascular Structures: Application to Identify Biomarkers of Microstrokes

Les réseaux microvasculaires corticaux sont responsables du transport de l’oxygène et des substrats énergétiques vers les neurones. Ces réseaux réagissent dynamiquement aux demandes énergétiques lors d’une activation neuronale par le biais du couplage neurovasculaire. Afin d’élucider le rôle de la composante microvasculaire dans ce processus de couplage, l’utilisation de la modélisation in-formatique pourrait se révéler un élément clé. Cependant, la manque de méthodologies de calcul appropriées et entièrement automatisées pour modéliser et caractériser les réseaux microvasculaires reste l’un des principaux obstacles. Le développement d’une solution entièrement automatisée est donc important pour des explorations plus avancées, notamment pour quantifier l’impact des mal-formations vasculaires associées à de nombreuses maladies cérébrovasculaires. Une observation courante dans l’ensemble des troubles neurovasculaires est la formation de micro-blocages vascu-laires cérébraux (mAVC) dans les artérioles pénétrantes de la surface piale. De récents travaux ont démontré l’impact de ces événements microscopiques sur la fonction cérébrale. Par conséquent, il est d’une importance vitale de développer une approche non invasive et comparative pour identifier leur présence dans un cadre clinique. Dans cette thèse,un pipeline de traitement entièrement automatisé est proposé pour aborder le prob-lème de la modélisation anatomique microvasculaire. La méthode de modélisation consiste en un réseau de neurones entièrement convolutif pour segmenter les capillaires sanguins, un générateur de modèle de surface 3D et un algorithme de contraction de la géométrie pour produire des mod-èles graphiques vasculaires ne comportant pas de connections multiples. Une amélioration de ce pipeline est développée plus tard pour alléger l’exigence de maillage lors de la phase de représen-tation graphique. Un nouveau schéma permettant de générer un modèle de graphe est développé avec des exigences d’entrée assouplies et permettant de retenir les informations sur les rayons des vaisseaux. Il est inspiré de graphes géométriques déformants construits en respectant les morpholo-gies vasculaires au lieu de maillages de surface. Un mécanisme pour supprimer la structure initiale du graphe à chaque exécution est implémenté avec un critère de convergence pour arrêter le pro-cessus. Une phase de raffinement est introduite pour obtenir des modèles vasculaires finaux. La modélisation informatique développée est ensuite appliquée pour simuler les signatures IRM po-tentielles de mAVC, combinant le marquage de spin artériel (ASL) et l’imagerie multidirectionnelle pondérée en diffusion (DWI). L’hypothèse est basée sur des observations récentes démontrant une réorientation radiale de la microvascularisation dans la périphérie du mAVC lors de la récupéra-tion chez la souris. Des lits capillaires synthétiques, orientés aléatoirement et radialement, et des angiogrammes de tomographie par cohérence optique (OCT), acquis dans le cortex de souris (n = 5) avant et après l’induction d’une photothrombose ciblée, sont analysés. Les graphes vasculaires informatiques sont exploités dans un simulateur 3D Monte-Carlo pour caractériser la réponse par résonance magnétique (MR), tout en considérant les effets des perturbations du champ magnétique causées par la désoxyhémoglobine, et l’advection et la diffusion des spins nucléaires. Le pipeline graphique proposé est validé sur des angiographies synthétiques et réelles acquises avec différentes modalités d’imagerie. Comparé à d’autres méthodes effectuées dans le milieu de la recherche, les expériences indiquent que le schéma proposé produit des taux d’erreur géométriques et topologiques amoindris sur divers angiogrammes. L’évaluation confirme également l’efficacité de la méthode proposée en fournissant des modèles représentatifs qui capturent tous les aspects anatomiques des structures vasculaires. Ensuite, afin de trouver des signatures de mAVC basées sur le signal IRM, la modélisation vasculaire proposée est exploitée pour quantifier le rapport de perte de signal intravoxel minimal lors de l’application de plusieurs directions de gradient, à des paramètres de séquence variables avec et sans ASL. Avec l’ASL, les résultats démontrent une dif-férence significative (p <0,05) entre le signal calculé avant et 3 semaines après la photothrombose. La puissance statistique a encore augmenté (p <0,005) en utilisant des angiogrammes capturés à la semaine suivante. Sans ASL, aucun changement de signal significatif n’est trouvé. Des rapports plus élevés sont obtenus à des intensités de champ magnétique plus faibles (par exemple, B0 = 3) et une lecture TE plus courte (<16 ms). Cette étude suggère que les mAVC pourraient être carac-térisés par des séquences ASL-DWI, et fournirait les informations nécessaires pour les validations expérimentales postérieures et les futurs essais comparatifs.----------ABSTRACT Cortical microvascular networks are responsible for carrying the necessary oxygen and energy substrates to our neurons. These networks react to the dynamic energy demands during neuronal activation through the process of neurovascular coupling. A key element in elucidating the role of the microvascular component in the brain is through computational modeling. However, the lack of fully-automated computational frameworks to model and characterize these microvascular net-works remains one of the main obstacles. Developing a fully-automated solution is thus substantial for further explorations, especially to quantify the impact of cerebrovascular malformations associ-ated with many cerebrovascular diseases. A common pathogenic outcome in a set of neurovascular disorders is the formation of microstrokes, i.e., micro occlusions in penetrating arterioles descend-ing from the pial surface. Recent experiments have demonstrated the impact of these microscopic events on brain function. Hence, it is of vital importance to develop a non-invasive and translatable approach to identify their presence in a clinical setting. In this thesis, a fully automatic processing pipeline to address the problem of microvascular anatom-ical modeling is proposed. The modeling scheme consists of a fully-convolutional neural network to segment microvessels, a 3D surface model generator and a geometry contraction algorithm to produce vascular graphical models with a single connected component. An improvement on this pipeline is developed later to alleviate the requirement of water-tight surface meshes as inputs to the graphing phase. The novel graphing scheme works with relaxed input requirements and intrin-sically captures vessel radii information, based on deforming geometric graphs constructed within vascular boundaries instead of surface meshes. A mechanism to decimate the initial graph struc-ture at each run is formulated with a convergence criterion to stop the process. A refinement phase is introduced to obtain final vascular models. The developed computational modeling is then ap-plied to simulate potential MRI signatures of microstrokes, combining arterial spin labeling (ASL) and multi-directional diffusion-weighted imaging (DWI). The hypothesis is driven based on recent observations demonstrating a radial reorientation of microvasculature around the micro-infarction locus during recovery in mice. Synthetic capillary beds, randomly- and radially oriented, and op-tical coherence tomography (OCT) angiograms, acquired in the barrel cortex of mice (n=5) before and after inducing targeted photothrombosis, are analyzed. The computational vascular graphs are exploited within a 3D Monte-Carlo simulator to characterize the magnetic resonance (MR) re-sponse, encompassing the effects of magnetic field perturbations caused by deoxyhemoglobin, and the advection and diffusion of the nuclear spins. The proposed graphing pipeline is validated on both synthetic and real angiograms acquired with different imaging modalities. Compared to other efficient and state-of-the-art graphing schemes, the experiments indicate that the proposed scheme produces the lowest geometric and topological error rates on various angiograms. The evaluation also confirms the efficiency of the proposed scheme in providing representative models that capture all anatomical aspects of vascular struc-tures. Next, searching for MRI-based signatures of microstokes, the proposed vascular modeling is exploited to quantify the minimal intravoxel signal loss ratio when applying multiple gradient di-rections, at varying sequence parameters with and without ASL. With ASL, the results demonstrate a significant difference (p<0.05) between the signal-ratios computed at baseline and 3 weeks after photothrombosis. The statistical power further increased (p<0.005) using angiograms captured at week 4. Without ASL, no reliable signal change is found. Higher ratios with improved significance are achieved at low magnetic field strengths (e.g., at 3 Tesla) and shorter readout TE (<16 ms). This study suggests that microstrokes might be characterized through ASL-DWI sequences, and provides necessary insights for posterior experimental validations, and ultimately, future transla-tional trials

Introduction: Ways of Machine Seeing

How do machines, and, in particular, computational technologies, change the way we see the world? This special issue brings together researchers from a wide range of disciplines to explore the entanglement of machines and their ways of seeing from new critical perspectives. This 'editorial' is for a special issue of AI & Society, which includes contributions from: María Jesús Schultz Abarca, Peter Bell, Tobias Blanke, Benjamin Bratton, Claudio Celis Bueno, Kate Crawford, Iain Emsley, Abelardo Gil-Fournier, Daniel Chávez Heras, Vladan Joler, Nicolas Malevé, Lev Manovich, Nicholas Mirzoeff, Perle Møhl, Bruno Moreschi, Fabian Offert, Trevor Paglan, Jussi Parikka, Luciana Parisi, Matteo Pasquinelli, Gabriel Pereira, Carloalberto Treccani, Rebecca Uliasz, and Manuel van der Veen

On Improving Generalization of CNN-Based Image Classification with Delineation Maps Using the CORF Push-Pull Inhibition Operator

Deployed image classification pipelines are typically dependent on the images captured in real-world environments. This means that images might be affected by different sources of perturbations (e.g. sensor noise in low-light environments). The main challenge arises by the fact that image quality directly impacts the reliability and consistency of classification tasks. This challenge has, hence, attracted wide interest within the computer vision communities. We propose a transformation step that attempts to enhance the generalization ability of CNN models in the presence of unseen noise in the test set. Concretely, the delineation maps of given images are determined using the CORF push-pull inhibition operator. Such an operation transforms an input image into a space that is more robust to noise before being processed by a CNN. We evaluated our approach on the Fashion MNIST data set with an AlexNet model. It turned out that the proposed CORF-augmented pipeline achieved comparable results on noise-free images to those of a conventional AlexNet classification model without CORF delineation maps, but it consistently achieved significantly superior performance on test images perturbed with different levels of Gaussian and uniform noise

Deep Learning in Medical Image Analysis

The accelerating power of deep learning in diagnosing diseases will empower physicians and speed up decision making in clinical environments. Applications of modern medical instruments and digitalization of medical care have generated enormous amounts of medical images in recent years. In this big data arena, new deep learning methods and computational models for efficient data processing, analysis, and modeling of the generated data are crucially important for clinical applications and understanding the underlying biological process. This book presents and highlights novel algorithms, architectures, techniques, and applications of deep learning for medical image analysis

Scene and crowd analysis using synthetic data generation with 3D quality improvements and deep network architectures

In this thesis, a scene analysis mainly focusing on vision-based techniques have been explored. The vision-based scene analysis techniques have a wide range of applications from surveillance, security to agriculture. A vision sensor can provide rich information about the environment such as colour, depth, shape, size and much more. This information can be further processed to have an in-depth knowledge of the scene such as type of environment, objects and distances. Hence, this thesis covers initially the background on human detection in particular pedestrian and crowd detection methods and introduces various vision-based techniques used in human detection. Followed by a detailed analysis of the use of synthetic data to improve the performance of state-of-the-art Deep Learning techniques and a multi-purpose synthetic data generation tool is proposed. The tool is a real-time graphics simulator which generates multiple types of synthetic data applicable for pedestrian detection, crowd density estimation, image segmentation, depth estimation, and 3D pose estimation. In the second part of the thesis, a novel technique has been proposed to improve the quality of the synthetic data. The inter-reflection also known as global illumination is a naturally occurring phenomena and is a major problem for 3D scene generation from an image. Thus, the proposed methods utilised a reverted ray-tracing technique to reduce the effect of inter-reflection problem and increased the quality of generated data. In addition, a method to improve the quality of the density map is discussed in the following chapter. The density map is the most commonly used technique to estimate crowds. However, the current procedure used to generate the map is not content-aware i.e., density map does not highlight the humans’ heads according to their size in the image. Thus, a novel method to generate a content-aware density map was proposed and demonstrated that the use of such maps can elevate the performance of an existing Deep Learning architecture. In the final part, a Deep Learning architecture has been proposed to estimate the crowd in the wild. The architecture tackled the challenging aspect such as perspective distortion by implementing several techniques like pyramid style inputs, scale aggregation method and self-attention mechanism to estimate a crowd density map and achieved state-of-the-art results at the time

Simulation Intelligence: Towards a New Generation of Scientific Methods

The original "Seven Motifs" set forth a roadmap of essential methods for the field of scientific computing, where a motif is an algorithmic method that captures a pattern of computation and data movement. We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simulation, and artificial intelligence. We call this merger simulation intelligence (SI), for short. We argue the motifs of simulation intelligence are interconnected and interdependent, much like the components within the layers of an operating system. Using this metaphor, we explore the nature of each layer of the simulation intelligence operating system stack (SI-stack) and the motifs therein: (1) Multi-physics and multi-scale modeling; (2) Surrogate modeling and emulation; (3) Simulation-based inference; (4) Causal modeling and inference; (5) Agent-based modeling; (6) Probabilistic programming; (7) Differentiable programming; (8) Open-ended optimization; (9) Machine programming. We believe coordinated efforts between motifs offers immense opportunity to accelerate scientific discovery, from solving inverse problems in synthetic biology and climate science, to directing nuclear energy experiments and predicting emergent behavior in socioeconomic settings. We elaborate on each layer of the SI-stack, detailing the state-of-art methods, presenting examples to highlight challenges and opportunities, and advocating for specific ways to advance the motifs and the synergies from their combinations. Advancing and integrating these technologies can enable a robust and efficient hypothesis-simulation-analysis type of scientific method, which we introduce with several use-cases for human-machine teaming and automated science

EVA London 2022: Electronic Visualisation and the Arts

The Electronic Visualisation and the Arts London 2022 Conference (EVA London 2022) is co-sponsored by the Computer Arts Society (CAS) and BCS, the Chartered Institute for IT, of which the CAS is a Specialist Group. Of course, this has been a difficult time for all conferences, with the Covid-19 pandemic. For the first time since 2019, the EVA London 2022 Conference is a physical conference. It is also an online conference, as it was in the previous two years. We continue with publishing the proceedings, both online, with open access via ScienceOpen, and also in our traditional printed form, for the second year in full colour. Over recent decades, the EVA London Conference on Electronic Visualisation and the Arts has established itself as one of the United Kingdom’s most innovative and interdisciplinary conferences. It brings together a wide range of research domains to celebrate a diverse set of interests, with a specialised focus on visualisation. The long and short papers in this volume cover varied topics concerning the arts, visualisations, and IT, including 3D graphics, animation, artificial intelligence, creativity, culture, design, digital art, ethics, heritage, literature, museums, music, philosophy, politics, publishing, social media, and virtual reality, as well as other related interdisciplinary areas. The EVA London 2022 proceedings presents a wide spectrum of papers, demonstrations, Research Workshop contributions, other workshops, and for the seventh year, the EVA London Symposium, in the form of an opening morning session, with three invited contributors. The conference includes a number of other associated evening events including ones organised by the Computer Arts Society, Art in Flux, and EVA International. As in previous years, there are Research Workshop contributions in this volume, aimed at encouraging participation by postgraduate students and early-career artists, accepted either through the peer-review process or directly by the Research Workshop chair. The Research Workshop contributors are offered bursaries to aid participation. In particular, EVA London liaises with Art in Flux, a London-based group of digital artists. The EVA London 2022 proceedings includes long papers and short “poster” papers from international researchers inside and outside academia, from graduate artists, PhD students, industry professionals, established scholars, and senior researchers, who value EVA London for its interdisciplinary community. The conference also features keynote talks. A special feature this year is support for Ukrainian culture after its invasion earlier in the year. This publication has resulted from a selective peer review process, fitting as many excellent submissions as possible into the proceedings. This year, submission numbers were lower than previous years, mostly likely due to the pandemic and a new requirement to submit drafts of long papers for review as well as abstracts. It is still pleasing to have so many good proposals from which to select the papers that have been included. EVA London is part of a larger network of EVA international conferences. EVA events have been held in Athens, Beijing, Berlin, Brussels, California, Cambridge (both UK and USA), Canberra, Copenhagen, Dallas, Delhi, Edinburgh, Florence, Gifu (Japan), Glasgow, Harvard, Jerusalem, Kiev, Laval, London, Madrid, Montreal, Moscow, New York, Paris, Prague, St Petersburg, Thessaloniki, and Warsaw. Further venues for EVA conferences are very much encouraged by the EVA community. As noted earlier, this volume is a record of accepted submissions to EVA London 2022. Associated online presentations are in general recorded and made available online after the conference