Hierarchical and Spatial Structures for Interpreting Images of Man-made Scenes Using Graphical Models

The task of semantic scene interpretation is to label the regions of an image and their relations into meaningful classes. Such task is a key ingredient to many computer vision applications, including object recognition, 3D reconstruction and robotic perception. It is challenging partially due to the ambiguities inherent to the image data. The images of man-made scenes, e. g. the building facade images, exhibit strong contextual dependencies in the form of the spatial and hierarchical structures. Modelling these structures is central for such interpretation task. Graphical models provide a consistent framework for the statistical modelling. Bayesian networks and random fields are two popular types of the graphical models, which are frequently used for capturing such contextual information. The motivation for our work comes from the belief that we can find a generic formulation for scene interpretation that having both the benefits from random fields and Bayesian networks. It should have clear semantic interpretability. Therefore our key contribution is the development of a generic statistical graphical model for scene interpretation, which seamlessly integrates different types of the image features, and the spatial structural information and the hierarchical structural information defined over the multi-scale image segmentation. It unifies the ideas of existing approaches, e. g. conditional random field (CRF) and Bayesian network (BN), which has a clear statistical interpretation as the maximum a posteriori (MAP) estimate of a multi-class labelling problem. Given the graphical model structure, we derive the probability distribution of the model based on the factorization property implied in the model structure. The statistical model leads to an energy function that can be optimized approximately by either loopy belief propagation or graph cut based move making algorithm. The particular type of the features, the spatial structure, and the hierarchical structure however is not prescribed. In the experiments, we concentrate on terrestrial man-made scenes as a specifically difficult problem. We demonstrate the application of the proposed graphical model on the task of multi-class classification of building facade image regions. The framework for scene interpretation allows for significantly better classification results than the standard classical local classification approach on man-made scenes by incorporating the spatial and hierarchical structures. We investigate the performance of the algorithms on a public dataset to show the relative importance of the information from the spatial structure and the hierarchical structure. As a baseline for the region classification, we use an efficient randomized decision forest classifier. Two specific models are derived from the proposed graphical model, namely the hierarchical CRF and the hierarchical mixed graphical model. We show that these two models produce better classification results than both the baseline region classifier and the flat CRF.Hierarchische und räumliche Strukturen zur Interpretation von Bildern anthropogener Szenen unter Nutzung graphischer Modelle Ziel der semantischen Bildinterpretation ist es, Bildregionen und ihre gegenseitigen Beziehungen zu kennzeichnen und in sinnvolle Klassen einzuteilen. Dies ist eine der Hauptaufgabe in vielen Bereichen des maschinellen Sehens, wie zum Beispiel der Objekterkennung, 3D Rekonstruktion oder der Wahrnehmung von Robotern. Insbesondere Bilder anthropogener Szenen, wie z.B. Fassadenaufnahmen, sind durch starke räumliche und hierarchische Strukturen gekennzeichnet. Diese Strukturen zu modellieren ist zentrale Teil der Interpretation, für deren statistische Modellierung graphische Modelle ein geeignetes konsistentes Werkzeug darstellen. Bayes Netze und Zufallsfelder sind zwei bekannte und häufig genutzte Beispiele für graphische Modelle zur Erfassung kontextabhängiger Informationen. Die Motivation dieser Arbeit liegt in der überzeugung, dass wir eine generische Formulierung der Bildinterpretation mit klarer semantischer Bedeutung finden können, die die Vorteile von Bayes Netzen und Zufallsfeldern verbindet. Der Hauptbeitrag der vorliegenden Arbeit liegt daher in der Entwicklung eines generischen statistischen graphischen Modells zur Bildinterpretation, welches unterschiedlichste Typen von Bildmerkmalen und die räumlichen sowie hierarchischen Strukturinformationen über eine multiskalen Bildsegmentierung integriert. Das Modell vereinheitlicht die existierender Arbeiten zugrunde liegenden Ideen, wie bedingter Zufallsfelder (conditional random field (CRF)) und Bayesnetze (Bayesian network (BN)). Dieses Modell hat eine klare statistische Interpretation als Maximum a posteriori (MAP) Schätzer eines mehrklassen Zuordnungsproblems. Gegeben die Struktur des graphischen Modells und den dadurch definierten Faktorisierungseigenschaften leiten wir die Wahrscheinlichkeitsverteilung des Modells ab. Dies führt zu einer Energiefunktion, die näherungsweise optimiert werden kann. Der jeweilige Typ der Bildmerkmale, die räumliche sowie hierarchische Struktur ist von dieser Formulierung unabhängig. Wir zeigen die Anwendung des vorgeschlagenen graphischen Modells anhand der mehrklassen Zuordnung von Bildregionen in Fassadenaufnahmen. Wir demonstrieren, dass das vorgeschlagene Verfahren zur Bildinterpretation, durch die Berücksichtigung räumlicher sowie hierarchischer Strukturen, signifikant bessere Klassifikationsergebnisse zeigt, als klassische lokale Klassifikationsverfahren. Die Leistungsfähigkeit des vorgeschlagenen Verfahrens wird anhand eines öffentlich verfügbarer Datensatzes evaluiert. Zur Klassifikation der Bildregionen nutzen wir ein Verfahren basierend auf einem effizienten Random Forest Klassifikator. Aus dem vorgeschlagenen allgemeinen graphischen Modell werden konkret zwei spezielle Modelle abgeleitet, ein hierarchisches bedingtes Zufallsfeld (hierarchical CRF) sowie ein hierarchisches gemischtes graphisches Modell. Wir zeigen, dass beide Modelle bessere Klassifikationsergebnisse erzeugen als die zugrunde liegenden lokalen Klassifikatoren oder die einfachen bedingten Zufallsfelder

Advances in Evolutionary Algorithms

With the recent trends towards massive data sets and significant computational power, combined with evolutionary algorithmic advances evolutionary computation is becoming much more relevant to practice. Aim of the book is to present recent improvements, innovative ideas and concepts in a part of a huge EA field

Flood Forecasting Using Machine Learning Methods

This book is a printed edition of the Special Issue Flood Forecasting Using Machine Learning Methods that was published in Wate

Transcriptomic cellular diversity of the early human developing brain

The complexity of the mammalian brain is partly reflected in its cell type diversity which influences the function of neurons that encode the behavior of animals. Brain cell type diversity emerges during embryonic stages, a critical period when neurons start to become functionally active and establish their connectivity across the brain. Since the pioneering of single-cell RNA-sequencing (scRNA-seq), we can question when and how cellular diversity arises in the brain in a large-scale manner. This thesis aims to study the human brain during the first trimester by using scRNA-seq to obtain a global view of the basic principles of the developing brain. First, I introduce human embryology from a historical perspective and summarize key concepts in central nervous system (CNS) development. I review few gaps in the field related to our findings, followed by current approaches and nomenclatures used in the field of single-cell genomics that applies to development. To put our work into perspective, I present an overview of the latest efforts to study human brain development at the single-cell level, both in the healthy and diseased brain. Then I present the following two papers and a manuscript: In Paper I we used scRNA-seq to construct a cell taxonomy of the adult mouse nervous system. We describe two major groups: neuronal- and non-neuronal cells that were subdivided into distinct cell types. Overall, the neurons were transcriptionally similar across brain regions, whereas non-neuronal cells such as astrocytes, formed subgroups and were regionally distinct. The whole dataset revealed an organization that reflects the developmental origin of all cell types. Paper II describes a method, RNA velocity, that infers temporal changes from static scRNAseq gene expression measurements. By realigning sequencing reads, this method detects and makes use of the unspliced and spliced mRNA, whose relative abundance is used to measure the change of rate in gene expression (the time derivative) in different tissues. This method is particularly suitable for developmental lineages, which was shown and validated both in vitro and in situ in this study. Paper III presents a single-cell atlas of the human developing CNS across all major brain regions during postconceptional weeks (p.c.w.) 5 to 14. We observe that major cell classes emerge during this period, most of them being regionally diverse and to a surprisingly high degree among glial cells. We display the high resolution of this data by resolving several lineages in the forebrain and validated the spatial location of transcriptional cell types at 5 p.c.w. by using single-molecule FISH. As a whole, this study represents a reference of human brain development during the first critical period in life. To tie these studies together, our findings on glial diversity were partially shared between the adult mouse and developing human CNS. We further showed that an RNA velocity-based method can be used to model the cell cycle dynamics in cortical tissue. To conclude, I discuss advantages and limitations of single-cell transcriptomics, its future challenges and how using this technology sheds light on the early human developing brain as is described in this thesis

多目的進化戦略方法によるタンパク質立体構造予測研究

富山大学・富理工博甲第174号・宋双宝・2020/3/24富山大学202

Using single-cell RNA-seq to assess the effect of common genetic variants on gene expression during development

Over the last fifteen years, genome-wide association studies (GWAS) have been used to identify thousands of DNA variants associated with complex traits and diseases, by exploiting naturally occurring genetic variation in large populations of individuals. More recently, similar approaches have been applied to RNA sequencing (RNA-seq) data to find variants associated with expression level, called expression quantitative trait loci (eQTL). Recent advances in experimental techniques have provided an unprecedented opportunity to measure gene expression at the single cell level, and the chance to study cellular heterogeneity. This represents a remarkable advance over traditional bulk sequencing methods, particularly to study cell fate commitment events in development. The challenge of studying early human development is partially overcome by advances in stem cell technologies. In particular, induced pluripotent stem cells (iPSCs) and cells derived therefrom represent a fantastic system to study development in vitro. In this thesis, I investigate the computational challenges of using single cell expression profiles to perform expression quantitative trait locus (eQTL) mapping, and provide suitable approaches for the identification of cell type and context-specific eQTL using single cell expression profiles. I further explore the application of such methods across a range of human iPSC-derived cell types, using data from the human induced pluripotent stem cell initiative (HipSci) project.Funding was provided via the EMBL international PhD programme

Unsupervised Attributed Graph Learning: Models and Applications

abstract: Graph is a ubiquitous data structure, which appears in a broad range of real-world scenarios. Accordingly, there has been a surge of research to represent and learn from graphs in order to accomplish various machine learning and graph analysis tasks. However, most of these efforts only utilize the graph structure while nodes in real-world graphs usually come with a rich set of attributes. Typical examples of such nodes and their attributes are users and their profiles in social networks, scientific articles and their content in citation networks, protein molecules and their gene sets in biological networks as well as web pages and their content on the Web. Utilizing node features in such graphs---attributed graphs---can alleviate the graph sparsity problem and help explain various phenomena (e.g., the motives behind the formation of communities in social networks). Therefore, further study of attributed graphs is required to take full advantage of node attributes. In the wild, attributed graphs are usually unlabeled. Moreover, annotating data is an expensive and time-consuming process, which suffers from many limitations such as annotators’ subjectivity, reproducibility, and consistency. The challenges of data annotation and the growing increase of unlabeled attributed graphs in various real-world applications significantly demand unsupervised learning for attributed graphs. In this dissertation, I propose a set of novel models to learn from attributed graphs in an unsupervised manner. To better understand and represent nodes and communities in attributed graphs, I present different models in node and community levels. In node level, I utilize node features as well as the graph structure in attributed graphs to learn distributed representations of nodes, which can be useful in a variety of downstream machine learning applications. In community level, with a focus on social media, I take advantage of both node attributes and the graph structure to discover not only communities but also their sentiment-driven profiles and inter-community relations (i.e., alliance, antagonism, or no relation). The discovered community profiles and relations help to better understand the structure and dynamics of social media.Dissertation/ThesisDoctoral Dissertation Computer Science 201

The impact of genotype on the cellular architecture of dilated and arrhythmogenic cardiomyopathies

Herzinsuffizienz ist ein klinisches Syndrom, welches durch funktionelle und strukturelle Anomalien des Herzens verursacht wird, und ist weltweit die häufigste Todesursache. Die dilatative Kardiomyopathie, welche durch eine Vergrößerung der linken Herzkammer definiert ist, und die arrhythmogene Kardiomyopathie, welche im Gegensatz durch eine Dysfunktion der rechten Herzkammer definiert ist, sind eine der häufigsten Ursachen für Herzinsuffizienz. Trotz vieler Bemühungen die molekularen Veränderungen der Herzinsuffizienz zu charakterisieren, sind Zelltypzusammensetzung, Genexpressionsänderungen, und zelluläre Interaktionen unter pathologischen Bedingungen unbekannt. Um diese Fragen zu adressieren wurde ein Protokoll zur Isolation intakter Zellkerne entwickelt um Einzelkernsequenzierung im Herzen durchzuführen. Anschließend wurde mit dem entwickelten Protokoll die zelluläre Zusammensetzung des erwachsenen gesunden menschlichen Herzens charakterisiert. Hier war mein Fokus die Charakterisierung und Identifikation von Subformen von Fibroblasten, und deren Genexpressionsunterschiede in den linken und rechten Vorhöfen und Herzkammern. Basierend auf dieser Annotation wurden die Zelltypen und Subtypen von ungefähr 900.000 Zellkernen von 61 nicht-ischämischen Herzinsuffizienzpatienten mit unterschliedlichen pathogenen Varianten in DCM- und ACM-assoziierten Genen oder idiopathischen Erkrankungen charakterisiert und mit 18 gesunden Spenderherzen verglichen. Dieser Datensatz zeigte spezifische Unterschiede des linken und rechten Ventrikels mit differenziell regulierten Genen und Signalwegen, and Veränderungen in der Zusammensetzung der verschiedenen Zelltypen und Subtypen. Um genotyp-spezifische Antworten unabhängig zu bestätigen wurden Algorithmen des maschinellen Lernens angewendet, welche die zugehörige Genotyp-Untergruppe des Patienten mit hoher Genauigkeit vorhersagten. Zusammenfassend stellen die in dieser Arbeit veröffentlichten Daten das vorherrschende Dogma in Frage, dass Herzinsuffizienz auf einen gemeinsamen finalen Signalweg zurückzuführen ist.Heart failure is a clinical syndrom and leading cause of death worldwide, caused by functional and structural abnormalities of the heart. Dilated Cardiomyopathy, defined by a left ventricular enlargement, and arrhythmogenic cardiomyopathy, defined by a right ventricular dysfunction, are leading causes of heart failure. Despite previous efforts to characterise molecular changes in the failing heart, little is known on cell-type specific abundance and expression changes under pathological conditions, and how individual cell-types interact during heart failure and cardiac remodelling. To address this question, a protocol for the isolation of intact nuclei was firstly established to perform robust single-nucleus RNA sequencing in the heart. Next, the cell-type composition of the healthy adult human heart was characterised. Here my focus was on the fibroblast nieche by characterising fibroblast states, their composition and their atria- and ventricle-specific expression patterns. Cell type and state annotation was then used to characterize the transcriptome of roughly 900,000 nuclei from 61 failing, non-ischemic human hearts with distinct pathogenic variants in DCM and ACM genes or idiopathic disease and compared those to 18 healthy donor hearts. This dataset revealed distinct responses of the right and left ventricle with differently regulated genes and pathways, and compositional changes across cell types and states. To independently confirm genotype-specific responses, machine learning approaches were applied, predicting genotype subgroups with high accuracy. Taken together, the findings published in this thesis upend the prevalent dogma that heart failure results in a final common pathway