Applied microlocal analysis of deep neural networks for inverse problems

Deep neural networks have recently shown state-of-the-art performance in different imaging tasks. As an example, EfficientNet is today the best image classifier on the ImageNet challenge. They are also very powerful for image reconstruction, for example, deep learning currently yields the best methods for CT reconstruction. Most imaging problems, such as CT reconstruction, are ill-posed inverse problems, which hence require regularization techniques typically based on a-priori information. Also, due to the human visual system, singularities such as edge-like features are the governing structures of images. This leads to the question of how to incorporate such information into a solver of an inverse problem in imaging and how deep neural networks operate on singularities. The main research theme of this thesis is to introduce theoretically founded approaches to use deep neural networks in combination with model-based methods to solve inverse problems from imaging science. We do this by heavily exploring the singularity structure of images as a-priori information. We then develop a comprehensive analysis of how neural networks act on singularities using predominantly methods from the microlocal analysis. For analyzing the interaction of deep neural networks with singularities, we introduce a novel technique to compute the propagation of wavefront sets through convolutional residual neural networks (conv-ResNet). This is achieved in a two-fold manner: We first study the continuous case where the neural network is defined in an infinite-dimensional continuous space. This problem is tackled by using the structure of these networks as a sequential application of continuous convolutional operators and ReLU non-linearities and applying microlocal analysis techniques to track the propagation of the wavefront set through the layers. This then leads to the so-called \emph{microcanonical relation} that describes the propagation of the wavefront set under the action of such a neural network. Secondly, for studying real-world discrete problems, we digitize the necessary microlocal analysis methods via the digital shearlet transform. The key idea is the fact that the shearlet transform optimally represents Fourier integral operators hence such a discretization decays rapidly, allowing a finite approximation. Fourier integral operators play an important role in microlocal analysis, since it is well known that they preserve singularities on functions, and, in addition, they have a closed form microcanonical relation. Also, based on the newly developed theoretical analysis, we introduce a method that uses digital shearlet coefficients to compute the digital wavefront set of images by a convolutional neural network. Our approach is then used for a similar analysis of the microlocal behavior of the learned-primal dual architecture, which is formed by a sequence of conv-ResNet blocks. This architecture has shown state-of-the-art performance in inverse problem regularization, in particular, computed tomography reconstruction related to the Radon transform. Since the Radon operator is a Fourier integral operator, our microlocal techniques can be applied. Therefore, we can study with high precision the singularities propagation of this architecture. Aiming to empirically analyze our theoretical approach, we focus on the reconstruction of X-ray tomographic data. We approach this problem by using a task-adapted reconstruction framework, in which we combine the task of reconstruction with the task of computing the wavefront set of the original image as a-priori information. Our numerical results show superior performance with respect to current state-of-the-art tomographic reconstruction methods; hence we anticipate our work to also be a significant contribution to the biomedical imaging community.Tiefe neuronale Netze haben in letzter Zeit bei verschiedenen Bildverarbeitungsaufgaben Spitzenleistungen gezeigt. Zum Beispiel ist AlexNet heute der beste Bildklassifikator bei der ImageNet-Challenge. Sie sind auch sehr leistungsfaehig fue die Bildrekonstruktion, zum Beispiel liefert Deep Learning derzeit die besten Methoden fuer die CT-Rekonstruktion. Die meisten Bildgebungsprobleme wie die CT-Rekonstruktion sind schlecht gestellte inverse Probleme, die daher Regularisierungstechniken erfordern, die typischerweise auf vorherigen Informationen basieren. Auch aufgrund des menschlichen visuellen Systems sind Singularitaeten wie kantenartige Merkmale die bestimmenden Strukturen von Bildern. Dies fuehrt zu der Frage, wie man solche Informationen in einen Loeser eines inversen Problems in der Bildverarbeitung einbeziehen kann und wie tiefe neuronale Netze mit Singularitaeten arbeiten. Das Hauptforschungsthema dieser Arbeit ist die Einfuehrung theoretisch fundierter konzeptioneller Ansaetze zur Verwendung von tiefen neuronalen Netzen in Kombination mit modellbasierten Methoden zur Loesung inverser Probleme aus der Bildwissenschaft. Wir tun dies, indem wir die Singularitaetsstruktur von Bildern als Vorinformation intensiv erforschen. Dazu entwickeln wir eine umfassende Analyse, wie neuronale Netze auf Singularitaeten wirken, indem wir vorwiegend Methoden aus der mikrolokalen Analyse verwenden. Um die Interaktion von tiefen neuronalen Netzen mit Singularitaeten zu analysieren, fuehren wir eine neuartige Technik ein, um die Ausbreitung von Wellenfrontsaetzen mit Hilfe von Convolutional Residual neuronalen Netzen (Conv-ResNet) zu berechnen. Dies wird auf zweierlei Weise erreicht: Zunaechst untersuchen wir den kontinuierlichen Fall, bei dem das neuronale Netz in einem unendlich dimensionalen kontinuierlichen Raum definiert ist. Dieses Problem wird angegangen, indem wir die besondere Struktur dieser Netze als sequentielle Anwendung von kontinuierlichen Faltungsoperatoren und ReLU-Nichtlinearitaeten nutzen und mikrolokale Analyseverfahren anwenden, um die Ausbreitung einer Wellenfrontmenge durch die Schichten zu verfolgen. Dies fuehrt dann zu einer mikrokanonischen Beziehung, die die Ausbreitung der Wellenfrontmenge unter ihrer Wirkung beschreibt. Zweitens digitalisieren wir die notwendigen mikrolokalen Analysemethoden ueber die digitale Shearlet-Transformation, wobei die Digitalisierung fuer die Untersuchung realer Probleme notwendig ist. Die Schluesselidee ist die Tatsache, dass die Shearlet-Transformation Fourier-Integraloperatoren optimal repraesentiert, so dass eine solche Diskretisierung schnell abklingt und eine endliche Approximation ermoeglicht. Nebenbei stellen wir auch eine Methode vor, die digitale Shearlet-Koeffizienten verwendet, um den digitalen Wellenfrontsatz von Bildern durch ein Faltungsneuronales Netzwerk zu berechnen. Unser Ansatz wird dann fuer eine aehnliche Analyse fuer die gelernte primale-duale Architektur verwendet, die durch eine Sequenz von conv-ResNet-Bloecken gebildet wird. Diese Architektur hat bei der Rekonstruktion inverser Probleme, insbesondere bei der Rekonstruktion der Computertomographie im Zusammenhang mit der Radon-Transformation, Spitzenleistungen gezeigt. Da der Radon-Operator ein Fourier-Integraloperator ist, koennen unsere mikrolokalen Techniken angewendet werden. Um unseren theoretischen Ansatz numerisch zu analysieren, konzentrieren wir uns auf die Rekonstruktion von Roentgentomographiedaten. Wir naehern uns diesem Problem mit Hilfe eines aufgabenangepassten Rekonstruktionsrahmens, in dem wir die Aufgabe der Rekonstruktion mit der Aufgabe der Berechnung der Wellenfrontmenge des Originalbildes als Vorinformation kombinieren. Unsere numerischen Ergebnisse zeigen eine ueberragende Leistung, daher erwarten wir, dass dies auch ein interessanter Beitrag fuer die biomedizinische Bildgebung sein wird

Use of expert elicitation to assign weights to climate and hydrological models in climate impact studies

Various methods are available for assessing uncertainties in climate impact studies. Among such methods, model weighting by expert elicitation is a practical way to provide a weighted ensemble of models for specific real-world impacts. The aim is to decrease the influence of improbable models in the results and easing the decisionmaking process. In this study both climate and hydrological models are analysed, and the result of a research experiment is presented using model weighting with the participation of six climate model experts and six hydrological model experts. For the experiment, seven climate models are a priori selected from a larger EURO-CORDEX (Coordinated Regional Downscaling Experiment – European Domain) ensemble of climate models, and three different hydrological models are chosen for each of the three European river basins. The model weighting is based on qualitative evaluation by the experts for each of the selected models based on a training material that describes the overall model structure and literature about climate models and the performance of hydrological models for the present period. The expert elicitation process follows a three-stage approach, with two individual rounds of elicitation of probabilities and a final group consensus, where the experts are separated into two different community groups: a climate and a hydrological modeller group. The dialogue reveals that under the conditions of the study, most climate modellers prefer the equal weighting of ensemble members, whereas hydrological-impact modellers in general are more open for assigning weights to different models in a multi-model ensemble, based on model performance and model structure. Climate experts are more open to exclude models, if obviously flawed, than to put weights on selected models in a relatively small ensemble. The study shows that expert elicitation can be an efficient way to assign weights to different hydrological models and thereby reduce the uncertainty in climate impact. However, for the climate model ensemble, comprising seven models, the elicitation in the format of this study could only re-establish a uniform weight between climate models.European Commission European Commission Joint Research Centre 69046

Improving the usability of climate services for the water sector: The AQUACLEW experience

In AQUACLEW (Advancing the QUAlity of CLimate services for European Water), a project funded by JPI Climate and the ERA-NET Consortium ‘European Research Area for Climate Services (ERA4CS), we examined different ways of improving the usability of existing Climate Services across Europe tackling key aspects in Climate Services improvement: user engagement, lack of resolution, uncertainties, and the need of an evaluation. The rationale of the project is based on an interactive process between service developers and users in seven study cases across Europe assessing the implications of Climate Service’ advancement in users’ decision-making process. A qualitative evaluation assessment allowed us to identify-four pillars when improving the quality of the climate services in the water sector: (1) Robustness, accounting for better quality of the service’ information in certain aspects; (2) Recruitment, understood as a need of involving users more actively in CS structures; (3) Reform, highlighting the possible need for changes in both the service structure and users mindsets; and (4) Reflection, as a process of continuous evaluation of the climate service during its life.Swedish Research Council FormasBMWFWSpanish GovernmentFrench National Research Agency (ANR)European Commission European Commission Joint Research CentreHelmholtz AssociationGerman Aerospace Centre (DLR)IFD 69046

Future changes and seasonal variability of the directional wave spectra in the Mediterranean Sea for the 21st century

A state-of-the-art regional assessment of future directional wave spectra in the Mediterranean Sea and the projected changes with respect to hindcast is presented. A multi-model EURO-CORDEX regional ensemble of bias-adjusted wave climate projections in eleven locations of the Mediterranean are used for the assessment of future seasonal changes in the directional wave spectra under the high-emission scenario RCP8.5. This analysis allows us to identify climate change effects on the spectral energy of the swell and wind-sea systems and their seasonal variability which cannot be captured with the standard integrated wave parameters, such as significant wave height and mean wave direction. The results show an overall robust decrease in the predominant wave systems, resulting in a likely decrease in the significant wave height that is in agreement with previous studies. However, the results depict a robust increase in other less energetic frequencies and directions leading to a projected behavioral change from unimodal to bimodal/multimodal wave climate in many locations which has strong repercussions on the vulnerability of coastal assets and ports operability

Leukemia and Retroviral Disease

Two human retroviruses, identified as the human T-cell leukemia virus type 1 (HTLV-1) and human immunodeficiency virus type 1 (HIV-1), have been shown to affect millions of people worldwide. In the context of coinfection, the impact of their interactions with respect to HTLV-1-induced adult T-cell leukemia and neurologic disease as well as HIV-1 disease progression has been an understudied area of investigation. HTLV-1/HIV-1 coinfections occur frequently, particularly in large metropolitan areas of the Americas, Africa, Europe, and Japan. The retroviruses HTLV-1 and HIV-1 share some similarities with regard to their genetic structure, general mechanisms of replication, modes of transmission, and cellular tropism; however, there are also significant differences in the details of these properties as well, and they also differ significantly with respect to the etiology of their pathogenic and disease outcomes. Both viruses impair the host immune system with HIV-1 demonstrated to cause the hallmark lethal disease known as the acquired immune deficiency syndrome (AIDS), while HTLV-1 infection has been shown to cause several different forms of T-cell leukemia. In addition, both viruses have also been shown to cause a spectrum of neurologic disorders with HIV-1 shown to cause an array of neurologic syndromes referred to as HIV-1-associated neurologic disorders or HAND, while HTLV-1 has been shown to be the etiologic agent of HTLV-1-associated myelopathy/tropical spastic paraparesis or HAM/TSP. The natural history of the coinfection, however, is different from that observed in monoinfections. Several studies have demonstrated utilizing a number of in vitro models of HTLV-1/HIV-1 coinfection that the two viruses interact in a manner that results in enhanced expression of both viral genomes. Nevertheless, there remains unresolved controversy regarding the overall impact of each virus on progression of disease caused by both viruses during the course of coinfection. Although combination antiretroviral therapy has been shown to work very effectively with respect to maintaining HIV-1 viral loads in the undetectable range, these therapeutic strategies exhibit no benefit for HTLV-1-infected individuals, unless administered immediately after exposure. Furthermore, the treatment options for HTLV-1/HIV-1-coinfected patients are very limited. In recent years, allogeneic stem cell transplantation (alloSCT) has been used for the treatment of leukemia. In this regard, the case of a leukemic patient positive for HIV-1 who was cured of their HIV-1 infection while treated with alloSCT for acute myeloid leukemia has also been examined with regard to impact on HIV-1 disease