Expanding the capabilities of normalizing flows in deep generative models and variational inference

Deep generative models and variational Bayesian inference are two frameworks for reasoning about observed high-dimensional data, which may even be combined. A fundamental requirement of either approach is the parametrization of an expressive family of density models. Normalizing flows, sometimes also referred to as invertible neural networks, are one class of models providing this: they are formulated to be bijective and differentiable, and thus produce a tractable density model via the change-of-variable formula. Beyond just deep generative modelling and variational inference, normalizing flows have shown promise as a plug-in density model in other settings such as approximate Bayesian computation and lossless compression. However, the bijectivity constraint can pose quite a restriction on the expressiveness of these approaches, and forces the learned distribution to have full support over the ambient space which is not well-aligned with the common assumption that low-dimensional manifold structure is embedded within high-dimensional data. In this thesis, we challenge this requirement of strict bijectivity over the space of interest to modify normalizing flow models. The first work focuses on the setting of variational inference, defining a normalizing flow based on a discretized time-inhomogeneous Hamiltonian dynamics over an extended position-momentum space. This enables the flow to be guided by the true posterior unlike baseline flow-based models, thus requiring fewer parameters in the inference model to achieve comparable improvements in inference. The next chapter proposes a new deep generative model which relaxes the bijectivity requirement of normalizing flows by injecting learned noise at each layer, sacrificing easy evaluation of the density for expressiveness. We show, theoretically and empirically, the benefits of these models in density estimation over baseline flows. We then demonstrate in the next chapter that the benefits of this model class extend to the setting of variational inference, relying on auxiliary methods to train our models. Finally, the last paper in this thesis addresses the issue of full support in the ambient space and proposes injective flow models directly embedding low-dimensional structure into high dimensions. Our method is the first to optimize the injective change-of-variable term and produces promising results on out-of-distribution detection, which had previous eluded deep generative models. We conclude with some directions for future work and a broader perspective on the field

New Directions for Contact Integrators

Contact integrators are a family of geometric numerical schemes which guarantee the conservation of the contact structure. In this work we review the construction of both the variational and Hamiltonian versions of these methods. We illustrate some of the advantages of geometric integration in the dissipative setting by focusing on models inspired by recent studies in celestial mechanics and cosmology.Comment: To appear as Chapter 24 in GSI 2021, Springer LNCS 1282

Advanced deep active learning & data subset selection: unifying principles with information-theory intuitions

At its core, this thesis aims to enhance the practicality of deep learning by improving the label and training efficiency of deep learning models. To this end, we investigate data subset selection techniques, specifically active learning and active sampling, grounded in information-theoretic principles. Active learning improves label efficiency, while active sampling enhances training efficiency. Supervised deep learning models often require extensive training with labeled data. Label acquisition can be expensive and time-consuming, and training large models is resource-intensive, hindering the adoption outside academic research and "big tech." Existing methods for data subset selection in deep learning often rely on heuristics or lack a principled information-theoretic foundation. In contrast, this thesis examines several objectives for data subset selection and their applications within deep learning, striving for a more principled approach inspired by information theory. We begin by disentangling epistemic and aleatoric uncertainty in single forward-pass deep neural networks, which provides helpful intuitions and insights into different forms of uncertainty and their relevance for data subset selection. We then propose and investigate various approaches for active learning and data subset selection in (Bayesian) deep learning. Finally, we relate various existing and proposed approaches to approximations of information quantities in weight or prediction space. Underpinning this work is a principled and practical notation for information-theoretic quantities that includes both random variables and observed outcomes. This thesis demonstrates the benefits of working from a unified perspective and highlights the potential impact of our contributions to the practical application of deep learning

Safety and Reliability - Safe Societies in a Changing World

The contributions cover a wide range of methodologies and application areas for safety and reliability that contribute to safe societies in a changing world. These methodologies and applications include: - foundations of risk and reliability assessment and management - mathematical methods in reliability and safety - risk assessment - risk management - system reliability - uncertainty analysis - digitalization and big data - prognostics and system health management - occupational safety - accident and incident modeling - maintenance modeling and applications - simulation for safety and reliability analysis - dynamic risk and barrier management - organizational factors and safety culture - human factors and human reliability - resilience engineering - structural reliability - natural hazards - security - economic analysis in risk managemen