Generative models: theory and applications

Given some samples from a data distribution, the central aim of generative modeling is to generate more samples from approximately the same distribution. This framework has recently seen an explosion in popularity, with impressive applications in image generation, language modeling and protein synthesis. The striking success of these methods motivates two key questions: under what conditions do generative models provide accurate approximations to the underlying data distribution, and can we extend the range of scenarios in which they may be applied? This thesis considers these questions in the context of two classes of generative models: diffusion models and importance weighted autoencoders. Diffusion models work by iteratively applying noise to the data distribution and then learning to remove this noise. They were originally introduced for real-valued data. However, for many potential applications our data is most naturally defined on another state space - perhaps a manifold, or a discrete space. We describe extensions of diffusion models to arbitrary state spaces, using generic Markov processes for noising, and show how such models can be effectively learned. We also provide a detailed study of a specific extension to discrete state spaces. Next, we investigate the approximation accuracy of diffusion models. We derive error bounds for flow matching - a generalization of diffusion models - and improve upon state-of-the-art bounds for diffusion models, using techniques inspired by stochastic localization. Importance weighted autoencoders (IWAEs) work by learning a latent variable representation of the data, using importance sampling in the evidence lower bound to get a tighter variational objective. IWAEs suffer from several limitations, including posterior variance underestimation, poor signal-to-noise ratio during training, and weight collapse in the importance sampling ratios. We propose an extension of the IWAE - the VR-IWAE - that addresses the first two of these three issues. We then provide a detailed theoretical study of the third, showing that it persists even for the VR-IWAE. We provide empirical demonstrations of these phenomena on a range of simulated and real-world data

New Directions for Contact Integrators

Contact integrators are a family of geometric numerical schemes which guarantee the conservation of the contact structure. In this work we review the construction of both the variational and Hamiltonian versions of these methods. We illustrate some of the advantages of geometric integration in the dissipative setting by focusing on models inspired by recent studies in celestial mechanics and cosmology.Comment: To appear as Chapter 24 in GSI 2021, Springer LNCS 1282

Wavelet Multiresolution Analysis of High-Frequency FX Rates, Summer 1997

FX pricing processes are nonstationary and their frequency characteristics are time-dependent. Most do not conform to geometric Brownian motion, since they exhibit a scaling law with a Hurst exponent between zero and 0.5 and fractal dimensions between 1.5 and 2. This paper uses wavelet multiresolution analysis, with Haar wavelets, to analyze the nonstationarity (time-dependence) and self-similarity (scale-dependence) of intra-day Asian currency spot exchange rates.foreign exchange, anti-persistence, multi-resolution analysis, wavelets, Asia

Wavelet Multiresolution Analysis of High-Frequency Asian FX Rates, Summer 1997

FX pricing processes are nonstationary and their frequency characteristics are time-dependent. Most do not conform to geometric Brownian motion, since they exhibit a scaling law with a Hurst exponent between zero and 0.5 and fractal dimensions between 1.5 and 2. This paper uses wavelet multiresolution analysis, with Haar wavelets, to analyze the nonstationarity (time-dependence) and self-similarity (scale-dependence) of intra-day Asian currency spot exchange rates. These are the ask and bid quotes of the currencies of eight Asian countries (Japan, Hong Kong, Indonesia, Malaysia, Philippines, Singapore, Taiwan, Thailand), and of Germany for comparison, for the crisis period May 1, 1998 - August 31, 1997, provided by Telerate (U.S. dollar is the numeraire). Their time-scale dependent spectra, which are localized in time, are observed in wavelet based scalograms. The FX increments can be characterized by the irregularity of their singularities. This degrees of irregularity are measured by homogeneous Hurst exponents. These critical exponents are used to identify the fractal dimension, relative stability and long term dependence of each Asian FX series. The invariance of each identified Hurst exponent is tested by comparing it at varying time and scale (frequency) resolutions. It appears that almost all FX markets show anti-persistent pricing behavior. The anchor currencies of the D-mark and Japanese Yen are ultra-efficient in the sense of being most anti-persistent. The Taiwanese dollar is the most persistent, and thus unpredictable, most likely due to administrative control. FX markets exhibit these non- linear, non-Gaussian dynamic structures, long term dependence, high kurtosis, and high degrees of non-informational (noise) trading, possibly because of frequent capital flows induced by non-synchronized regional business cycles, rapidly changing political risks, unexpected informational shocks to investment opportunities, and, in particular, investment strategies synthesizing interregional claims using cash swaps with different duration horizons.foreign exchange markets, anti-persistence, long-term dependence, multi-resolution analysis, wavelets, time-scale analysis, scaling laws, irregularity analysis, randomness, Asia

Computational Multiscale Methods

Almost all processes in engineering and the sciences are characterised by the complicated relation of features on a large range of nonseparable spatial and time scales. The workshop concerned the computer-aided simulation of such processes, the underlying numerical algorithms and the mathematics behind them to foresee their performance in practical applications

How market access shapes human capital investment in a peripheral country

Human capital endowment is one of the main factors influencing the level of development of a region. This paper analyses whether remoteness from economic activity has a negative effect on human capital accumulation and, consequently, on economic development. Making use of microdata this research proves that remoteness from economic activity has contributed to explain the divergences in the level of education observed across Spanish provinces over the last 50 years. The effect is significant even when controlling for the improvement of education supply. Nonetheless, the accessibility effect has been petering out since the 1960s due to the decreasing barriers to mobility

Authors' reply to the discussion of 'Automatic change-point detection in time series via deep learning' at the discussion meeting on 'Probabilistic and statistical aspects of machine learning'

We would like to thank the proposer, seconder, and all discussants for their time in reading our article and their thought-provoking comments. We are glad to find a broad consensus that neural-network-based approach offers a flexible framework for automatic change-point analysis. There are a number of common themes to the comments, and we have therefore structured our response around the topics of the theory, training, the importance of standardization and possible extensions, before addressing some of the remaining individual comments

A precise bare simulation approach to the minimization of some distances. Foundations

In information theory -- as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -- many flexibilizations of the omnipresent Kullback-Leibler information distance (relative entropy) and of the closely related Shannon entropy have become frequently used tools. To tackle corresponding constrained minimization (respectively maximization) problems by a newly developed dimension-free bare (pure) simulation method, is the main goal of this paper. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). As a side effect, we also derive an innovative way of constructing new useful distances/divergences. To illustrate the core of our approach, we present numerous examples. The potential for widespread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences and entropies in various different research fields (which may also serve as an interdisciplinary interface)