Dilated Spatial Generative Adversarial Networks for Ergodic Image Generation

Generative models have recently received renewed attention as a result of adversarial learning. Generative adversarial networks consist of samples generation model and a discrimination model able to distinguish between genuine and synthetic samples. In combination with convolutional (for the discriminator) and de-convolutional (for the generator) layers, they are particularly suitable for image generation, especially of natural scenes. However, the presence of fully connected layers adds global dependencies in the generated images. This may lead to high and global variations in the generated sample for small local variations in the input noise. In this work we propose to use architec-tures based on fully convolutional networks (including among others dilated layers), architectures specifically designed to generate globally ergodic images, that is images without global dependencies. Conducted experiments reveal that these architectures are well suited for generating natural textures such as geologic structures

Variational Bayesian inference with complex geostatistical priors using inverse autoregressive flows

We combine inverse autoregressive flows (IAF) and variational Bayesian inference (variational Bayes) in the context of geophysical inversion parameterized with deep generative models encoding complex priors. Variational Bayes approximates the unnormalized posterior distribution parametrically within a given family of distributions by solving an optimization problem. Although prone to bias if the chosen family of distributions is too limited, it provides a computationally-efficient approach that scales well to high-dimensional inverse problems. To enhance the expressiveness of the variational distribution, we explore its combination with IAFs that allow samples from a simple base distribution to be pushed forward through a series of invertible transformations onto an approximate posterior. The IAF is learned by maximizing the lower bound of the evidence (marginal likelihood), which is equivalent to minimizing the Kullback–Leibler divergence between the approximation and the target posterior distribution. In our examples, we use either a deep generative adversarial network (GAN) or a variational autoencoder (VAE) to parameterize complex geostatistical priors. Although previous attempts to perform Gauss–Newton inversion in combination with GANs of the same architecture were proven unsuccessful, the trained IAF provides a good reconstruction of channelized subsurface models for both GAN- and VAE-based inversions using synthetic crosshole ground-penetrating-radar data. For the considered examples, the computational cost of our approach is seven times lower than for Markov chain Monte Carlo (MCMC) inversion. Furthermore, the VAE-based approximations in the latent space are in good agreement. The VAE-based inversion requires only one sample to estimate gradients with respect to the IAF parameters at each iteration, while the GAN-based inversions need more samples and the corresponding posterior approximation is less accurate

Hydrogeological multiple-point statistics inversion by adaptive sequential Monte Carlo

For strongly non-linear and high-dimensional inverse problems, Markov chain Monte Carlo (MCMC) methods may fail to properly explore the posterior probability density function (PDF) given a realistic computational budget and are generally poorly amenable to parallelization. Particle methods approximate the posterior PDF using the states and weights of a population of evolving particles and they are very well suited to parallelization. We focus on adaptive sequential Monte Carlo (ASMC), an extension of annealed importance sampling (AIS). In AIS and ASMC, importance sampling is performed over a sequence of intermediate distributions, known as power posteriors, linking the prior to the posterior PDF. The AIS and ASMC algorithms also provide estimates of the evidence (marginal likelihood) as needed for Bayesian model selection, at basically no additional cost. ASMC performs better than AIS as it adaptively tunes the tempering schedule and performs resampling of particles when the variance of the particle weights becomes too large. We consider a challenging synthetic groundwater transport inverse problem with a categorical channelized 2D hydraulic conductivity field defined such that the posterior facies distribution includes two distinct modes. The model proposals are obtained by iteratively re-simulating a fraction of the current model using conditional multiple-point statistics (MPS) simulations. We examine how ASMC explores the posterior PDF and compare with results obtained with parallel tempering (PT), a state-of-the-art MCMC inversion approach that runs multiple interacting chains targeting different power posteriors. For a similar computational budget, ASMC outperforms PT as the ASMC-derived models fit the data better and recover the reference likelihood. Moreover, we show that ASMC partly retrieves both posterior modes, while none of them is recovered by PT. Lastly, we demonstrate how the power posteriors obtained by ASMC can be used to assess the influence of the assumed data errors on the posterior means and variances, as well as on the evidence. We suggest that ASMC can advantageously replace MCMC for solving many challenging inverse problems arising in the field of water resources

Estimation of spatially-structured subsurface parameters using variational autoencoders and gradient-based optimization

Environmental models of the subsurface usually require the estimation of high-dimensional spatially-distributed parameters. However, the sparsity of subsurface data hinders such estimation which may in turn affect predictions using these models. In order to mitigate this issue, parameter estimation can be constrained to prior information on the expected subsurface patterns (e.g. geological facies). By using several examples of such patterns (e.g. those obtained from a training image), a variational autoencoder (VAE) learns a low-dimensional latent space that can be seen as a reparameterization of the original high-dimensional parameters and then estimation by means of gradient-based optimization can be performed in this latent space. Spatial parameters estimated in this way display the enforced patterns. VAEs usually include deep neural networks within their architecture and have shown good performance in reproducing high-dimensional structured subsurface models. However, they use a highly nonlinear function to map from latent space to the original high-dimensional parameter space which may give rise to local minima where gradient-based optimization gets trapped and therefore fails to reach the global minimum. Global optimization strategies may be used to solve this issue, however, a gradient-based inversion is preferred because of its lower computational demand. We propose using VAE together with gradient-based optimization in a linear traveltime tomography synthetic case with added noise and show that it often reaches low data error values and produces visually similar spatial parameters when compared with different test subsurface realizations. We also add regularization to the objective function to improve the number of times it reaches an adequate minimum. Finally, we perform a synthetic test with nonlinear traveltime tomography and show that the proposed strategy is able to recover visually similar spatial parameters with an error close to the added noise level

Constraining gradient-based inversion with a variational autoencoder to reproduce geological patterns

Given the sparsity of geophysical data it is useful to rely on prior information on the expected geological patterns to constrain the inverse problem and obtain a realistic image of the subsurface. By using several examples of such patterns (e.g. those obtained from a training image), deep generative models learn a low-dimensional latent space that can be seen as a reparameterization of the original high-dimensional parameters and then inversion can be done in this latent space. Examples of such generative models are the variational autoencoder (VAE) and the generative adversarial network (GAN). Both usually include deep neural networks within their architecture and have shown good performance in reproducing high-dimensional structured subsurface models. However, they both use a highly nonlinear function to map from latent space to the original high-dimensional parameter space which hinders the optimization of the objective function during inversion. Particularly, such nonlinearity may give rise to local minima where gradient-based inversion gets trapped and therefore fails to reach the global minimum. GAN has been previously used with gradient-based inversion in a linear traveltime tomography synthetic test where it was shown to often fail in reaching a consistent RMSE (compared to the added noise) because optimization converges to local minima. On the other hand, inversion with MCMC and GAN was shown to reach acceptable RMSE values. When applicable, however, a gradient-based inversion is preferred because of its lower computational demand. We propose using VAE together with gradient-based inversion and show that optimization reaches lower RMSE values on average compared to GAN in a linear traveltime tomography synthetic case. We also compare the subsurface models that are generated during the iterations of the optimization to explore the effect of the different latent spaces used by GAN and VAE. We identify a trade-off between a strict following of the patterns and getting trapped in local minima during optimization, i.e. VAE seems to be able to break some continuous channels in order to not get trapped in local minima whereas GAN does not break channels. Finally, we perform some synthetic tests with nonlinear traveltime tomography and show that gradient-based inversion with VAE is able to recover a similar global structure to the true model but its final RMSE values are still far from the added noise level