Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.Comment: The first two authors contributed equally to this wor

Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.Comment: Journal preprint of arXiv:1607.02586 (IEEE TPAMI, 2019). The first two authors contributed equally to this work. Project page: http://visualdynamics.csail.mit.ed

Generative Plug and Play: Posterior Sampling for Inverse Problems

Over the past decade, Plug-and-Play (PnP) has become a popular method for reconstructing images using a modular framework consisting of a forward and prior model. The great strength of PnP is that an image denoiser can be used as a prior model while the forward model can be implemented using more traditional physics-based approaches. However, a limitation of PnP is that it reconstructs only a single deterministic image. In this paper, we introduce Generative Plug-and-Play (GPnP), a generalization of PnP to sample from the posterior distribution. As with PnP, GPnP has a modular framework using a physics-based forward model and an image denoising prior model. However, in GPnP these models are extended to become proximal generators, which sample from associated distributions. GPnP applies these proximal generators in alternation to produce samples from the posterior. We present experimental simulations using the well-known BM3D denoiser. Our results demonstrate that the GPnP method is robust, easy to implement, and produces intuitively reasonable samples from the posterior for sparse interpolation and tomographic reconstruction. Code to accompany this paper is available at https://github.com/gbuzzard/generative-pnp-allerton .Comment: 8 pages, submitted to 2023 IEEE Allerton Conferenc

Effects of fallow replacement green manuring with annual legumes on soil nitrogen availability

Non-Peer ReviewedLegume green manures are used in crop rotations to enhance soil nitrogen availability and to provide ground cover for soil conservation. The annual legumes Black Lentil, Chickling Vetch, Tangier Flatpea, and a Feedpea were grown with tall stubble snow trapping in rotation with spring wheat to determine their suitability for fallow replacement in wheat production systems in the Brown soil zone. The field experiment, conducted at Swift Current Research Station from 1984 to 1991, was designed to assess the different legume types and green manure management practices (inoculation, incorporation, and desiccation). The RCB included fallow-wheat and continuous wheat as controls. Only continuous wheat received nitrogen (N) fertilizer. Soil N availability was measured by analyzing five soil segments down to 120 cm depth for exchangeable ammonium and nitrate up to four times each year. The topsoil was also analyzed by an incubation/leaching technique to determine the potential N mineralization after three cycles of the rotation in 1990. Soil nitrate showed large treatment related variation. Nitrate values for legume green manuring were consistently higher after incorporation than after desiccation. In all treatments, topsoil nitrate was high in spring and lowest in July. In fall and spring following fallow and green manuring nitrate was high for the fallow-wheat and the legume-wheat rotations and low for continuous wheat. The initial potential rate of nitrogen mineralization reveals a lower nitrogen supplying power of the topsoil for fallow wheat than for legume/wheat and for fertilized continuous wheat

Efficient Bayesian Computational Imaging with a Surrogate Score-Based Prior

We propose a surrogate function for efficient use of score-based priors for Bayesian inverse imaging. Recent work turned score-based diffusion models into probabilistic priors for solving ill-posed imaging problems by appealing to an ODE-based log-probability function. However, evaluating this function is computationally inefficient and inhibits posterior estimation of high-dimensional images. Our proposed surrogate prior is based on the evidence lower-bound of a score-based diffusion model. We demonstrate the surrogate prior on variational inference for efficient approximate posterior sampling of large images. Compared to the exact prior in previous work, our surrogate prior accelerates optimization of the variational image distribution by at least two orders of magnitude. We also find that our principled approach achieves higher-fidelity images than non-Bayesian baselines that involve hyperparameter-tuning at inference. Our work establishes a practical path forward for using score-based diffusion models as general-purpose priors for imaging

Estimating the Material Properties of Fabric from Video

Passively estimating the intrinsic material properties of deformable objects moving in a natural environment is essential for scene understanding. We present a framework to automatically analyze videos of fabrics moving under various unknown wind forces, and recover two key material properties of the fabric: stiffness and area weight. We extend features previously developed to compactly represent static image textures to describe video textures, such as fabric motion. A discriminatively trained regression model is then used to predict the physical properties of fabric from these features. The success of our model is demonstrated on a new, publicly available database of fabric videos with corresponding measured ground truth material properties. We show that our predictions are well correlated with ground truth measurements of stiffness and density for the fabrics. Our contributions include: (a) a database that can be used for training and testing algorithms for passively predicting fabric properties from video, (b) an algorithm for predicting the material properties of fabric from a video, and (c) a perceptual study of humans' ability to estimate the material properties of fabric from videos and images.National Science Foundation (U.S.) (CGV-1111415)National Science Foundation (U.S.) (CGV-1212928)National Science Foundation (U.S.). Graduate Research FellowshipMassachusetts Institute of Technology (Intelligent Initiative Postdoctoral Fellowship)United States. Intelligence Advanced Research Projects Activity (D10PC20023

Primary Production: Sensitivity to Surface Irradiance and Implications for Archiving Data

An equation is derived to express the sensitivity of daily, watercolumn production by phytoplankton in the ocean to variations in irradiance at the sea surface. Assuming no spectral effects, and a vertically uniform chlorophyll profile, the sensitivity is a function only of the dimensionless irradiance. Spectral effects can be accounted for as a function of the chlorophyll concentration. At the global scale, the relative reduction in daily production consequent on halving the surface irradiance (representing the expected scope for variation in surface irradiance under natural conditions) is found to be from 30 to 40%. Choice of data source for irradiance may incur a further systematic error of up to 15%. Given that local irradiance (the principal forcing for primary production) may vary from day to day, the issue of how to archive production data for the most generality is discussed and recommendations made in this regard