Context-aware Synthesis for Video Frame Interpolation

Video frame interpolation algorithms typically estimate optical flow or its variations and then use it to guide the synthesis of an intermediate frame between two consecutive original frames. To handle challenges like occlusion, bidirectional flow between the two input frames is often estimated and used to warp and blend the input frames. However, how to effectively blend the two warped frames still remains a challenging problem. This paper presents a context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame. Specifically, we first use a pre-trained neural network to extract per-pixel contextual information for input frames. We then employ a state-of-the-art optical flow algorithm to estimate bidirectional flow between them and pre-warp both input frames and their context maps. Finally, unlike common approaches that blend the pre-warped frames, our method feeds them and their context maps to a video frame synthesis neural network to produce the interpolated frame in a context-aware fashion. Our neural network is fully convolutional and is trained end to end. Our experiments show that our method can handle challenging scenarios such as occlusion and large motion and outperforms representative state-of-the-art approaches.Comment: CVPR 2018, http://graphics.cs.pdx.edu/project/ctxsy

Video Frame Interpolation via Adaptive Separable Convolution

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.Comment: ICCV 2017, http://graphics.cs.pdx.edu/project/sepconv

Development of a bidentate lewis acid catalyzed inverse electron demand diels-alder reaction of 1,2-diazines for the synthesis of substituted arenes

The efficient use of resources, as well as the minimization of waste and production costs is now more import than ever. Therefore, the development of new catalytic synthetic methods is an integral part of the research in the field of Organic Chemistry. Herein presented is a novel and very successful principle of activation of 1,2-diazines trough a bidentate Lewis acid catalyst for the inverse electron demand Diels-Alder (IEDDA) reaction. The catalyst consists of a dihydroboranthrene core structure and its synthesis has been optimized to a convenient three step procedure. The new catalytic method was applied for the synthesis of a variety of different aromatic compounds, especially substituted naphthalenes. In addition, by the application of furans, domino processes have been discovered, were consecutively to the initial IEDDA reaction, depending on the furan substrate, either a [3,9] or a [1,9]-sigmatropic rearrangement occurred. In the case of the [3,9]-sigmatropic rearrangement the method was elaborated to produce substituted cyclopropanaphthalenes in excellent yields. Due to the unavailability of a general method for the synthesis of benzo-substituted phthalazines the range of applied 1,2-diazines was at first considerably reduced. This gap was filled with the development of a one-pot synthesis of substituted phthalazines and pyridazino-aromatics starting from commercially available aromatic aldehydes. Together with the catalytic IEDDA reaction the one-pot synthesis of 1,2-diazines comprised a novel two-step procedure to synthesize complex substituted naphthalenes and derivatives thereof. In order to put this powerful two step strategy into context, (±)-Naproxen, one of the most common non-steroidal anti-inflammatory drugs, was synthesized to demonstrate the utility of the methodology. The development of the catalytic methods as well as the mechanistic investigations were supported and corroborated by computational studies

Softmax Splatting for Video Frame Interpolation

Differentiable image sampling in the form of backward warping has seen broad adoption in tasks like depth estimation and optical flow prediction. In contrast, how to perform forward warping has seen less attention, partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way. We propose softmax splatting to address this paradigm shift and show its effectiveness on the application of frame interpolation. Specifically, given two input frames, we forward-warp the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. We then use a synthesis network to predict the interpolation result from the warped representations. Our softmax splatting allows us to not only interpolate frames at an arbitrary time but also to fine tune the feature pyramid and the optical flow. We show that our synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.Comment: CVPR 2020, http://sniklaus.com/softspla

Evidence Against Novelty-Gated Encoding in Serial Recall

Novelty-gated encoding is the assumption that events are encoded more strongly into memory when they are more novel in comparison to previously encoded events. It is a core assumption of the SOB model of serial recall (Farrell & Lewandowsky, 2002). We present three experiments testing some predictions from novelty-gated encoding. Experiment 1 shows that the probability of recalling the third item in a list correctly does not depend on whether it is preceded by phonologically similar or dissimilar items. Experiment 2 shows that in lists of items from three classes (nonwords, spatial locations, and abstract drawings) the probability of recalling an item does not depend on whether it is preceded by items from the same or another class. Experiment 3 used a complex-span paradigm varying the phonological similarity of words that are read aloud as distractors in between memory items. Contrary to a prediction from novelty-gated encoding, similar distractors did not impair memory more than dissimilar distractors. The results question the assumption of novelty-gated encoding in serial recall. We discuss alternative explanations for the phenomena that this assumption has previously helped to explain. The present evidence against novelty-gated encoding might point to boundary conditions for the role of prediction error in the acquisition of memories

Evidence against novelty-gated encoding in serial recall

Novelty-gated encoding is the assumption that events are encoded more strongly into memory when they are more novel in comparison to previously encoded events. It is a core assumption of the SOB model of serial recall (Farrell & Lewandowsky, 2002). We present three experiments testing some predictions from novelty-gated encoding. Experiment 1 shows that the probability of recalling the third item in a list correctly does not depend on whether it is preceded by phonologically similar or dissimilar items. Experiment 2 shows that in lists of items from three classes (nonwords, spatial locations, and abstract drawings) the probability of recalling an item does not depend on whether it is preceded by items from the same or another class. Experiment 3 used a complex-span paradigm varying the phonological similarity of words that are read aloud as distractors in between memory items. Contrary to a prediction from novelty-gated encoding, similar distractors did not impair memory more than dissimilar distractors. The results question the assumption of novelty-gated encoding in serial recall. We discuss alternative explanations for the phenomena that this assumption has previously helped to explain. The present evidence against novelty-gated encoding might point to boundary conditions for the role of prediction error in the acquisition of memories

Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement

In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Given a frame pair, we estimate multiple bidirectional flows to directly forward warp the pixels to the desired time step before fusing overlapping pixels. In doing so, each source pixel renders multiple target pixels and each target pixel can be synthesized from a larger area of visual context, establishing a many-to-many splatting scheme with robustness to undesirable artifacts. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames, hence achieving fast multi-frame interpolation. However, directly warping and fusing pixels in the intensity domain is sensitive to the quality of motion estimation and may suffer from less effective representation capacity. To improve interpolation accuracy, we further extend an M2M++ framework by introducing a flexible Spatial Selective Refinement (SSR) component, which allows for trading computational efficiency for interpolation quality and vice versa. Instead of refining the entire interpolated frame, SSR only processes difficult regions selected under the guidance of an estimated error map, thereby avoiding redundant computation. Evaluation on multiple benchmark datasets shows that our method is able to improve the efficiency while maintaining competitive video interpolation quality, and it can be adjusted to use more or less compute as needed.Comment: T-PAMI. arXiv admin note: substantial text overlap with arXiv:2204.0351