159 research outputs found
Context-aware Synthesis for Video Frame Interpolation
Video frame interpolation algorithms typically estimate optical flow or its
variations and then use it to guide the synthesis of an intermediate frame
between two consecutive original frames. To handle challenges like occlusion,
bidirectional flow between the two input frames is often estimated and used to
warp and blend the input frames. However, how to effectively blend the two
warped frames still remains a challenging problem. This paper presents a
context-aware synthesis approach that warps not only the input frames but also
their pixel-wise contextual information and uses them to interpolate a
high-quality intermediate frame. Specifically, we first use a pre-trained
neural network to extract per-pixel contextual information for input frames. We
then employ a state-of-the-art optical flow algorithm to estimate bidirectional
flow between them and pre-warp both input frames and their context maps.
Finally, unlike common approaches that blend the pre-warped frames, our method
feeds them and their context maps to a video frame synthesis neural network to
produce the interpolated frame in a context-aware fashion. Our neural network
is fully convolutional and is trained end to end. Our experiments show that our
method can handle challenging scenarios such as occlusion and large motion and
outperforms representative state-of-the-art approaches.Comment: CVPR 2018, http://graphics.cs.pdx.edu/project/ctxsy
Video Frame Interpolation via Adaptive Separable Convolution
Standard video frame interpolation methods first estimate optical flow
between input frames and then synthesize an intermediate frame guided by
motion. Recent approaches merge these two steps into a single convolution
process by convolving input frames with spatially adaptive kernels that account
for motion and re-sampling simultaneously. These methods require large kernels
to handle large motion, which limits the number of pixels whose kernels can be
estimated at once due to the large memory demand. To address this problem, this
paper formulates frame interpolation as local separable convolution over input
frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D
kernels require significantly fewer parameters to be estimated. Our method
develops a deep fully convolutional neural network that takes two input frames
and estimates pairs of 1D kernels for all pixels simultaneously. Since our
method is able to estimate kernels and synthesizes the whole video frame at
once, it allows for the incorporation of perceptual loss to train the neural
network to produce visually pleasing frames. This deep neural network is
trained end-to-end using widely available video data without any human
annotation. Both qualitative and quantitative experiments show that our method
provides a practical solution to high-quality video frame interpolation.Comment: ICCV 2017, http://graphics.cs.pdx.edu/project/sepconv
Development of a bidentate lewis acid catalyzed inverse electron demand diels-alder reaction of 1,2-diazines for the synthesis of substituted arenes
The efficient use of resources, as well as the minimization of waste and production costs is now more import than ever. Therefore, the development of new catalytic synthetic methods is an integral part of the research in the field of Organic Chemistry. Herein presented is a novel and very successful principle of activation of 1,2-diazines trough a bidentate Lewis acid catalyst for the inverse electron demand Diels-Alder (IEDDA) reaction. The catalyst consists of a dihydroboranthrene core structure and its synthesis has been optimized to a convenient three step procedure. The new catalytic method was applied for the synthesis of a variety of different aromatic compounds, especially substituted naphthalenes. In addition, by the application of furans, domino processes have been discovered, were consecutively to the initial IEDDA reaction, depending on the furan substrate, either a [3,9] or a [1,9]-sigmatropic rearrangement occurred. In the case of the [3,9]-sigmatropic rearrangement the method was elaborated to produce substituted cyclopropanaphthalenes in excellent yields. Due to the unavailability of a general method for the synthesis of benzo-substituted phthalazines the range of applied 1,2-diazines was at first considerably reduced. This gap was filled with the development of a one-pot synthesis of substituted phthalazines and pyridazino-aromatics starting from commercially available aromatic aldehydes. Together with the catalytic IEDDA reaction the one-pot synthesis of 1,2-diazines comprised a novel two-step procedure to synthesize complex substituted naphthalenes and derivatives thereof. In order to put this powerful two step strategy into context, (±)-Naproxen, one of the most common non-steroidal anti-inflammatory drugs, was synthesized to demonstrate the utility of the methodology. The development of the catalytic methods as well as the mechanistic investigations were supported and corroborated by computational studies
Softmax Splatting for Video Frame Interpolation
Differentiable image sampling in the form of backward warping has seen broad
adoption in tasks like depth estimation and optical flow prediction. In
contrast, how to perform forward warping has seen less attention, partly due to
additional challenges such as resolving the conflict of mapping multiple pixels
to the same target location in a differentiable way. We propose softmax
splatting to address this paradigm shift and show its effectiveness on the
application of frame interpolation. Specifically, given two input frames, we
forward-warp the frames and their feature pyramid representations based on an
optical flow estimate using softmax splatting. In doing so, the softmax
splatting seamlessly handles cases where multiple source pixels map to the same
target location. We then use a synthesis network to predict the interpolation
result from the warped representations. Our softmax splatting allows us to not
only interpolate frames at an arbitrary time but also to fine tune the feature
pyramid and the optical flow. We show that our synthesis approach, empowered by
softmax splatting, achieves new state-of-the-art results for video frame
interpolation.Comment: CVPR 2020, http://sniklaus.com/softspla
Evidence Against Novelty-Gated Encoding in Serial Recall
Novelty-gated encoding is the assumption that events are encoded more strongly into memory when they are more novel in comparison to previously encoded events. It is a core assumption of the SOB model of serial recall (Farrell & Lewandowsky, 2002). We present three experiments testing some predictions from novelty-gated encoding. Experiment 1 shows that the probability of recalling the third item in a list correctly does not depend on whether it is preceded by phonologically similar or dissimilar items. Experiment 2 shows that in lists of items from three classes (nonwords, spatial locations, and abstract drawings) the probability of recalling an item does not depend on whether it is preceded by items from the same or another class. Experiment 3 used a complex-span paradigm varying the phonological similarity of words that are read aloud as distractors in between memory items. Contrary to a prediction from novelty-gated encoding, similar distractors did not impair memory more than dissimilar distractors. The results question the assumption of novelty-gated encoding in serial recall. We discuss alternative explanations for the phenomena that this assumption has previously helped to explain. The present evidence against novelty-gated encoding might point to boundary conditions for the role of prediction error in the acquisition of memories
Evidence against novelty-gated encoding in serial recall
Novelty-gated encoding is the assumption that events are encoded more strongly into memory when they are more novel in comparison to previously encoded events. It is a core assumption of the SOB model of serial recall (Farrell & Lewandowsky, 2002). We present three experiments testing some predictions from novelty-gated encoding. Experiment 1 shows that the probability of recalling the third item in a list correctly does not depend on whether it is preceded by phonologically similar or dissimilar items. Experiment 2 shows that in lists of items from three classes (nonwords, spatial locations, and abstract drawings) the probability of recalling an item does not depend on whether it is preceded by items from the same or another class. Experiment 3 used a complex-span paradigm varying the phonological similarity of words that are read aloud as distractors in between memory items. Contrary to a prediction from novelty-gated encoding, similar distractors did not impair memory more than dissimilar distractors. The results question the assumption of novelty-gated encoding in serial recall. We discuss alternative explanations for the phenomena that this assumption has previously helped to explain. The present evidence against novelty-gated encoding might point to boundary conditions for the role of prediction error in the acquisition of memories
Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement
In this work, we first propose a fully differentiable Many-to-Many (M2M)
splatting framework to interpolate frames efficiently. Given a frame pair, we
estimate multiple bidirectional flows to directly forward warp the pixels to
the desired time step before fusing overlapping pixels. In doing so, each
source pixel renders multiple target pixels and each target pixel can be
synthesized from a larger area of visual context, establishing a many-to-many
splatting scheme with robustness to undesirable artifacts. For each input frame
pair, M2M has a minuscule computational overhead when interpolating an
arbitrary number of in-between frames, hence achieving fast multi-frame
interpolation. However, directly warping and fusing pixels in the intensity
domain is sensitive to the quality of motion estimation and may suffer from
less effective representation capacity. To improve interpolation accuracy, we
further extend an M2M++ framework by introducing a flexible Spatial Selective
Refinement (SSR) component, which allows for trading computational efficiency
for interpolation quality and vice versa. Instead of refining the entire
interpolated frame, SSR only processes difficult regions selected under the
guidance of an estimated error map, thereby avoiding redundant computation.
Evaluation on multiple benchmark datasets shows that our method is able to
improve the efficiency while maintaining competitive video interpolation
quality, and it can be adjusted to use more or less compute as needed.Comment: T-PAMI. arXiv admin note: substantial text overlap with
arXiv:2204.0351
- …