SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation

Human-centric video frame interpolation has great potential for improving people's entertainment experiences and finding commercial applications in the sports analysis industry, e.g., synthesizing slow-motion videos. Although there are multiple benchmark datasets available in the community, none of them is dedicated for human-centric scenarios. To bridge this gap, we introduce SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video frames of high-resolution ($\geq$720p) slow-motion sports videos crawled from YouTube. We re-train several state-of-the-art methods on our benchmark, and the results show a decrease in their accuracy compared to other datasets. It highlights the difficulty of our benchmark and suggests that it poses significant challenges even for the best-performing methods, as human bodies are highly deformable and occlusions are frequent in sports videos. To improve the accuracy, we introduce two loss terms considering the human-aware priors, where we add auxiliary supervision to panoptic segmentation and human keypoints detection, respectively. The loss terms are model agnostic and can be easily plugged into any video frame interpolation approaches. Experimental results validate the effectiveness of our proposed loss terms, leading to consistent performance improvement over 5 existing models, which establish strong baseline models on our benchmark. The dataset and code can be found at: https://neu-vi.github.io/SportsSlomo/.Comment: Project Page: https://neu-vi.github.io/SportsSlomo

Deceptive-NeRF: Enhancing NeRF Reconstruction using Pseudo-Observations from Diffusion Models

This paper introduces Deceptive-NeRF, a new method for enhancing the quality of reconstructed NeRF models using synthetically generated pseudo-observations, capable of handling sparse input and removing floater artifacts. Our proposed method involves three key steps: 1) reconstruct a coarse NeRF model from sparse inputs; 2) generate pseudo-observations based on the coarse model; 3) refine the NeRF model using pseudo-observations to produce a high-quality reconstruction. To generate photo-realistic pseudo-observations that faithfully preserve the identity of the reconstructed scene while remaining consistent with the sparse inputs, we develop a rectification latent diffusion model that generates images conditional on a coarse RGB image and depth map, which are derived from the coarse NeRF and latent text embedding from input images. Extensive experiments show that our method is effective and can generate perceptually high-quality NeRF even with very sparse inputs

Revisiting Event-based Video Frame Interpolation

Dynamic vision sensors or event cameras provide rich complementary information for video frame interpolation. Existing state-of-the-art methods follow the paradigm of combining both synthesis-based and warping networks. However, few of those methods fully respect the intrinsic characteristics of events streams. Given that event cameras only encode intensity changes and polarity rather than color intensities, estimating optical flow from events is arguably more difficult than from RGB information. We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy. Moreover, in light of the quasi-continuous nature of the time signals provided by event cameras, we propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages rather than in a single, long stage. Extensive experiments on both synthetic and real-world datasets show that these modifications lead to more reliable and realistic intermediate frame results than previous video frame interpolation methods. Our findings underline that a careful consideration of event characteristics such as high temporal density and elevated noise benefits interpolation accuracy.Comment: Accepted by IROS2023 Project Site: https://jiabenchen.github.io/revisit_even

RNA Interference against ATP as a Gene Therapy Approach for Prostate Cancer

Chemotherapeutic agents targeting energy metabolism have not achieved satisfactory results in different types of tumors. Herein, we developed an RNA interference (RNAi) method against adenosine triphosphate (ATP) by constructing an interfering plasmid-expressing ATP-binding RNA aptamer, which notably inhibited the growth of prostate cancer cells through diminishing the availability of cytoplasmic ATP and impairing the homeostasis of energy metabolism, and both glycolysis and oxidative phosphorylation were suppressed after RNAi treatment. Further identifying the mechanism underlying the effects of ATP aptamer, we surprisingly found that it markedly reduced the activity of membrane ionic channels and membrane potential which led to the dysfunction of mitochondria, such as the decrease of mitochondrial number, reduction in the respiration rate, and decline of mitochondrial membrane potential and ATP production. Meanwhile, the shortage of ATP impeded the formation of lamellipodia that are essential for the movement of cells, consequently resulting in a significant reduction of cell migration. Both the downregulation of the phosphorylation of AMP-activated protein kinase (AMPK) and endoplasmic reticulum kinase (ERK) and diminishing of lamellipodium formation led to cell apoptosis as well as the inhibition of angiogenesis and invasion. In conclusion, as the first RNAi modality targeting the blocking of ATP consumption, the present method can disturb the respiratory chain and ATP pool, which provides a novel regime for tumor therapies.