HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera

Continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale. Recently, implicit neural representation (INR) has been applied to video restoration, representing videos as implicit fields that can be decoded at an arbitrary scale. However, the highly ill-posed nature of C-STVSR limits the effectiveness of current INR-based methods: they assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary spatiotemporal positions with two consecutive frames. This restrains C-STVSR from capturing rapid and nonlinear motion and long-term dependencies (involving more than two frames) in complex dynamic scenes. In this paper, we propose a novel C-STVSR framework, called HR-INR, which captures both holistic dependencies and regional motions based on INR. It is assisted by an event camera, a novel sensor renowned for its high temporal resolution and low latency. To fully utilize the rich temporal information from events, we design a feature extraction consisting of (1) a regional event feature extractor - taking events as inputs via the proposed event temporal pyramid representation to capture the regional nonlinear motion and (2) a holistic event-frame feature extractor for long-term dependence and continuity motion. We then propose a novel INR-based decoder with spatiotemporal embeddings to capture long-term dependencies with a larger temporal perception field. We validate the effectiveness and generalization of our method on four datasets (both simulated and real data), showing the superiority of our method.Comment: 30 pages, 20 figures, 8 tables. This work was submitted for review in the second half of 2023. Project page: https://github.com/yunfanLu/HR-IN

Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution

Event cameras sense the intensity changes asynchronously and produce event streams with high dynamic range and low latency. This has inspired research endeavors utilizing events to guide the challenging video superresolution (VSR) task. In this paper, we make the first attempt to address a novel problem of achieving VSR at random scales by taking advantages of the high temporal resolution property of events. This is hampered by the difficulties of representing the spatial-temporal information of events when guiding VSR. To this end, we propose a novel framework that incorporates the spatial-temporal interpolation of events to VSR in a unified framework. Our key idea is to learn implicit neural representations from queried spatial-temporal coordinates and features from both RGB frames and events. Our method contains three parts. Specifically, the Spatial-Temporal Fusion (STF) module first learns the 3D features from events and RGB frames. Then, the Temporal Filter (TF) module unlocks more explicit motion information from the events near the queried timestamp and generates the 2D features. Lastly, the SpatialTemporal Implicit Representation (STIR) module recovers the SR frame in arbitrary resolutions from the outputs of these two modules. In addition, we collect a real-world dataset with spatially aligned events and RGB frames. Extensive experiments show that our method significantly surpasses the prior-arts and achieves VSR with random scales, e.g., 6.5. Code and dataset are available at https: //vlis2022.github.io/cvpr23/egvsr.Comment: Accepted by CVPR202

Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations

Reconstructing intensity frames from event data while maintaining high temporal resolution and dynamic range is crucial for bridging the gap between event-based and frame-based computer vision. Previous approaches have depended on supervised learning on synthetic data, which lacks interpretability and risk over-fitting to the setting of the event simulator. Recently, self-supervised learning (SSL) based methods, which primarily utilize per-frame optical flow to estimate intensity via photometric constancy, has been actively investigated. However, they are vulnerable to errors in the case of inaccurate optical flow. This paper proposes a novel SSL event-to-video reconstruction approach, dubbed EvINR, which eliminates the need for labeled data or optical flow estimation. Our core idea is to reconstruct intensity frames by directly addressing the event generation model, essentially a partial differential equation (PDE) that describes how events are generated based on the time-varying brightness signals. Specifically, we utilize an implicit neural representation (INR), which takes in spatiotemporal coordinate $(x, y, t)$ and predicts intensity values, to represent the solution of the event generation equation. The INR, parameterized as a fully-connected Multi-layer Perceptron (MLP), can be optimized with its temporal derivatives supervised by events. To make EvINR feasible for online requisites, we propose several acceleration techniques that substantially expedite the training process. Comprehensive experiments demonstrate that our EvINR surpasses previous SSL methods by 38% w.r.t. Mean Squared Error (MSE) and is comparable or superior to SoTA supervised methods. Project page: https://vlislab22.github.io/EvINR/

Energy Optimization in Multi-UAV-Assisted Edge Data Collection System

In the IoT (Internet of Things) system, the introduction of UAV (Unmanned Aerial Vehicle) as a new data collection platform can solve the problem that IoT devices are unable to transmit data over long distances due to the limitation of their battery energy. However, the unreasonable distribution of UAVs will still lead to the problem of the high total energy consumption of the system. In this work, to deal with the problem, a deployment model of a mobile edge computing (MEC) system based on multi-UAV is proposed. The goal of the model is to minimize the energy consumption of the system in the process of data transmission by optimizing the deployment of UAVs. The DEVIPSK (differential evolution algorithm with variable population size based on a mutation strategy pool initialized by K-Means) is proposed to solve the model. In DEVIPSK, the population is initialized by K-Means to obtain better initial positions of UAVs. Besides, considering the limitation of the fixed mutation strategy in the traditional evolutionary algorithm, a mutation strategy pool is used to update the positions of UAVs. The experimental results show the superiority of the DEVIPSK and provide guidance for the deployment of UAVs in the field of edge data collection in the IoT system

Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in addressing math reasoning problems. In particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the \textit{Code Usage Frequency} of GPT-4 Code Interpreter. We found that its success can be largely attributed to its powerful skills in generating and executing code, evaluating the output of code execution, and rectifying its solution when receiving unreasonable outputs. Based on this insight, we propose a novel and effective prompting method, explicit \uline{c}ode-based \uline{s}elf-\uline{v}erification~(CSV), to further boost the mathematical reasoning potential of GPT-4 Code Interpreter. This method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. In instances where the verification state registers as ``False'', the model shall automatically amend its solution, analogous to our approach of rectifying errors during a mathematics examination. Furthermore, we recognize that the states of the verification result indicate the confidence of a solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy on MATH dataset \textbf{(53.9\% $\to$ 84.3\%)}.Comment: Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificatio

Herb-Drug Interaction: Effects of Relinqing® Granule on the Pharmacokinetics of Ciprofloxacin, Sulfamethoxazole, and Trimethoprim in Rats

Relinqing granule (RLQ) is the best-selling Chinese patent drug for treatment of urinary system diseases. In this study, the effects of RLQ on the pharmacokinetics of ciprofloxacin, sulfamethoxazole, and trimethoprim in SD rats were investigated. Rats were randomly divided into control group 1, control group 2, RLQ group 1, and RLQ group 2. RLQ group 1 and RLQ group 2 were treated orally with RLQ for 7 days, and rats were treated with the same volume of water in control group 1 and control group 2. Then, RLQ group 1 and control group 1 were given intragastrically ciprofloxacin on day 8, while RLQ group 2 and control group 2 were given intragastrically sulfamethoxazole and trimethoprim on day 8. Blood samples were collected and determined. There was no significant influence of pharmacokinetic parameters of trimethoprim on two groups. But some pharmacokinetic parameters of ciprofloxacin and sulfamethoxazole in RLQ pretreated rats were evidently altered (P < 0.05), which indicated that absorption of ciprofloxacin and sulfamethoxazole in RLQ pretreated rats was significantly affected. It indicated the coadministration of RLQ would have an influence on the efficacy of ciprofloxacin and sulfamethoxazole, and the doses of ciprofloxacin tablet and compound sulfamethoxazole tablet need adjustment

Quantity and quality of memory and guesses in pattern recall of complex soccer scenarios

In soccer, players need to keep in mind the locations of the ball and numerous teammates and opponents in complex scenarios. This study aimed to examine both the quantity and quality of players’ memory and guess in pattern recall for complex soccer scenarios. We developed an adapted pattern recall task in which participants were required to reproduce player locations in complex scenarios through two stages, allowing for the separation of memory from guesses and the subsequent analysis of accurate memory and false memory. The results showed that soccer athletes could maintain the locations of about eight players, along with the ball, in accurate memory for complex soccer scenarios, showing a greater quantity of accurate memory than non-athletes, whereas the precision of memory did not differ between the two groups. On the other hand, non-athletes falsely memorised the locations of a larger number of attacking players and tended to make more guesses about the locations of defending players than soccer athletes. Moreover, soccer athletes displayed superior precision when guessing player locations compared to non-athletes. These findings suggest that soccer athletes leverage their knowledge to enhance the chunking of individual players to be maintained in short-term memory and facilitate the inferences about player locations exceeding the limit of short-term memory, providing them with an advantage in both memory quantity and guess quality in complex soccer scenarios.</p