7,610 research outputs found
Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames
This paper makes the first attempt to tackle the challenging task of
recovering arbitrary frame rate latent global shutter (GS) frames from two
consecutive rolling shutter (RS) frames, guided by the novel event camera data.
Although events possess high temporal resolution, beneficial for video frame
interpolation (VFI), a hurdle in tackling this task is the lack of paired GS
frames. Another challenge is that RS frames are susceptible to distortion when
capturing moving objects. To this end, we propose a novel self-supervised
framework that leverages events to guide RS frame correction and VFI in a
unified framework. Our key idea is to estimate the displacement field (DF)
non-linear dense 3D spatiotemporal information of all pixels during the
exposure time, allowing for the reciprocal reconstruction between RS and GS
frames as well as arbitrary frame rate VFI. Specifically, the displacement
field estimation (DFE) module is proposed to estimate the spatiotemporal motion
from events to correct the RS distortion and interpolate the GS frames in one
step. We then combine the input RS frames and DF to learn a mapping for
RS-to-GS frame interpolation. However, as the mapping is highly
under-constrained, we couple it with an inverse mapping (i.e., GS-to-RS) and RS
frame warping (i.e., RS-to-RS) for self-supervision. As there is a lack of
labeled datasets for evaluation, we generate two synthetic datasets and collect
a real-world dataset to train and test our method. Experimental results show
that our method yields comparable or better performance with prior supervised
methods.Comment: This paper has been submitted for review in March 202
Recommended from our members
Methods of Parallel Voxel Manipulation for 3D Digital Printing
A novel digital printing concept is explored for desktop fabrication of multimaterial objects with arbitrary 3D geometry. Digital objects are composed of
many discrete, self-aligning voxels instead of continuous (analog) deposition
techniques. Overall accuracy is determined by the individual voxels instead of the
printer, and digital properties such as perfect replication and error correction are
physically meaningful. The key challenge in digital printing is massively parallel,
deterministic voxel manipulation. To quickly print millions of voxels while
keeping errors low, we propose a parallel manufacturing process that exploits
electrostatic forces to place an entire 2D pattern of voxels concurrently. Using a
custom charged print head, we demonstrate selective 1.5mm voxel pick-up within
a larger, self-aligned layer. We expect the principle to scale to million voxel
layers using currently available technology.Mechanical Engineerin
Big data and the SP theory of intelligence
This article is about how the "SP theory of intelligence" and its realisation
in the "SP machine" may, with advantage, be applied to the management and
analysis of big data. The SP system -- introduced in the article and fully
described elsewhere -- may help to overcome the problem of variety in big data:
it has potential as "a universal framework for the representation and
processing of diverse kinds of knowledge" (UFK), helping to reduce the
diversity of formalisms and formats for knowledge and the different ways in
which they are processed. It has strengths in the unsupervised learning or
discovery of structure in data, in pattern recognition, in the parsing and
production of natural language, in several kinds of reasoning, and more. It
lends itself to the analysis of streaming data, helping to overcome the problem
of velocity in big data. Central in the workings of the system is lossless
compression of information: making big data smaller and reducing problems of
storage and management. There is potential for substantial economies in the
transmission of data, for big cuts in the use of energy in computing, for
faster processing, and for smaller and lighter computers. The system provides a
handle on the problem of veracity in big data, with potential to assist in the
management of errors and uncertainties in data. It lends itself to the
visualisation of knowledge structures and inferential processes. A
high-parallel, open-source version of the SP machine would provide a means for
researchers everywhere to explore what can be done with the system and to
create new versions of it.Comment: Accepted for publication in IEEE Acces
Iterative Prompt Learning for Unsupervised Backlit Image Enhancement
We propose a novel unsupervised backlit image enhancement method, abbreviated
as CLIP-LIT, by exploring the potential of Contrastive Language-Image
Pre-Training (CLIP) for pixel-level image enhancement. We show that the
open-world CLIP prior not only aids in distinguishing between backlit and
well-lit images, but also in perceiving heterogeneous regions with different
luminance, facilitating the optimization of the enhancement network. Unlike
high-level and image manipulation tasks, directly applying CLIP to enhancement
tasks is non-trivial, owing to the difficulty in finding accurate prompts. To
solve this issue, we devise a prompt learning framework that first learns an
initial prompt pair by constraining the text-image similarity between the
prompt (negative/positive sample) and the corresponding image (backlit
image/well-lit image) in the CLIP latent space. Then, we train the enhancement
network based on the text-image similarity between the enhanced result and the
initial prompt pair. To further improve the accuracy of the initial prompt
pair, we iteratively fine-tune the prompt learning framework to reduce the
distribution gaps between the backlit images, enhanced results, and well-lit
images via rank learning, boosting the enhancement performance. Our method
alternates between updating the prompt learning framework and enhancement
network until visually pleasing results are achieved. Extensive experiments
demonstrate that our method outperforms state-of-the-art methods in terms of
visual quality and generalization ability, without requiring any paired data.Comment: Accepted to ICCV 2023 as Oral. Project page:
https://zhexinliang.github.io/CLIP_LIT_page
- …