13 research outputs found
Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation
Unsupervised video object segmentation (VOS) is a task that aims to detect
the most salient object in a video without external guidance about the object.
To leverage the property that salient objects usually have distinctive
movements compared to the background, recent methods collaboratively use motion
cues extracted from optical flow maps with appearance cues extracted from RGB
images. However, as optical flow maps are usually very relevant to segmentation
masks, the network is easy to be learned overly dependent on the motion cues
during network training. As a result, such two-stream approaches are vulnerable
to confusing motion cues, making their prediction unstable. To relieve this
issue, we design a novel motion-as-option network by treating motion cues as
optional. During network training, RGB images are randomly provided to the
motion encoder instead of optical flow maps, to implicitly reduce motion
dependency of the network. As the learned motion encoder can deal with both RGB
images and optical flow maps, two different predictions can be generated
depending on which source information is used as motion input. In order to
fully exploit this property, we also propose an adaptive output selection
algorithm to adopt optimal prediction result at test time. Our proposed
approach affords state-of-the-art performance on all public benchmark datasets,
even maintaining real-time inference speed
Domain Alignment and Temporal Aggregation for Unsupervised Video Object Segmentation
Unsupervised video object segmentation aims at detecting and segmenting the
most salient object in videos. In recent times, two-stream approaches that
collaboratively leverage appearance cues and motion cues have attracted
extensive attention thanks to their powerful performance. However, there are
two limitations faced by those methods: 1) the domain gap between appearance
and motion information is not well considered; and 2) long-term temporal
coherence within a video sequence is not exploited. To overcome these
limitations, we propose a domain alignment module (DAM) and a temporal
aggregation module (TAM). DAM resolves the domain gap between two modalities by
forcing the values to be in the same range using a cross-correlation mechanism.
TAM captures long-term coherence by extracting and leveraging global cues of a
video. On public benchmark datasets, our proposed approach demonstrates its
effectiveness, outperforming all existing methods by a substantial margin
Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognition
Skeleton-based action recognition has attracted considerable attention due to
its compact skeletal structure of the human body. Many recent methods have
achieved remarkable performance using graph convolutional networks (GCNs) and
convolutional neural networks (CNNs), which extract spatial and temporal
features, respectively. Although spatial and temporal dependencies in the human
skeleton have been explored, spatio-temporal dependency is rarely considered.
In this paper, we propose the Inter-Frame Curve Network (IFC-Net) to
effectively leverage the spatio-temporal dependency of the human skeleton. Our
proposed network consists of two novel elements: 1) The Inter-Frame Curve (IFC)
module; and 2) Dilated Graph Convolution (D-GC). The IFC module increases the
spatio-temporal receptive field by identifying meaningful node connections
between every adjacent frame and generating spatio-temporal curves based on the
identified node connections. The D-GC allows the network to have a large
spatial receptive field, which specifically focuses on the spatial domain. The
kernels of D-GC are computed from the given adjacency matrices of the graph and
reflect large receptive field in a way similar to the dilated CNNs. Our IFC-Net
combines these two modules and achieves state-of-the-art performance on three
skeleton-based action recognition benchmarks: NTU-RGB+D 60, NTU-RGB+D 120, and
Northwestern-UCLA.Comment: 12 pages, 5 figure
Pixel-Level Equalized Matching for Video Object Segmentation
Feature similarity matching, which transfers the information of the reference
frame to the query frame, is a key component in semi-supervised video object
segmentation. If surjective matching is adopted, background distractors can
easily occur and degrade the performance. Bijective matching mechanisms try to
prevent this by restricting the amount of information being transferred to the
query frame, but have two limitations: 1) surjective matching cannot be fully
leveraged as it is transformed to bijective matching at test time; and 2)
test-time manual tuning is required for searching the optimal hyper-parameters.
To overcome these limitations while ensuring reliable information transfer, we
introduce an equalized matching mechanism. To prevent the reference frame
information from being overly referenced, the potential contribution to the
query frame is equalized by simply applying a softmax operation along with the
query. On public benchmark datasets, our proposed approach achieves a
comparable performance to state-of-the-art methods
Global-Local Aggregation with Deformable Point Sampling for Camouflaged Object Detection
The camouflaged object detection (COD) task aims to find and segment objects
that have a color or texture that is very similar to that of the background.
Despite the difficulties of the task, COD is attracting attention in medical,
lifesaving, and anti-military fields. To overcome the difficulties of COD, we
propose a novel global-local aggregation architecture with a deformable point
sampling method. Further, we propose a global-local aggregation transformer
that integrates an object's global information, background, and boundary local
information, which is important in COD tasks. The proposed transformer obtains
global information from feature channels and effectively extracts important
local information from the subdivided patch using the deformable point sampling
method. Accordingly, the model effectively integrates global and local
information for camouflaged objects and also shows that important boundary
information in COD can be efficiently utilized. Our method is evaluated on
three popular datasets and achieves state-of-the-art performance. We prove the
effectiveness of the proposed method through comparative experiments
Synchronizing Vision and Language: Bidirectional Token-Masking AutoEncoder for Referring Image Segmentation
Referring Image Segmentation (RIS) aims to segment target objects expressed
in natural language within a scene at the pixel level. Various recent RIS
models have achieved state-of-the-art performance by generating contextual
tokens to model multimodal features from pretrained encoders and effectively
fusing them using transformer-based cross-modal attention. While these methods
match language features with image features to effectively identify likely
target objects, they often struggle to correctly understand contextual
information in complex and ambiguous sentences and scenes. To address this
issue, we propose a novel bidirectional token-masking autoencoder (BTMAE)
inspired by the masked autoencoder (MAE). The proposed model learns the context
of image-to-language and language-to-image by reconstructing missing features
in both image and language features at the token level. In other words, this
approach involves mutually complementing across the features of images and
language, with a focus on enabling the network to understand interconnected
deep contextual information between the two modalities. This learning method
enhances the robustness of RIS performance in complex sentences and scenes. Our
BTMAE achieves state-of-the-art performance on three popular datasets, and we
demonstrate the effectiveness of the proposed method through various ablation
studies
Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report
The role of mobile cameras increased dramatically over the past few years,
leading to more and more research in automatic image quality enhancement and
RAW photo processing. In this Mobile AI challenge, the target was to develop an
efficient end-to-end AI-based image signal processing (ISP) pipeline replacing
the standard mobile ISPs that can run on modern smartphone GPUs using
TensorFlow Lite. The participants were provided with a large-scale Fujifilm
UltraISP dataset consisting of thousands of paired photos captured with a
normal mobile camera sensor and a professional 102MP medium-format FujiFilm
GFX100 camera. The runtime of the resulting models was evaluated on the
Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the
majority of common deep learning ops. The proposed solutions are compatible
with all recent mobile GPUs, being able to process Full HD photos in less than
20-50 milliseconds while achieving high fidelity results. A detailed
description of all models developed in this challenge is provided in this
paper
PyNET-QxQ: A Distilled PyNET for QxQ Bayer Pattern Demosaicing in CMOS Image Sensor
The deep learning-based ISP models for mobile cameras produce high-quality
images comparable to the professional DSLR camera. However, many of them are
computationally expensive, which may not be appropriate for mobile
environments. Also, the recent mobile cameras adopt non-Bayer CFAs (e.g., Quad
Bayer, Nona Bayer, and QxQ Bayer) to improve image quality; however, most deep
learning-based ISP models mainly focus on standard Bayer CFA. In this work, we
propose PyNET-QxQ based on PyNET, a light-weighted ISP explicitly designed for
the QxQ CFA pattern. The number of parameters of PyNET-QxQ is less than 2.5% of
PyNET. We also introduce a novel knowledge distillation technique, progressive
distillation, to train the compressed network effectively. Finally, experiments
with QxQ images (obtained by an actual QxQ camera sensor, under development)
demonstrate the outstanding performance of PyNET-QxQ despite significant
parameter reductions.Comment: in revie
An SGX-Based Key Management Framework for Data Centric Networking
© 2020, Springer Nature Switzerland AG.As the Internet has evolved from host-to-host communications to content distribution, data-centric networking platforms are gaining a momentum. Especially, as the cloud computing becomes the norm, there is a consensus that data is to be distributed over some potentially untrusted servers to which its publishers/subscribers are connected. While data-centric networking platforms have been an area of active research, there have been few studies on how to distribute and manage keys for data protection in such platforms with untrusted servers. We present a key management framework in which symmetric and asymmetric keys are securely managed. A writer publishes not only his (encrypted) data but also the symmetric key for the data. Likewise, a reader retrieves the symmetric key as well as the data of interest. To make the key distribution securely between a writer and a reader via an untrusted server, we introduce a key server running on top of the Intel SGX technology. In this way, we can manage and distribute keys for data protection in an efficient and flexible manner. We demonstrate that the prototype of the proposed framework is running with the negligible overhead.N