SODFormer: Streaming Object Detection with Transformer Using Events and Frames

DAVIS camera, streaming two complementary sensing modalities of asynchronous events and frames, has gradually been used to address major object detection challenges (e.g., fast motion blur and low-light). However, how to effectively leverage rich temporal cues and fuse two heterogeneous visual streams remains a challenging endeavor. To address this challenge, we propose a novel streaming object detector with Transformer, namely SODFormer, which first integrates events and frames to continuously detect objects in an asynchronous manner. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-DAVIS-SOD) over 1080.1k manual labels. Then, we design a spatiotemporal Transformer architecture to detect objects via an end-to-end sequence prediction problem, where the novel temporal Transformer module leverages rich temporal cues from two visual streams to improve the detection performance. Finally, an asynchronous attention-based fusion module is proposed to integrate two heterogeneous sensing modalities and take complementary advantages from each end, which can be queried at any time to locate objects and break through the limited output frequency from synchronized frame-based fusion strategies. The results show that the proposed SODFormer outperforms four state-of-the-art methods and our eight baselines by a significant margin. We also show that our unifying framework works well even in cases where the conventional frame-based camera fails, e.g., high-speed motion and low-light conditions. Our dataset and code can be available at https://github.com/dianzl/SODFormer.Comment: 18 pages, 15 figures, in IEEE Transactions on Pattern Analysis and Machine Intelligenc

Sub-action Prototype Learning for Point-level Weakly-supervised Temporal Action Localization

Point-level weakly-supervised temporal action localization (PWTAL) aims to localize actions with only a single timestamp annotation for each action instance. Existing methods tend to mine dense pseudo labels to alleviate the label sparsity, but overlook the potential sub-action temporal structures, resulting in inferior performance. To tackle this problem, we propose a novel sub-action prototype learning framework (SPL-Loc) which comprises Sub-action Prototype Clustering (SPC) and Ordered Prototype Alignment (OPA). SPC adaptively extracts representative sub-action prototypes which are capable to perceive the temporal scale and spatial content variation of action instances. OPA selects relevant prototypes to provide completeness clue for pseudo label generation by applying a temporal alignment loss. As a result, pseudo labels are derived from alignment results to improve action boundary prediction. Extensive experiments on three popular benchmarks demonstrate that the proposed SPL-Loc significantly outperforms existing SOTA PWTAL methods

Expression and functional analysis of a baculovirus gene encodinga truncated protein kinase homolog

AbstractAutographa californica nuclear polyhedrosis virus (AcMNPV) potentially encodes a 215-amino acid polypeptide containing6 out of 11 motifs conserved among eukaryotic protein kinases (Morris et al., Virology 200, 360–369, 1994). We examined the expression of this gene, named pk2, at the transcriptional and translational levels and the possible role of the gene during baculovirus replication in cell culture and insect larvae. Northern (RNA) blot analysis revealed that pk2 was transcribed primarily as an early 1.2-kb RNA. Western blot analysis showed that pk2 was expressed as a 25-kDa protein, PK2, which was present both early and late during virus infection. To examine the function(s) of pk2, we constructed a mutant baculovirus, vKINdel, in which one-third of the PK2-coding region was deleted and then compared the characteristics of vKINdel with wild-type AcMNPV and a marker-rescued revertant. The pk2 deletion mutation had no discernable effect on the number, size, or appearance of plaques, the kinetics of protein synthesis or protein phosphorylation profiles during virus infection of cultured SF-21 cells. Deletion of pk2 also had no significant influence on the infectivity or virulence of the baculovirus in larval bioassays and the level of occluded virus production was normal. Thus, pk2 does not appear to have a significant influence on virus replication in the host systems examined

Action recognition based on joint trajectory maps using convolutional neural networks

Recently, Convolutional Neural Networks (ConvNets) have shown promising performances in many computer vision tasks, especially image-based recognition. How to effectively use ConvNets for video-based recognition is still an open problem. In this paper, we propose a compact, effective yet simple method to encode spatiotemporal information carried in 3D skeleton sequences into multiple 2D images, referred to as Joint Trajectory Maps (JTM), and ConvNets are adopted to exploit the discriminative features for realtime human action recognition. The proposed method has been evaluated on three public benchmarks, i.e., MSRC-12 Kinect gesture dataset (MSRC-12), G3D dataset and UTD multimodal human action dataset (UTD-MHAD) and achieved the state-of-the-art results