Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet

Multi-person pose understanding from RGB videos involves three complex tasks: pose estimation, tracking and motion forecasting. Intuitively, accurate multi-person pose estimation facilitates robust tracking, and robust tracking builds crucial history for correct motion forecasting. Most existing works either focus on a single task or employ multi-stage approaches to solving multiple tasks separately, which tends to make sub-optimal decision at each stage and also fail to exploit correlations among the three tasks. In this paper, we propose Snipper, a unified framework to perform multi-person 3D pose estimation, tracking, and motion forecasting simultaneously in a single stage. We propose an efficient yet powerful deformable attention mechanism to aggregate spatiotemporal information from the video snippet. Building upon this deformable attention, a video transformer is learned to encode the spatiotemporal features from the multi-frame snippet and to decode informative pose features for multi-person pose queries. Finally, these pose queries are regressed to predict multi-person pose trajectories and future motions in a single shot. In the experiments, we show the effectiveness of Snipper on three challenging public datasets where our generic model rivals specialized state-of-art baselines for pose estimation, tracking, and forecasting

Object Tracking in Video Using the TLD and CMT Fusion Model

Object tracking has been an attractive study topic in computer vision in recent years, thanks to the development of video monitoring systems. Tracking-Learning Detection (TLD), Compressive Tracking (CT), and Clustering of Static-Adaptive Correspondences for Deformable Object Tracking are some of the state-of-the-art methods for motion object tracking (CMT). We present a fusion model that combines TLD and CMT in this study. To restrict the calculation time of the CMT technique, the fusion TLD CMT model enhanced the TLD benefits of computation time and accuracy on t no deformable objects. The experimental results on the Vojir dataset for three techniques (TLD, CMT, and TLD CMT) demonstrated that our fusion proposal successfully trades off CMT accuracy for computing time

HYPERPROPERTIES-BASED OPTICAL FLOW-BASED AUTONOMOUS DRIVING SYSTEM

This study presents an autonomous driving system based on the principles of trace vectors derived from hyperproperty of a modified optical flowalgorithm. This technique allows keeping track of the past motion vectors by tracking the constraint sets to overcome the non-linear attributes ofthe deformable feature points and motion vectors. The results presented in this work exhibits stable tracking and multi-step prediction in a limitednumber of steps with less training vectors

Feature-based tracking of urethral motion in low-resolution trans-perineal ultrasound

This paper describes a novel algorithm for tracking the motion of the urethra from trans-perineal ultrasound. Our work is based on the structure-from-motion paradigm and therefore handles well structures with ill-defined and partially missing boundaries. The proposed approach is particularly well-suited for video sequences of low resolution and variable levels of blurriness introduced by anatomical motion of variable speed. Our tracking method identifies feature points on a frame by frame basis using the SURF detector/descriptor. Inter-frame correspondence is achieved using nearest-neighbor matching in the feature space. The motion is estimated using a non-linear bi-quadratic model, which adequately describes the deformable motion of the urethra. Experimental results are promising and show that our algorithm performs well when compared to manual tracking

Deformable Object Tracking with Gated Fusion

The tracking-by-detection framework receives growing attentions through the integration with the Convolutional Neural Networks (CNNs). Existing tracking-by-detection based methods, however, fail to track objects with severe appearance variations. This is because the traditional convolutional operation is performed on fixed grids, and thus may not be able to find the correct response while the object is changing pose or under varying environmental conditions. In this paper, we propose a deformable convolution layer to enrich the target appearance representations in the tracking-by-detection framework. We aim to capture the target appearance variations via deformable convolution, which adaptively enhances its original features. In addition, we also propose a gated fusion scheme to control how the variations captured by the deformable convolution affect the original appearance. The enriched feature representation through deformable convolution facilitates the discrimination of the CNN classifier on the target object and background. Extensive experiments on the standard benchmarks show that the proposed tracker performs favorably against state-of-the-art methods

A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild"

Recently, technologies such as face detection, facial landmark localisation and face recognition and verification have matured enough to provide effective and efficient solutions for imagery captured under arbitrary conditions (referred to as "in-the-wild"). This is partially attributed to the fact that comprehensive "in-the-wild" benchmarks have been developed for face detection, landmark localisation and recognition/verification. A very important technology that has not been thoroughly evaluated yet is deformable face tracking "in-the-wild". Until now, the performance has mainly been assessed qualitatively by visually assessing the result of a deformable face tracking technology on short videos. In this paper, we perform the first, to the best of our knowledge, thorough evaluation of state-of-the-art deformable face tracking pipelines using the recently introduced 300VW benchmark. We evaluate many different architectures focusing mainly on the task of on-line deformable face tracking. In particular, we compare the following general strategies: (a) generic face detection plus generic facial landmark localisation, (b) generic model free tracking plus generic facial landmark localisation, as well as (c) hybrid approaches using state-of-the-art face detection, model free tracking and facial landmark localisation technologies. Our evaluation reveals future avenues for further research on the topic.Comment: E. Antonakos and P. Snape contributed equally and have joint second authorshi

Automatic facial expression tracking for 4D range scans

This paper presents a fully automatic approach of spatio-temporal facial expression tracking for 4D range scans without any manual interventions (such as specifying landmarks). The approach consists of three steps: rigid registration, facial model reconstruction, and facial expression tracking. A Scaling Iterative Closest Points (SICP) algorithm is introduced to compute the optimal rigid registration between a template facial model and a range scan with consideration of the scale problem. A deformable model, physically based on thin shells, is proposed to faithfully reconstruct the facial surface and texture from that range data. And then the reconstructed facial model is used to track facial expressions presented in a sequence of range scans by the deformable model