Super-Resolution Approaches for Depth Video Enhancement

Sensing using 3D technologies has seen a revolution in the past years where cost-effective depth sensors are today part of accessible consumer electronics. Their ability in directly capturing depth videos in real-time has opened tremendous possibilities for multiple applications in computer vision. These sensors, however, have major shortcomings due to their high noise contamination, including missing and jagged measurements, and their low spatial resolutions. In order to extract detailed 3D features from this type of data, a dedicated data enhancement is required. We propose a generic depth multi–frame super–resolution framework that addresses the limitations of state-of-theart depth enhancement approaches. The proposed framework doesnot need any additional hardware or coupling with different modalities. It is based on a new data model that uses densely upsampled low resolution observations. This results in a robust median initial estimation, further refined by a deblurring operation using a bilateraltotal variation as the regularization term. The upsampling operation ensures a systematic improvement in the registration accuracy. This is explored in different scenarios based on the motions involved in the depth video. For the general and most challenging case of objects deforming non-rigidly in full 3D, we propose a recursive dynamic multi–frame super-resolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per–pixel tracking where both depth measurements and deformations are optimized. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time

Towards Generalization of 3D Human Pose Estimation In The Wild

In this paper, we propose 3DBodyTex.Pose, a dataset that addresses the task of 3D human pose estimation in-the-wild. Generalization to in-the-wild images remains limited due to the lack of adequate datasets. Existent ones are usually collected in indoor controlled environments where motion capture systems are used to obtain the 3D ground-truth annotations of humans. 3DBodyTex.Pose offers high quality and rich data containing 405 different real subjects in various clothing and poses, and 81k image samples with ground-truth 2D and 3D pose annotations. These images are generated from 200 viewpoints among which 70 challenging extreme viewpoints. This data was created starting from high resolution textured 3D body scans and by incorporating various realistic backgrounds. Retraining a state-of-the-art 3D pose estimation approach using data augmented with 3DBodyTex.Pose showed promising improvement in the overall performance, and a sensible decrease in the per joint position error when testing on challenging viewpoints. The 3DBodyTex.Pose is expected to offer the research community with new possibilities for generalizing 3D pose estimation from monocular in-the-wild images

Bilateral Filter Evaluation Based on Exponential Kernels

The well-known bilateral filter is used to smooth noisy images while keeping their edges. This filter is commonly used with Gaussian kernel functions without real justification. The choice of the kernel functions has a major effect on the filter behavior. We propose to use exponential kernels with L1 distances instead of Gaussian ones. We derive Stein's Unbiased Risk Estimate to find the optimal parameters of the new filter and compare its performance with the conventional one. We show that this new choice of the kernels has a comparable smoothing effect but with sharper edges due to the faster, smoothly decaying kernels

Leveraging Equivariant Features for Absolute Pose Regression

While end-to-end approaches have achieved state-of-the-art performance in many perception tasks, they are not yet able to compete with 3D geometry-based methods in pose estimation. Moreover, absolute pose regression has been shown to be more related to image retrieval. As a result, we hypothesize that the statistical features learned by classical Convolutional Neural Networks do not carry enough geometric information to reliably solve this inherently geometric task. In this paper, we demonstrate how a translation and rotation equivariant Convolutional Neural Network directly induces representations of camera motions into the feature space. We then show that this geometric property allows for implicitly augmenting the training data under a whole group of image plane-preserving transformations. Therefore, we argue that directly learning equivariant features is preferable than learning data-intensive intermediate representations. Comprehensive experimental validation demonstrates that our lightweight model outperforms existing ones on standard datasets.Comment: 11 pages, 8 figures, CVPR202

Temporal 3D Human Pose Estimation for Action Recognition from Arbitrary Viewpoints

This work presents a new view-invariant action recognition system that is able to classify human actions by using a single RGB camera, including challenging camera viewpoints. Understanding actions from different viewpoints remains an extremely challenging problem, due to depth ambiguities, occlusion, and a large variety of appearances and scenes. Moreover, using only the information from the 2D perspective gives different interpretations for the same action seen from different viewpoints. Our system operates in two subsequent stages. The first stage estimates the 2D human pose using a convolution neural network. In the next stage, the 2D human poses are lifted to 3D human poses, using a temporal convolution neural network that enforces the temporal coherence over the estimated 3D poses. The estimated 3D poses from different viewpoints are then aligned to the same camera reference frame. Finally, we propose to use a temporal convolution network-based classifier for cross-view action recognition. Our results show that we can achieve state of art view-invariant action recognition accuracy even for the challenging viewpoints by only using RGB videos, without pre-training on synthetic or motion capture data

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

We propose a novel approach for enhancing depth videos containing non-rigidly deforming objects. Depth sensors are capable of capturing depth maps in real-time but suffer from high noise levels and low spatial resolutions. While solutions for reconstructing 3D details in static scenes, or scenes with rigid global motions have been recently proposed, handling unconstrained non-rigid deformations in relative complex scenes remains a challenge. Our solution consists in a recursive dynamic multi-frame superresolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per-pixel tracking where both depth measurements and deformations are dynamically optimized. The geometric smoothness is subsequently added using a multi-level L1 minimization with a bilateral total variation regularization. The performance of this method is thoroughly evaluated on both real and synthetic data. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time

Real-Time Non-Rigid Multi-Frame Depth Video Super-Resolution

This paper proposes to enhance low resolution dynamic depth videos containing freely non–rigidly moving objects with a new dynamic multi–frame super–resolution algorithm. Existent methods are either limited to rigid objects, or restricted to global lateral motions discarding radial displacements. We address these shortcomings by accounting for non–rigid displacements in 3D. In addition to 2D optical flow, we estimate the depth displacement, and simultaneously correct the depth measurement by Kalman filtering. This concept is incorporated efficiently in a multi–frame super–resolution framework. It is formulated in a recursive manner that ensures an efficient deployment in real–time. Results show the overall improved performance of the proposed method as compared to alternative approaches, and specifically in handling relatively large 3D motions. Test examples range from a full moving human body to a highly dynamic facial video with varying expressions

Serological and molecular characterization of Syrian Tomato spotted wilt virus isolates

Thirty four Syrian isolates of Tomato spotted wilt virus (TSWV) collected from tomato and pepper were tested against five specific monoclonal antibodies using TAS-ELISA. The isolates were in two serogroups. Fourteen tomato and sixteen pepper isolates were similar in their reaction with MAb-2, MAb-4, MAb-5 and MAb-6, but did not react with MAb-7 (Serogroup 1). Meanwhile, four isolates collected from pepper reacted with all the MAbs used (Serogroup 2). The expected 620 bp DNA fragment was obtained by RT-PCR from six samples using a specific primer pair designed to amplify the nucleocapsid protein (NP) gene of TSWV. The PCR products were sequenced and a phylogenetic tree was constructed. Sequence analysis revealed that the Syrian TSWV isolates were very similar at the nucleotide (97.74 to 99.84% identity) and amino acid (96.17 to 99.03% identity) sequences levels. The phylogenetic tree showed high similarity of Syrian TSWV isolates with many other representative isolates from different countries

Leveraging Equivariant Features for Absolute Pose Regression

Pose estimation enables vision-based systems to refer to their environment, supporting activities ranging from scene navigation to object manipulation. However, end-to-end approaches, that have achieved state-of-the-art performance in many perception tasks, are still unable to compete with 3D geometry-based methods in pose estimation. Indeed, absolute pose regression has been proven to be more related to image retrieval than to 3D structure. Our assumption is that statistical features learned by classical convolutional neural networks do not carry enough geometrical information for reliably solving this task. This paper studies the use of deep equivariant features for end-to-end pose regression. We further propose a translation and rotation equivariant Convolutional Neural Network whose architecture directly induces representations of camera motions into the feature space. In the context of absolute pose regression, this geometric property allows for implicitly augmenting the training data under a whole group of image plane-preserving transformations. Therefore, directly learning equivariant features efficiently compensates for learning intermediate representations that are indirectly equivariant yet data-intensive. Extensive experimental validation demonstrates that our lightweight model outperforms existing ones on standard datasets

Slow turning reveals enormous quadrupolar interactions (STREAQI)

We introduce a new solid-state NMR method, which uses very slow sample rotation to visualize NMR spectra whose width exceeds feasible spectrometer bandwidths. It is based on the idea that if we reorient a tensor by a known angle about a known axis, the shifts in the NMR frequencies observed across the spectral width allow us to reconstruct the entire tensor. Called STREAQI (Slow Turning Reveals Enormous Anisotropic Quadrupolar Interactions), this method allows us to probe NMR nuclei that are intractable to current methods. To prove the concept and demonstrate its promise we have implemented the method for several 79Br containing samples with quadrupolar coupling constants in the range of 10–50 MHz