3D Face Tracking and Texture Fusion in the Wild

We present a fully automatic approach to real-time 3D face reconstruction from monocular in-the-wild videos. With the use of a cascaded-regressor based face tracking and a 3D Morphable Face Model shape fitting, we obtain a semi-dense 3D face shape. We further use the texture information from multiple frames to build a holistic 3D face representation from the video frames. Our system is able to capture facial expressions and does not require any person-specific training. We demonstrate the robustness of our approach on the challenging 300 Videos in the Wild (300-VW) dataset. Our real-time fitting framework is available as an open source library at http://4dface.org

Fast Compressive 3D Single-pixel Imaging

In this work, we demonstrate a modified photometric stereo system with perfect pixel registration, capable of reconstructing continuous real-time 3D video at ~8 Hz for 64 x 64 image resolution by employing evolutionary compressed sensing

Real-time Spatial Detection and Tracking of Resources in a Construction Environment

Construction accidents with heavy equipment and bad decision making can be based on poor knowledge of the site environment and in both cases may lead to work interruptions and costly delays. Supporting the construction environment with real-time generated three-dimensional (3D) models can help preventing accidents as well as support management by modeling infrastructure assets in 3D. Such models can be integrated in the path planning of construction equipment operations for obstacle avoidance or in a 4D model that simulates construction processes. Detecting and guiding resources, such as personnel, machines and materials in and to the right place on time requires methods and technologies supplying information in real-time. This paper presents research in real-time 3D laser scanning and modeling using high range frame update rate scanning technology. Existing and emerging sensors and techniques in three-dimensional modeling are explained. The presented research successfully developed computational models and algorithms for the real-time detection, tracking, and three-dimensional modeling of static and dynamic construction resources, such as workforce, machines, equipment, and materials based on a 3D video range camera. In particular, the proposed algorithm for rapidly modeling three-dimensional scenes is explained. Laboratory and outdoor field experiments that were conducted to validate the algorithm’s performance and results are discussed

LiveCap: Real-time Human Performance Capture from Monocular Video

We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting non-linear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques, while being orders of magnitude faster

Algorithms for Fast Computing of the 3D-DCT Transform

The algorithm for video compression based on the Three-Dimensional Discrete Cosine Transform (3D-DCT) is presented. The original algorithm of the 3D-DCT has high time complexity. We propose several enhancements to the original algorithm and make the calculation of the DCT algorithm feasible for future real-time video compression

A study of user perceptions of the relationship between bump-mapped and non-bump-mapped materials, and lighting intensity in a real-time virtual environment

The video and computer games industry has taken full advantage of the human sense of vision by producing games that utilize complex high-resolution textures and materials, and lighting technique. This results to the creation of an almost life-like real-time 3D virtual environment that can immerse the end-users. One of the visual techniques used is real-time display of bump-mapped materials. However, this sense of visual phenomenon has yet to be fully utilized for 3D design visualization in the architecture and construction domain. Virtual environments developed in the architecture and construction domain are often basic and use low-resolution images, which under represent the real physical environment. Such virtual environment is seen as being non-realistic to the user resulting in a misconception of the actual potential of it as a tool for 3D design visualization. A study was conducted to evaluate whether subjects can see the difference between bump-mapped and nonbump-mapped materials in different lighting conditions. The study utilized a real-time 3D virtual environment that was created using a custom-developed software application tool called BuildITC4. BuildITC4 was developed based upon the C4Engine which is classified as a next-generation 3D Game Engine. A total of thirty-five subjects were exposed to the virtual environment and were asked to compare the various types of material in different lighting conditions. The number of lights activated, the lighting intensity, and the materials used in the virtual environment were all interactive and changeable in real-time. The goal is to study how subjects perceived bump-mapped and non-bump mapped materials, and how different lighting conditions affect realistic representation. Results from this study indicate that subjects could tell the difference between the bump-mapped and non-bump mapped materials, and how different material reacts to different lighting condition

Linear-time Online Action Detection From 3D Skeletal Data Using Bags of Gesturelets

Sliding window is one direct way to extend a successful recognition system to handle the more challenging detection problem. While action recognition decides only whether or not an action is present in a pre-segmented video sequence, action detection identifies the time interval where the action occurred in an unsegmented video stream. Sliding window approaches for action detection can however be slow as they maximize a classifier score over all possible sub-intervals. Even though new schemes utilize dynamic programming to speed up the search for the optimal sub-interval, they require offline processing on the whole video sequence. In this paper, we propose a novel approach for online action detection based on 3D skeleton sequences extracted from depth data. It identifies the sub-interval with the maximum classifier score in linear time. Furthermore, it is invariant to temporal scale variations and is suitable for real-time applications with low latency

Co-operative surveillance cameras for high quality face acquisition in a real-time door monitoring system

The increasing number of CCTV cameras in use poses a problem of information overloading for end users. Smart technologies are used in video surveillance to automatically analyze and detect events of interest in real-time, through 2D and 3D video processing techniques called video analytics. This paper presents a smart surveillance stereo vision system for real-time intelligent door access monitoring. The system uses two IP cameras in a stereo configuration and a pan-tilt-zoom (PTZ) camera, to obtain real-time localised, high quality images of any triggering events