MonoPerfCap: Human Performance Capture from Monocular Video

We present the first marker-less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium-scale non-rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non-rigid deformations, even from multi-view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch-based pose estimation strategy. Joint recovery of per-batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium-scale non-rigid alignment. We demonstrate state-of-the-art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.Comment: Accepted to ACM TOG 2018, to be presented on SIGGRAPH 201

A multimedia package for patient understanding and rehabilitation of non-contact anterior cruciate ligament injuries

Non-contact anterior cruciate ligament (ACL) injury is one of the most common ligament injuries in the body. Many patients’ receive graft surgery to repair the damage, but have to undertake an extensive period of rehabilitation. However, non-compliance and lack of understanding of the injury, healing process and rehabilitation means patient’s return to activities before effective structural integrity of the graft has been reached. When clinicians educate the patient, to encourage compliance with treatment and rehabilitation, the only tools that are currently widely in use are static plastic models, line diagrams and pamphlets. As modern technology grows in use in anatomical education, we have developed a unique educational and training package for patient’s to use in gaining a better understanding of their injury and treatment plan. We have combined cadaveric dissections of the knee (and captured with high resolution digital images) with reconstructed 3D modules from the Visible Human dataset, computer generated animations, and images to produce a multimedia package, which can be used to educate the patient in their knee anatomy, the injury, the healing process and their rehabilitation, and how this links into key stages of improving graft integrity. It is hoped that this will improve patient compliance with their rehabilitation programme, and better long-term prognosis in returning to normal or near-normal activities. Feedback from healthcare professionals about this package has been positive and encouraging for its long-term use

Gait recognition and understanding based on hierarchical temporal memory using 3D gait semantic folding

Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness