일반화된 4차원 동작 특징을 이용한 시선각에 무관한 행동 인식

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2014. 8. 최진영.본 논문은 일반화된 4차원 [x,y,z,t] 동작 특징을 이용하여 시선각에 무관한 행동 및 행동 방향 인식 문제를 해결하는 것을 목적으로 수행되었다. 기존의 행동 인식 연구는 주로 카메라의 위치는 고정되어있고 사람들은 카메라를 바라보고 서있는 상황을 가정하였다. 그러나 실제 비디오나 감시카메라에 등장하는 사람들은 카메라를 의식하지 않고 자연스럽게 행동하기 때문에 제한된 조건, 환경에서 행동을 인식하는 것과 달리, 카메라의 위치와 사람의 방향에 따라서 다양한 시선각에서 영상이 촬영될 수 있다. 따라서 실제 어플리케이션에 적용하기 위해서는 무작위의 시선각에서 영상이 들어왔을 때 행동 인식을 하는 것이 필수적이며, 어떤 방향으로 행동하는 지 알 수 있다면 누구와 상호작용을 하는 지 아는데 도움을 줄 수 있다. 본 논문에서는 몇 개의 다른 시선각에서 찍힌 영상을 이용하여 3차원 [x,y,z] 입체를 복원하고, 연속된 3차원 입체에서 4차원 시공간 특징점을 구하는 방법을 제안하여 시선각에 무관한 행동 및 행동 방향 인식을 수행하였다. 3차원 입체 및 연속된 3차원 입체에서 구한 4차원 시공간 특징점은 모든 시선각에서의 정보를 갖고 있으므로, 원하는 시선각으로 사영을 하여 각 시선각에서의 특징을 얻을 수 있다. 사영된 실루엣과 4차원 시공간 특징점의 위치를 바탕으로 각각 움직이는 부분과 움직이지 않는 부분에 대한 정보를 포함하는 motion history images (MHIs)와 non motion history images (NMHIs) 를 만들어 행동 인식을 위한 특징으로 사용을 하였다. MHIs만으로는 행동 시 움직이는 부분이 비슷한 패턴을 보일 때 좋은 성능을 보장할 수 없고 따라서 행동 시 움직이지 않는 부분에 대한 정보를 줄 수 있는 NMHIs를 제안하였다. 행동 인식을 위한 학습 단계에서 MHIs와 NMHIs는 클래스를 고려한 차원 축소 알고리즘인 class-augmented principal component analysis (CA-PCA)를 통해서 차원이 축소되며, 이 때 행동 라벨을 이용하여 차원을 축소하므로 각 행동이 잘 분리가 되도록하는 principal axis를 찾을 수 있다. 차원이 축소된 MHIs와 NMHIs는 support vector data description (SVDD) 방법으로 학습되고, support vector domain density description (SVDDD)를 이용하여 인식된다. 행동 방향을 학습할때에는 각 행동에 대해 방향 라벨을 사용하여 principal axis를 구하며, 마찬가지로 SVDD로 학습을 하고 SVDDD를 이용하여 인식된다. 제안된 4차원 시공간 특징점은 시선각에 무관한 행동 및 행동 방향 인식에 사용될 수 있으며 실험을 통해 4차원 시공간 특징점이 각 행동의 특징을 압축적으로 잘 보여주고 있음을 보였다. 또한 실제 어플리케이션에서처럼 무작위의 시선각에서 영상이 들어왔을 경우를 가정하기 위하여 학습 데이터셋과 전혀 다른 새로운 인식 데이터셋을 구축하였다. 기존의 여러 시선각에서 촬영 된 IXMAS 행동 인식 데이터셋을 이용하여 학습을 하고, 학습 데이터셋과 다른 시선각에서 촬영한 SNU 데이터셋에서 인식 실험을 하여 제안한 알고리즘을 검증하였다. 실험 결과 제안한 방법은 학습을 위해 촬영한 영상에 포함되지 않는 시선각에서 테스트 영상이 들어왔을 경우에도 좋은 성능을 보이는 것을 확인하였다. 또한 5개의 방향으로 촬영된 SNU 데이터셋을 이용하여 행동 방향 인식 실험을 하였으며, 좋은 방향 인식률을 보이는 것을 확인하였다. 행동 방향 인식을 통해서 영상 내에서 여러 사람이 등장할 때 다른사람들과 어떻게 상호 작용을 하는지 정보를 알 수 있고, 이는 영상을 해석하는데 도움을 줄 수 있을 것으로 생각된다.In this thesis, we propose a method to recognize human action and their orientation independently of viewpoints using generalized 4D [x,y,z,t] motion features. The conventional action recognition methods assume that the camera view is fixed and people are standing towards the cameras. However, in real life scenarios, the cameras are installed at various positions for their purposes and the orientation of people are chosen arbitrarily. Therefore, the images can be taken with various views according to the position of camera and the orientation of people. To recognize human action and their orientation under this difficult scenario, we focus on the view invariant action recognition method which can recognize the test videos from any arbitrary view. For this purpose, we propose a method to recognize human action and their orientation independently of viewpoints by developing 4D space-time interest points (4D-STIPs, [x,y,z,t]) using 3D space (3D-S, [x,y,z]) volumes reconstructed from images of a finite number of different views. Since the 3D-S volumes and the 4D-STIPs are constructed using volumetric information, the features for arbitrary 2D space (2D-S, [x,y]) viewpoint can be generated by projecting the 3D-S volumes and 4D-STIPs on corresponding test image planes. With these projected features, we construct motion history images (MHIs) and non-motion history images (NMHIs) which encode the moving and non-moving parts of an action respectively. Since MHIs cannot guarantee a good performance when moving parts of an object show similar patterns, we propose NMHIs and combine it with MHIs to add the information from stationary parts of an object in the description of the particular action class. To reduce the dimension of MHIs and NMHIs, we apply class-augmented principal component analysis (CA-PCA) which uses class information for dimension reduction. Since we use the action label for reducing the dimension of features, we obtain the principal axis which can separate each action well. After reducing the feature dimension, the final features are trained by support vector data description method (SVDD) and tested by support vector domain density description (SVDDD). As for the recognition of action orientation, the features are reduced the dimension using orientation label. Similarly, the reduced features are trained by SVDD and tested by SVDDD. The proposed 4D-STIPs can be applied to view invariant recognition of action and their orientation, and we verify that they represent the properties of each action compactly in experiments. To assume arbitrary test view as in real applications, we develop a new testing dataset which is totally different from the training dataset. We verify our algorithm by training action models using the multi-view IXMAS dataset and testing using SNU dataset. Experimental results show that the proposed method is more generalized and outperforms the state-of-the-art methods, especially when training the classifier with the information insufficient about the test views. As for the recognition of action orientation, we experiment with SNU dataset taken from 5 different orientations to verify recognition performance. The recognition of action orientation can be helpful in analyzing the video by providing the information about interactions of people.1 Introduction 1 1.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Contents of the research . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.1 Generalized 4D motion features . . . . . . . . . . . . . . . . 10 1.2.2 View invariant action recognition . . . . . . . . . . . . . . . 11 1.2.3 Recognition of action orientation . . . . . . . . . . . . . . . 12 2 Generalized 4D Motion Features 14 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 Harris corner detector . . . . . . . . . . . . . . . . . . . . . 18 2.2.2 3D space-time interest points . . . . . . . . . . . . . . . . . 21 2.2.3 3D reconstruction . . . . . . . . . . . . . . . . . . . . . . . 23 2.3 Proposed method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3.1 Modified 3D space-time interest points . . . . . . . . . . . . 27 2.3.2 4D space-time interest points . . . . . . . . . . . . . . . . . 30 2.4 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3 View Invariant Action Recognition 40 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2.1 Motion history images . . . . . . . . . . . . . . . . . . . . . 45 3.2.2 Class-augmented principal component analysis . . . . . . . 47 3.2.3 Support vector data description . . . . . . . . . . . . . . . . 53 3.2.4 Support vector domain density description . . . . . . . . . . 56 3.3 Proposed method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.3.1 Silhouettes . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.3.2 Space-time interest points . . . . . . . . . . . . . . . . . . . 67 3.3.3 Motion history images and Non-motion history images . . . 69 3.3.4 Training and Testing . . . . . . . . . . . . . . . . . . . . . . 72 3.4 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4 Recognition of Action Orientation 87 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.2 Proposed method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.2.1 Training and Testing . . . . . . . . . . . . . . . . . . . . . . 93 4.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5 Conclusions 100 Bibliography 103 Abstract in Korean 113Docto

Multimodal Multipart Learning for Action Recognition in Depth Videos

The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart features are formulated into the learning framework via the proposed hierarchical mixed norm, to regularize the structured features of each part and to apply sparsity between them, in favor of a group feature selection. Our experimental results expose the effectiveness of the proposed learning method in which it outperforms other methods in all three tested datasets while saturating one of them by achieving perfect accuracy

Dual Projection and Selfduality in Three Dimensions

We discuss the notion of duality and selfduality in the context of the dual projection operation that creates an internal space of potentials. Contrary to the prevailing algebraic or group theoretical methods, this technique is applicable to both even and odd dimensions. The role of parity in the kernel of the Gauss law to determine the dimensional dependence is clarified. We derive the appropriate invariant actions, discuss the symmetry groups and their proper generators. In particular, the novel concept of duality symmetry and selfduality in Maxwell theory in (2+1) dimensions is analysed in details. The corresponding action is a 3D version of the familiar duality symmetric electromagnetic theory in 4D. Finally, the duality symmetric actions in the different dimensions constructed here manifest both the SO(2) and $Z_2$ symmetries, contrary to conventional results.Comment: 20 pages, late

Recognition of human activities and expressions in video sequences using shape context descriptor

The recognition of objects and classes of objects is of importance in the field of computer vision due to its applicability in areas such as video surveillance, medical imaging and retrieval of images and videos from large databases on the Internet. Effective recognition of object classes is still a challenge in vision; hence, there is much interest to improve the rate of recognition in order to keep up with the rising demands of the fields where these techniques are being applied. This thesis investigates the recognition of activities and expressions in video sequences using a new descriptor called the spatiotemporal shape context. The shape context is a well-known algorithm that describes the shape of an object based upon the mutual distribution of points in the contour of the object; however, it falls short when the distinctive property of an object is not just its shape but also its movement across frames in a video sequence. Since actions and expressions tend to have a motion component that enhances the capability of distinguishing them, the shape based information from the shape context proves insufficient. This thesis proposes new 3D and 4D spatiotemporal shape context descriptors that incorporate into the original shape context changes in motion across frames. Results of classification of actions and expressions demonstrate that the spatiotemporal shape context is better than the original shape context at enhancing recognition of classes in the activity and expression domains

View-invariant action recognition

Human action recognition is an important problem in computer vision. It has a wide range of applications in surveillance, human-computer interaction, augmented reality, video indexing, and retrieval. The varying pattern of spatio-temporal appearance generated by human action is key for identifying the performed action. We have seen a lot of research exploring this dynamics of spatio-temporal appearance for learning a visual representation of human actions. However, most of the research in action recognition is focused on some common viewpoints, and these approaches do not perform well when there is a change in viewpoint. Human actions are performed in a 3-dimensional environment and are projected to a 2-dimensional space when captured as a video from a given viewpoint. Therefore, an action will have a different spatio-temporal appearance from different viewpoints. The research in view-invariant action recognition addresses this problem and focuses on recognizing human actions from unseen viewpoints

DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences

3D Human Motion Indexing and Retrieval is an interesting problem due to the rise of several data-driven applications aimed at analyzing and/or re-utilizing 3D human skeletal data, such as data-driven animation, analysis of sports bio-mechanics, human surveillance etc. Spatio-temporal articulations of humans, noisy/missing data, different speeds of the same motion etc. make it challenging and several of the existing state of the art methods use hand-craft features along with optimization based or histogram based comparison in order to perform retrieval. Further, they demonstrate it only for very small datasets and few classes. We make a case for using a learned representation that should recognize the motion as well as enforce a discriminative ranking. To that end, we propose, a 3D human motion descriptor learned using a deep network. Our learned embedding is generalizable and applicable to real-world data - addressing the aforementioned challenges and further enables sub-motion searching in its embedding space using another network. Our model exploits the inter-class similarity using trajectory cues, and performs far superior in a self-supervised setting. State of the art results on all these fronts is shown on two large scale 3D human motion datasets - NTU RGB+D and HDM05.Comment: Under Review, Conferenc