SpatioTemporal LBP and shape feature for human activity representation and recognition

In this paper, we propose a histogram based feature to represent and recognize human action in video sequences. Motion History Image (MHI) merges a video sequence into a single image. However, in this method, we use Directional Motion History Image (DMHI) to create four directional spatiotemporal templates. We, then, extract the Local Binary Pattern (LBP) from those templates. Then, spatiotemporal LBP histograms are formed to represent the distribution of those patterns which makes the feature vector. We also use shape feature taken from three selective snippets and concatenate them with the LBP histograms. We measure the performance of the proposed representation method along with some variants of it by experimenting on the Weizmann action dataset. Higher recognition rates found in the experiment suggest that, compared to complex representation, the proposed simple and compact representation can achieve robust recognition of human activity for practical use

人の行動の表現と認識に関する研究

In recent years, analyzing human motion and recognizing a performed action from a video sequence has become very important and has been a well-researched topic in the field of computer vision. The reason behind such attention is its diverse applications in different domains like robotics, human computer interaction, video surveillance, controller-free gaming, video indexing, mixed or virtual reality, intelligent environments, etc. There are a number of researches performed on motion recognition in the last few decades. The state of the art action recognition schemes generally use a holistic or a body part based approach to represent actions. Most of the methods provide reasonable recognition results, but they are sometimes not suitable for online or real time systems because of their complexity in action representation. In this thesis, we address this issue by proposing a novel action representation scheme.The proposed action descriptor is based on a basic idea that rather than detecting the exact body parts or analyzing each action sequence, human action can be represented by a distribution of local texture patterns extracted from spatiotemporal templates. In this study, we use a novel way of generating those templates. Motion History Image (MHI) merges an action sequence into a single template. However, having the problem in overwriting old information by a new one in the MHI, we use a variant named Directional MHI (DMHI) to diffuse the action sequence into four directional templates. And then we use the Local Binary Pattern (LBP) operator, but with a unique way, a rotated bit arranged LBP, to extract the local texture patterns from those DMHI templates. These spatiotemporal patterns form the basis of our action descriptor which is formulated into a concatenated block histogram to serve as a feature vector for action recognition. However, the extracted patterns by LBP tends to lose the temporal information in a DMHI, therefore we take a linear combination of the motion history information and texture information to represent an action sequence. We also use some variants of the proposed action representation that include the shape or pose information of the action silhouettes as a form of histogram.We show that, by effective classification of such histograms, i.e., action descriptor, robust human action recognition is possible. We demonstrate the effectiveness of the proposed method along with some variants of the method over two benchmark dataset; the Weizmann dataset and KTH dataset. Our results are directly comparable or superior to the results reported over these datasets. Higher recognition rates found in the experiment suggest that, compared to complex representation, the proposed simple and compact representation can achieve robust recognition of human activity for practical use. Besides the recognition rate, due to the simplicity of the proposed technique, it is also advantageous with respect to computational load.九州工業大学博士学位論文 学位記番号:工博甲第409号 学位授与年月日:平成28年3月25日1.Introduction|2.Action Representation and Recognition|3.Experiments and Results|4.Conclusion九州工業大学平成27年

人の行動の表現と認識に関する研究

In recent years, analyzing human motion and recognizing a performed action from a video sequence has become very important and has been a well-researched topic in the field of computer vision. The reason behind such attention is its diverse applications in different domains like robotics, human computer interaction, video surveillance, controller-free gaming, video indexing, mixed or virtual reality, intelligent environments, etc. There are a number of researches performed on motion recognition in the last few decades. The state of the art action recognition schemes generally use a holistic or a body part based approach to represent actions. Most of the methods provide reasonable recognition results, but they are sometimes not suitable for online or real time systems because of their complexity in action representation. In this thesis, we address this issue by proposing a novel action representation scheme.The proposed action descriptor is based on a basic idea that rather than detecting the exact body parts or analyzing each action sequence, human action can be represented by a distribution of local texture patterns extracted from spatiotemporal templates. In this study, we use a novel way of generating those templates. Motion History Image (MHI) merges an action sequence into a single template. However, having the problem in overwriting old information by a new one in the MHI, we use a variant named Directional MHI (DMHI) to diffuse the action sequence into four directional templates. And then we use the Local Binary Pattern (LBP) operator, but with a unique way, a rotated bit arranged LBP, to extract the local texture patterns from those DMHI templates. These spatiotemporal patterns form the basis of our action descriptor which is formulated into a concatenated block histogram to serve as a feature vector for action recognition. However, the extracted patterns by LBP tends to lose the temporal information in a DMHI, therefore we take a linear combination of the motion history information and texture information to represent an action sequence. We also use some variants of the proposed action representation that include the shape or pose information of the action silhouettes as a form of histogram.We show that, by effective classification of such histograms, i.e., action descriptor, robust human action recognition is possible. We demonstrate the effectiveness of the proposed method along with some variants of the method over two benchmark dataset; the Weizmann dataset and KTH dataset. Our results are directly comparable or superior to the results reported over these datasets. Higher recognition rates found in the experiment suggest that, compared to complex representation, the proposed simple and compact representation can achieve robust recognition of human activity for practical use. Besides the recognition rate, due to the simplicity of the proposed technique, it is also advantageous with respect to computational load.九州工業大学博士学位論文 学位記番号:工博甲第409号 学位授与年月日:平成28年3月25日1.Introduction|2.Action Representation and Recognition|3.Experiments and Results|4.Conclusion九州工業大学平成27年

Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition

Open-set action recognition is to reject unknown human action cases which are out of the distribution of the training set. Existing methods mainly focus on learning better uncertainty scores but dismiss the importance of feature representations. We find that features with richer semantic diversity can significantly improve the open-set performance under the same uncertainty scores. In this paper, we begin with analyzing the feature representation behavior in the open-set action recognition (OSAR) problem based on the information bottleneck (IB) theory, and propose to enlarge the instance-specific (IS) and class-specific (CS) information contained in the feature for better performance. To this end, a novel Prototypical Similarity Learning (PSL) framework is proposed to keep the instance variance within the same class to retain more IS information. Besides, we notice that unknown samples sharing similar appearances to known samples are easily misclassified as known classes. To alleviate this issue, video shuffling is further introduced in our PSL to learn distinct temporal information between original and shuffled samples, which we find enlarges the CS information. Extensive experiments demonstrate that the proposed PSL can significantly boost both the open-set and closed-set performance and achieves state-of-the-art results on multiple benchmarks. Code is available at https://github.com/Jun-CEN/PSL.Comment: To appear at CVPR202