Efficient Human Facial Pose Estimation

Pose estimation has become an increasingly important area in computer vision and more specifically in human facial recognition and activity recognition for surveillance applications. Pose estimation is a process by which the rotation, pitch, or yaw of a human head is determined. Numerous methods already exist which can determine the angular change of a face, however, these methods vary in accuracy and their computational requirements tend to be too high for real-time applications. The objective of this thesis is to develop a method for pose estimation, which is computationally efficient, while still maintaining a reasonable degree of accuracy. In this thesis, a feature-based method is presented to determine the yaw angle of a human facial pose using a combination of artificial neural networks and template matching. The artificial neural networks are used for the feature detection portion of the algorithm along with skin detection and other image enhancement algorithms. The first head model, referred to as the Frontal Position Model, determines the pose of the face using two eyes and the mouth. The second model, referred to as the Side Position Model, is used when only one eye can be viewed and determines pose based on a single eye, the nose tip, and the mouth. The two models are presented to demonstrate the position change of facial features due to pose and to provide the means to determine the pose as these features change from the frontal position. The effectiveness of this pose estimation method is examined by looking at both the manual and automatic feature detection methods. Analysis is further performed on how errors in feature detection affect the resulting pose determination. The method resulted in the detection of facial pose from 30 to -30 degrees with an average error of 4.28 degrees for the Frontal Position Model and 5.79 degrees for the Side Position Model with correct feature detection. The Intel(R) Streaming SIMD Extensions (SSE) technology was employed to enhance the performance of floating point operations. The neural networks used in the feature detection process require a large amount of floating point calculations, due to the computation of the image data with weights and biases. With SSE optimization the algorithm becomes suitable for processing images in a real-time environment. The method is capable of determining features and estimating the pose at a rate of seven frames per second on a 1.8 GHz Pentium 4 computer

Infrared LEDs-based pose estimation with underground camera model for Boom-type roadheader in coal mining

Accurate and reliable pose estimation of boom-type roadheader is of great importance in order to maintain the efficiency of automatic coal mining. The stability and accuracy of conventional measurement methods are difficult to be guaranteed on account of vibration noise, magnetic disturbance, electrostatic interference and other factors in underground environment. In this paper a vision-based non-contact measurement method for cutting-head pose estimation is presented, which deploy a 16-point infrared LED target on the boom-type roadheader to tackle the low illumination, high dust and complicated background. By establishing monocular vision measurement system, the cutting-head pose is estimated through processing the LED target images obtained from an explosion-proof industrial camera mounted on the roadheader. After analyzing the measurement mechanism, an underground camera model based on the equivalent focal length is built to eliminate refraction errors caused by the two layer glasses for explosion-proof and dust removal glasses. Then the pose estimation processes, including infrared LEDs feature points extraction, spot center location, improved P4P method based on dual quaternions, are carried out. The influence factors of cutting-head pose estimation accuracy is further studied by modeling, and the error distribution of the main parameters is investigated and evaluated. Numerical simulation and experimental evaluation are designed to verify the performance of the proposed method. The results show that the pose estimation error is in line with the numerical prediction, achieving the requirements of cutting-head pose estimation in underground roadway construction in coal mine

2D 영상에서 시간 특징과 지역적 특징의 분석을 통한 사람 포즈 검출 및 추적

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2021. 2. 곽노준.2D human pose estimation and tracking aim to detect the location of a person's parts and their trajectory. A pose is composed of parts of a person, and a person's part is an element of the body such as arms, legs and head. Pose estimation technique is being utilized both industrially and academically. For example, in a home training system, pose detection can detect the user's pose and help the user correct the posture. Also, in human action recognition research, human pose information can be exploited as a helpful supplementary information. In order to apply human pose studies to real-world systems, the model is required to be of high performance and also light enough to run in a real-time manner. In this paper, we have focused on improving accuracy. We have considered how to utilize the feature values to achieve high accuracy using the spatial and temporal features. Spatial feature means characteristic values such as textures, patterns, and postures that can be extracted from images. We have made better use of the spatial feature by dividing it into local and global features. The global feature is likely to include a large number of parts, while the local feature focuses on a relatively small number of parts. First, we have proposed a structure that can use the global-local feature at the same time to improve the performance. The global network intensively learns the global feature, and the local network can learn various regional information from images. The local network performs as a function of refining the pose detected in the global network sequentially. To prove the efficiency of the proposed method, experiments have been conducted on the Leeds sports dataset (LSP) data, which is one of the single-person pose estimation datasets. Secondly, we define the rare pose using global feature and solve the imbalance in poses. First of all, the poses are classified using location information of the entire pose. Experiments have shown that the poses are distributed around certain poses (standing poses, upper body poses, etc.), and an imbalance between them apparently exists. We have proposed methods such as weighted loss, synthesizing rare pose data, etc. to resolve the imbalance. Experiments are conducted using MPII and COCO data, which are widely used in multi-person pose estimation. The temporal feature refers to the varying information of poses along the time. It is usually recommended to use time information when analyzing objects in a video. Therefore thirdly, we have estimated and tracked the poses with a map that expresses the change of a person's movement. The network learns the spatial and temporal maps together to create synergy between each other. The experiment has been conducted in multi-person pose tracking data, Posetrack 2017 and 2018. Even if the proposed three methods improve different issues, utilized together. For example, a new structure is a top-down approach and has parallel two deconvolutions for spatial (Heatmap) and temporal map (TML). Additionally, the rare pose data augmentation and the local network are applied to increase performance. Thus, adopting three methods is available to improve performance and more extensible in the pose estimation field.2D 이미지에서 사람의 포즈를 검출하는 연구는 사람의 파트들의 위치를 검출하는 것을 목표로한다. 포즈는 사람의 파트들로 구성되어 있고 사람의 파트는 팔, 다리, 머리 등으로 사람을 구성하는 신체의 요소들을 의미한다. 사람의 포즈 정보는 다양한 분야에서 활용 될 수 있다. 또한, 사람의 동작 감지 연구 분야에서는 사람의 포즈 정보가 매우 훌륭한 입력 특징 값으로 사용된다. 사람의 포즈 검출 연구를 실제 시스템에 적용하기 위해서는 높은 정확도, 실시간성, 다양한 기기에 사용 가능하도록 가벼운 모델이 필요하다. 본 논문에서는 정확도를 개선하는 연구에 초점을 맞췄다. 높은 정확도를 달성하기 위해서 특징값을 어떻게 활용할지에 대해 고민을 했으며, 지역적 특징값과 시간 특징값을 사용해서 문제를 개선했다 지역적 특징값은 사람의 텍스쳐, 형태와 같은 특징을 표현하는 것을 의미한다. 우리는 지역적 특징 값을 다수의 파트를 담고 있는 Global feature 와 소수의 파트를 담고 있는 Local feature로 분류해서 문제를 접근했다. 첫번째로는 global-local feature 을 동시에 사용해서 성능을 개선하는 연구에 집중했다. Global feature을 집중적으로 학습하는 네트워크와 다양한 형태의 local 정보를 학습 할 수 있는 local network을 설계했다. Local network에서는 global network에서 검출한 포즈를 다시 한번 개선하는 역할을 수행한다. 제안된 방법의 효율성을 증명하기 위해서 single-person pose estimation 데이터 중 하나인 Leeds sports dataset (LSP) 데이터에서 실험을 수행했다. 두번째로는 global한 정보를 통해 희귀한 포즈를 검출해서 포즈의 불균형을 해소해 성능을 개선하는 연구를 수행했다. 우선적으로 포즈 데이터 내에서 전체 포즈의 위치 정보를 사용해서 포즈들을 분류했다. 실험 결과, 일정 포즈를 (서있는 포즈, 상반신만 있는 포즈 등) 중심으로 포즈들이 분포 된며 포즈 간의 불균형이 있음을 밝혀냈다. 우리는 포즈 간의 불균형을 해소하기 위해 weight loss, generate rare pose data 등의 방법을 제안했다. 제안된 방법의 효율성을 증명하기 위해서 multi-person pose estimation 데이터에서 많이 사용되는 MPII와 COCO 데이터에서 실험을 수행했다. 시간 특징값은 시간 흐름에 따른 움직임 변화값을 의미한다. 동영상에서 객체를 분석하기 위해서는 시간 정보를 활용하는 것이 좋다. 그래서 세번째로 우리는 사람의 움직임 변화를 맵으로 표현해서 포즈를 추적했다. 이때 포즈의 지역적 특징값과 같이 학습해서 서로간의 시너지 효과를 낼 수 있도록 네트워크를 제안했다. 제안된 방법의 효율성을 증명하기 위해서 multi-person pose tracking 데이터인 posetrack 2017 과 2018에서 실험을 수행했다. 본 논문에서는 지역적 특징과 시간적 특징을 활용해서 포즈의 성능을 개선하는 방법들을 제안했다. 서로 다른 문제들을 해결했지만 나아가 하나로 묶여 문제를 해결 할 수 있다. 예를들어, top-down 형태의 네트워크 구조에서 Heatmap과 TML을 각각 학습 할 수 있는 평행적 구조의 decovolution network을 제안 할 수 있다. 여기에 Heatmap의 성능 개선을 위해 local network와 rare pose data augmentation 방식 또한 추가할 수 있다. 이렇게 제안된 방법을 결합해서 더 나은 포즈의 성능을 개선 할 수 있는 방법들이 제안 될 수 있다.Abstract i Contents iii List of Tables vi List of Figures viii 1. Introduction 1 1.1 Spatial feature 3 1.1.1 Global-local network 4 1.1.2 Exploring rare pose estimation using global pose information 5 1.2 Temporal feature 6 2. Related work 8 2.1 Single-person pose estimation 8 2.2 Multi-person pose estimation 10 2.3 Multi-person pose tracking 14 2.4 Datasets and measurements 15 2.4.1 Leeds Sports Pose (LSP) dataset 16 2.4.2 MPII dataset 16 2.4.3 COCO 2017 keypoint dataset 17 2.4.4 PoseTrack 19 3. Single-person pose estimation 22 3.1 Global-local Network 24 3.2 Experiments 28 4. Rare pose estimation 39 4.1 Identification of Rare Pose 39 4.2 Enhancing the performance of rare pose estimation 41 4.2.1 Duplication of Rare Pose Samples (DRP) 42 4.2.2 Addition of Synthetic Rare Pose data (ASRP) 43 4.2.3 Weighted Loss based on Cluster Distance (WLCD) 47 4.2.4 Divide and Conquer Strategy for pose estimation (DACP) 48 4.3 Experiments 48 4.3.1 Results of Rare Pose Identification 50 4.3.2 Results of Proposed Methods 57 5. Multi-person pose estimation and tracking 62 5.1 Temporal flow Maps for Limb movement (TML) 65 5.2 Multi-stride method 69 5.3 Inference 70 5.4 Experiments 74 6. Future work 81 Abstract (In Korean) 97 감사의 글 99Docto

Face Alignment Assisted by Head Pose Estimation

In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation. We first investigate the failure cases of most state of the art face alignment approaches and observe that these failures often share one common global property, i.e. the head pose variation is usually large. Inspired by this, we propose a deep convolutional network model for reliable and accurate head pose estimation. Instead of using a mean face shape, or randomly selected shapes for cascaded face alignment initialisation, we propose two schemes for generating initialisation: the first one relies on projecting a mean 3D face shape (represented by 3D facial landmarks) onto 2D image under the estimated head pose; the second one searches nearest neighbour shapes from the training set according to head pose distance. By doing so, the initialisation gets closer to the actual shape, which enhances the possibility of convergence and in turn improves the face alignment performance. We demonstrate the proposed method on the benchmark 300W dataset and show very competitive performance in both head pose estimation and face alignment.Comment: Accepted by BMVC201

Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regressions

Head-pose estimation has many applications, such as social event analysis, human-robot and human-computer interaction, driving assistance, and so forth. Head-pose estimation is challenging because it must cope with changing illumination conditions, variabilities in face orientation and in appearance, partial occlusions of facial landmarks, as well as bounding-box-to-face alignment errors. We propose tu use a mixture of linear regressions with partially-latent output. This regression method learns to map high-dimensional feature vectors (extracted from bounding boxes of faces) onto the joint space of head-pose angles and bounding-box shifts, such that they are robustly predicted in the presence of unobservable phenomena. We describe in detail the mapping method that combines the merits of unsupervised manifold learning techniques and of mixtures of regressions. We validate our method with three publicly available datasets and we thoroughly benchmark four variants of the proposed algorithm with several state-of-the-art head-pose estimation methods.Comment: 12 pages, 5 figures, 3 table

Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings

Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation

A Differential Approach for Gaze Estimation

Non-invasive gaze estimation methods usually regress gaze directions directly from a single face or eye image. However, due to important variabilities in eye shapes and inner eye structures amongst individuals, universal models obtain limited accuracies and their output usually exhibit high variance as well as biases which are subject dependent. Therefore, increasing accuracy is usually done through calibration, allowing gaze predictions for a subject to be mapped to his/her actual gaze. In this paper, we introduce a novel image differential method for gaze estimation. We propose to directly train a differential convolutional neural network to predict the gaze differences between two eye input images of the same subject. Then, given a set of subject specific calibration images, we can use the inferred differences to predict the gaze direction of a novel eye sample. The assumption is that by allowing the comparison between two eye images, annoyance factors (alignment, eyelid closing, illumination perturbations) which usually plague single image prediction methods can be much reduced, allowing better prediction altogether. Experiments on 3 public datasets validate our approach which constantly outperforms state-of-the-art methods even when using only one calibration sample or when the latter methods are followed by subject specific gaze adaptation.Comment: Extension to our paper A differential approach for gaze estimation with calibration (BMVC 2018) Submitted to PAMI on Aug. 7th, 2018 Accepted by PAMI short on Dec. 2019, in IEEE Transactions on Pattern Analysis and Machine Intelligenc