Current datasets for video-based person re-identification (re-ID) do not include structural knowledge in form of human pose annotations for the persons of interest. Nonetheless, pose information is very helpful to disentangle useful feature information from background or occlusion noise. Especially real-world scenarios, such as surveillance, contain a lot of occlusions in human crowds or by obstacles. On the other hand, video-based person re-ID can benefit other tasks such as multi-person pose tracking in terms of robust feature matching. For that reason, we present PoseTrackReID, a large-scale dataset for multi-person pose tracking and video-based person re-ID. With PoseTrackReID, we want to bridge the gap between person re-ID and multi-person pose tracking. Additionally, this dataset provides a good benchmark for current state-of-the-art methods on multi-frame person re-ID

Chen, D.

Doering, A.

Gall, J.

Schiele, B.

Zhang, S.

English

MPG.PuRe

arXiv:2011.06243v1  [cs.CV]  12 Nov 2020POSETRACKREID: DATASET DESCRIPTIONA PREPRINTAndreas Doering Di Chen Shanshan Zhang Bernt Schiele Juergen GallNovember 13, 2020ABSTRACTCurrent datasets for video-based person re-identification (re-ID) do not include structural knowledgein form of human pose annotations for the persons of interest. Nonetheless, pose information is veryhelpful to disentangle useful feature information from background or occlusion noise. Especiallyreal-world scenarios, such as surveillance, contain a lot of occlusions in human crowds or by obsta-cles. On the other hand, video-based person re-ID can benefit other tasks such as multi-person posetracking in terms of robust feature matching. For that reason, we present PoseTrackReID, a large-scale dataset for multi-person pose tracking and video-based person re-ID. With PoseTrackReID,we want to bridge the gap between person re-ID and multi-person pose tracking. Additionally, thisdataset provides a good benchmark for current state-of-the-art methods on multi-frame person re-ID.Keywords Dataset · Pose Estimation · Multi-Person Pose Tracking · Person re-identification1 IntroductionMulti-person pose tracking [1, 2, 3] and video-based person re-identification (re-ID) [4, 5, 6, 7] are two very activeresearch fields in computer vision, relevant for many applications like sports, autonomous driving and security. Bothtasks are challenging due to different viewpoints, varying (low) image resolutions, illumination changes, occlusions,motion blur and crowded scenes.In multi-person pose tracking, the target is to estimate all human poses for all frames of a video and link them overtime. Person re-identification, on the other hand, aims to find a person of interest across multiple sequences andnon-overlapping cameras. Although both tasks are mainly studied separately, they can benefit from each other. Forexample, person re-identification can benefit from human pose estimation by utilizing the semantic structure of theperson of interest [8]. Person re-ID, on the other hand, can be utilized to link all instances of the same person acrosstime [9].Common datasets for person re-ID [10, 11, 12, 13] provide tight person crops and mostly contain the person of interestonly. Though, in many real world scenarios such as surveillance or sports, persons are often occluded by obstacles.Further, these scenarios contain a lot of crowded scenes in which persons frequently occlude each other. This leads toa lot of ambiguities, especially if a person is partially visible or multiple persons are present within a single crop. Toovercome problems such as occlusion, Miao et al. [8] propose the Occluded-DukeMTMC dataset as an extension ofthe DukeMTMC-ReID dataset [12] and provide keypoint information obtained from an off-the-shelf pose estimator[14]. However, off-the-shelf pose estimators such as [14] perform poorly in crowded scenes with severe occlusions,resulting in misplaced keypoints.Video keypoint annotations, on the other hand, are costly and time consuming. Consequently, a limited amount oftraining data is publicly available, especially for the task of multi-person pose tracking. We argue that pose trackingand person re-ID are two related tasks which should be solved jointly. For that reason, we extended a large-scale multi-person pose estimation and tracking dataset (PoseTrack [15]) and propose PoseTrackReID. PoseTrackReID combinesboth tasks of multi-person pose tracking and video-based person re-ID with the aim to improve pose-based person re-ID methods in videos. To the best of our knowledge, this is the first dataset combining person re-ID and multi-personpose tracking.A PREPRINT - NOVEMBER 13, 20202 DatasetWe briefly describe the PoseTrack 2018 dataset [15] in Section 2.1 and introduce PoseTrackReID as well as its anno-tation process in Section 2.2.2.1 PoseTrackThe PoseTrack 2018 dataset1 [15] is a large-scale dataset for multi-person pose estimation and tracking, containing823 sequences. Following the split of the MPII Human Pose dataset [16], the sequences are split into 293, 170 and375 videos for training, validation and testing. The majority of sequences range from 41 to 151 frames and contain30 densely annotated frames around the middle of each sequence. On top, validation and test sequences are annotatedwith a step size of four frames. In total, PoseTrack 2018 consists of 46,933 labeled frames.PoseTrack provides ignore regions to exclude crowds and small people, keypoint annotations, head bounding boxes toestimate the scale of a person and track ids. Unfortunately, track ids are not unique throughout the dataset, not evenwithin a single sequence. If a person leaves a scene and re-enters, for instance, it is assigned a new track id. This canresult in ambiguities for appearance-based similarity approaches.2.2 PoseTrackReIDThe PoseTrack 2018 dataset contains multiple sequences sampled from the same video. For that reason, the datasetis well suited to be extended by person IDs as the same person appears in multiple sequences. In addition, PoseTrackcontains a lot of challenging sequences with a high degree of occlusion and motion blur. In contrast to related personre-ID datasets [10, 11, 12, 13], this results in person bounding boxes with a high degree of occlusion and multiplepersons within a single person crop. Due to its challenging nature, our proposed dataset can provide a new measureof robustness for current state-of-the-art approaches. The provided keypoint annotations further allow to incorporatestructural pose information paving the way for pose-based person re-ID methods. On the other hand, our dataset opensways for re-ID based pose tracking methods. .2.3 Data AnnotationWe developed a new tool to annotate the PoseTrack 2018 dataset and adopt the same annotation format. At the moment,we only provide annotations for the training and validation sets.The annotation process was performed in four steps. First, we re-annotated the existing bounding boxes on the trainingand validation set as the bounding boxes provided by PoseTrack have been an artifact of the annotation process andwere therefore not accurately annotated. Specifically, most bounding boxes only contained the visible part of a person.We extended all bounding boxes to cover the entire portion of a person, including occluded body parts. To increase thedifficulty of person re-ID, we further annotated bounding boxes for small persons and persons in crowds. Then, weinterpolated and revised the bounding boxes between non-keyframes2. This results in additional 25,478 and 11,238annotated frames on the training and validation sets, respectively. Second, we adapted the ignore regions on thetraining set such that the newly added persons are not anymore excluded. In a third step, we annotated unique personidentities throughout the training and validation sets. Additionally, we added head bounding boxes for newly addedpersons for all frames on the validation set. On the training set, we only provide head bounding boxes from PoseTrack2018.In a final step, we adapt and annotate person keypoints for all keyframes on the training and validation sets. Theoriginal annotations of the PoseTrack 2018 dataset consist of 15 keypoints where each keypoint contains a flag, whetherit is annotated or not. Unfortunately, there exist several cases in which these flags are not set reliably. In our dataset,we re-define the purpose of the keypoint flags and include occluded keypoints. In that way, pose estimation and re-ID approaches can utilize occlusion information within their training pipelines. We define a joint j = (x, y, v) asoccluded, if x ≥ 0, y ≥ 0 and v = 0. A joint is truncated if x = −1, y = −1 and v = 0. Otherwise, a joint is definedas visible if v = 1. After refining the original keypoints, we ran an off-the-shelf pose estimator [3] to estimate theposes for all newly added bounding boxes on keyframes only, which were then refined manually afterwards.1We refer to the final version of PoseTrack 2018 which was released in 2019, containing more sequences as described in [15].2We denote as keyframe all frames which were annotated in PoseTrack 2018.2A PREPRINT - NOVEMBER 13, 2020References[1] Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf. 15 keypoints is all you need. In CVPR, 2020.[2] Manchen Wang, Joseph Tighe, and Davide Modolo. Combining detection and tracking for human pose estimationin videos. In CVPR, 2020.[3] Umer Rafi, Andreas Doering, Bastian Leibe, and Juergen Gall. Self-supervised keypoint correspondences formulti-person pose estimation and tracking in videos. In ECCV, 2020.[4] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Temporal complementary learningfor video person re-identification. In ECCV, 2020.[5] Yichao Yan, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, and Ling Shao. Learning multi-granular hyper-graphs for video-based person re-identification. In CVPR, 2020.[6] Jinrui Yang, Wei-Shi Zheng, Qize Yang, Ying-Cong Chen, and Qi Tian. Spatial-temporal graph convolutionalnetwork for video-based person re-identification. In CVPR, 2020.[7] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Multi-granularity reference-aided attentive featureaggregation for video-based person re-identification. In CVPR, 2020.[8] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded personre-identification. In ICCV, 2019.[9] Rui Zhang, Zheng Zhu, Peng Li, Rui Wu, Chaoxu Guo, Guan Huang, and Hailun Xia. Exploiting offset-guidednetwork for pose estimation and tracking. In CVPRW, 2019.[10] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re-identification: A benchmark. InICCV, 2015.[11] Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang, and Qi Tian. Mars: A video bench-mark for large-scale person re-identification. In ECCV, 2016.[12] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In ICCV, 2017.[13] Yu Wu, Yutian Lin, Xuanyi Dong, Yan Yan, Wanli Ouyang, and Yi Yang. Exploit the unknown gradually: One-shot video-based person re-identification by stepwise learning. In The IEEE Conference on Computer Vision andPattern Recognition (CVPR), June 2018.[14] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using partaffinity fields. In CVPR, 2017.[15] M. Andriluka, U. Iqbal, E. Ensafutdinov, L. Pishchulin, A. Milan, J. Gall, and Schiele B. PoseTrack: A bench-mark for human pose estimation and tracking. In CVPR, 2018.[16] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: Newbenchmark and state of the art analysis. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2014.3

{PoseTrackReID}: {D}ataset Description

http://hdl.handle.net/21.11116/0000-0007-80FC-C

{PoseTrackReID}: {D}ataset Description

Abstract

Similar works

Full text

Available Versions

MPG.PuRe