This paper has been presented at : 9th International Conference on Pattern Recognition Systems (ICPRS 2018)Computer vision systems have become increasingly popular, being used to solve a wide range of problems. In this paper, a computer vision algorithm with a support vector machine (SVM) classifier is presented. The work focuses on the recognition of human actions through computer vision, using a multi-camera dataset of human actions called MuHAVi. The algorithm uses a method to extract features, based on silhouettes. The challenge is that in MuHAVi these silhouettes are noisy and in many cases include shadows. As there are many actions that need to be recognised, we take a multiclass classification ap-proach that combines binary SVM classifiers. The results are compared with previous results on the same dataset and show a significant improvement, especially for recognising actions on a different view, obtaining overall accuracy of 85.5% and of 93.5% for leave-one-camera-out and leave-one-actor-out tests respectively.Sergio A Velastin has received funding from the Universidad Carlos III de Madrid, the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 600371, el Ministerio de Economía, Industria y Competitividad (COFUND2013-51509) el Ministerio de Educación, cultura y Deporte (CEI-15-17) and Banco Santander

Acuña, Gonzalo

González, Luis

Velastin Carroza, Sergio Alejandro

English

Crossref

Silhouette-based human action recognition with a multi-class support vector machine

Acuña Leiva, Gonzalo

Universidad Carlos III de Madrid e-Archivo

This is a postprint version of the following published document:González, L., Velastin, S.A y Acuña, G. (2018). Silhouette-based human action recognition with a multi-class support vector machine. In 9th International Conference on Pattern Recognition Systems (ICPRS 2018). DOI: https://doi.org/10.1049/cp.2018.1290© 2018 The Institution of Engineering and Technology.Silhouette-based human action recognition with a multi-class supportvector machineLuis Gonza´lez*, Sergio A Velastin+, Gonzalo Acun˜a**Universidad de Santiago de Chile, Santiago, Chile+University Carlos III Madrid, Spain, Cortexica Vision Systems Ltd.and Queen Mary University of London, UKKeywords: Bag of key poses, MuHAVi, SVM, Computer vi-sion, Human Action RecognitionAbstractComputer vision systems have become increasingly popular,being used to solve a wide range of problems. In this pa-per, a computer vision algorithm with a support vector ma-chine (SVM) classifier is presented. The work focuses on therecognition of human actions through computer vision, using amulti-camera dataset of human actions called MuHAVi. The al-gorithm uses a method to extract features, based on silhouettes.The challenge is that in MuHAVi these silhouettes are noisy andin many cases include shadows. As there are many actions thatneed to be recognised, we take a multiclass classification ap-proach that combines binary SVM classifiers. The results arecompared with previous results on the same dataset and showa significant improvement, especially for recognising actionson a different view, obtaining overall accuracy of 85.5% and of93.5% for leave-one-camera-out and leave-one-actor-out testsrespectively.1 IntroductionAgeing statistics both in developing and developed countriesindicate a tendency of increases in average age and, conse-quently, a growth of the elderly population. Given also thetrends on family “nuclearisation” and the reductions in ex-tended family support, many of the elderly also prefer to liveindependently, when possible, so technologies that enable as-sisted living become desirable. In particular, the use of auto-mated video monitoring has the potential to increase levels ofsafety and security while maintaining acceptable privacy by re-ducing human observation, limiting it only to cases of emergen-cies. Furthermore, added to technologies such as the Internetof Things, smartphones and global connectivity, the promise ofubiquitous intelligent homes becomes foreseeable.Some countries such as Spain are already working on this(e.g. the AAL Joint Program), having 600 million euros be-tween 2008 and 2013, being the first major investment in re-search in this line. There is also a framework of the EuropeanUnion (EU) called “Horizon 2020”, which points to the devel-opment of technology that allows assisted monitoring in intelli-gent environments to better support personnel in care facilitiesfor the elderly. Central to those efforts is to emulate the abilityof human beings of understanding what a person is doing (ontheir own, with others and with objects in their environment).In the context of computer vision, this has been called “HumanAction Recognition” (HAR). It is important to point out thatthis ability is not only useful in the context of assisted living butalso in many other areas such as public space surveillance, re-tail services, media handling (e.g. to automatically summarisevideo material), entertainment and so on. The last decade hasseen many advances, but there is still significant progress tobe made before solutions can be used in real environments.The use of standard bench marking datasets such as MuHAViis part of the effort to make progress. The investigation of thestate of the art on the use of this dataset,revealed the need tocompare the features proposed by [1], but using an SVM clas-sifier, versus the method proposed by [2] to have identify thebetter method in future works. Therefore, this paper evaluatesthe computer vision algorithm first proposed in [1], originallytested with the MuHAVi-MAS [3] dataset, a small dataset ofhuman actions where all silhouettes have been manually de-fined and therefore they are as noise-free as possible. In thispaper we use a more realistic set of data known as MuHAVi-uncut [4] which contains much longer sequences also involvingmore actors and more camera views. In this case, silhouetteshave been obtained automatically by a foreground estimationalgorithm and therefore it contains noise and shadows likelyto be encountered in a real application. Thus, here we assesshow robust the action recognition algorithm is to these real im-perfections. In addition, we propose the use of a multiclassSupport Vector Machine classifier for the final stage in the pro-cess. The MuHAVi dataset contains 17 actions (table 1), carriedout by 7 different people and seen from 8 different cameras[3],as shown in figure 1 (these are from MuHAVi-MAS that weresegmented manually).The rest of the paper is organised as follows: Section 2presents a brief review, Section 3 describes the vision algo-rithm used, dataset and classifier. Section 4 reports the resultsobtained and section 5 concludes the paper.2 Related worksA wide range of different algorithms and systems have beenproposed for Human Action Recognition. As the literature is1Figure 1. Images of some actions of Muhavi dataset.getting rather large, here we focus on works that have experi-mented with the same dataset. The interested reader can con-sult a good review of the subject in [5],[6] and [7].A multiview approach was proposed in [8], introducing theconcept of bag of key poses, which serves to describe that anaction can be represented by a sequence of key pose featuresextracted from silhouettes from different people and differentviews. A clustering process inspired by bag of words is used togroup pose sequences that can later be used to classified previ-ously unseen actions.Returning to the idea of bag of key poses [1] integrates theK-means algorithm as a classifier, obtaining much better com-putational performance results than previous algorithms whilehaving better accuracy and speed. This is the algorithm we willtest and expand in this paper.In a different approach, [2] presents a human action repre-sentation, consisting of Motion History Images (MHIs), whichrepresent how often (in a given temporal window) a given pixelhas been seen as foreground, thus encoding the amount of ”mo-tion” (or presence) of that pixel, hypothesising that that is re-lated to actions. Then, it uses Histograms of Oriented Gradi-ents (HOG) applied to the MHIs as features that can be used fortraining. A simple approach based on Nearest Neighbour (NN)is used as a classifier to determine the class of a previously un-seen action sample by finding a trained sample to which it isthe closest.Action Class ActionC1 WalkTurnBackC2 RunStopC3 PullHeavyObjectC4 PickupThrowObjectC5 PunchC6 KickC7 ShotGunCollapseC8 WalkFallC9 LookInCarC10 CrawlOnKneesC11 WaveArmsC12 DrawGraffitiC13 JumpOverFenceC14 DrunkWalkC15 SmashObjectC16 JumpOverGapC17 ClimbLadderTable 1. Actions available in the MuHavi dataset.3 Proposed approachAs pointed out above, in this paper we use the approach firstproposed in [1] and test it under the much more stringent con-ditions of MuHaVi-uncut. For the sake of completeness, weoutline part of the algorithm here.3.1 Image segmentation and noise reductionAs outlined earlier, MuHAVi-uncut has been generated usingwhat was at the time state-of-the-art foreground estimation us-ing mixture of Gaussians. Although the method could also esti-mate shadows, not all shadows are removed and the silhouettesare also inherently noisy. Therefore, as proposed by [2] an im-age filtering process using a 15x15 median filter is performed,as illustrated in figure 2. Using the same filter allows to directlycompare results with [2].(a) (b)Figure 2. Images (a) without filter (b) with filter.3.2 Feature extractionA relatively fast feature extraction is proposed in [9], whichconsists of a series of steps as described below:This part of the algorithm follows what was proposed in[10]. In each frame of a given action sequence, from thesilhouette a contour of N points P = p1, p2, ..., pN , wherepi = (xi, yi), is calculated. If N is sufficiently large, itschoice is not too critical. From these N points the centroidC = (xc, yc) is calculated (please also see figure 3) by:2xc =∑Ni=1 xiN, yc =∑Ni=1 yiN(1)(a) (b)Figure 3. (a) Silhouette (b) with calculated centroidNext, the silhouette (centred atC) is inscribed by the small-est circumference and divided up into n equal segments asshown in figure 4.Figure 4. Silhouette inscribed by smallest circumference anddivision of the same.A list is computed B = [(α1, b1), ..., (αn, bn))] in a clock-wise direction, and the number of elements on list B is callednB (set to 18 in our experiment, so the circle is divided up inequal segments of 20 degrees), where each element αi repre-sents an arc (in radians) of the corresponding circle.αi ={arccos(yi−ycdi ) · 180pi if xi ≥ 0180 + arccos(yi−ycdi ) · 180pi otherwise(2)bi =⌈nB · αi360⌉,∀i ∈ [1...nB] (3)With reference to figure 5, a maximum distance is com-puted from the furthest and closest points from the centroid:frange(pk, pk+1, ..., pl) = max(dk, dk+1, ..., dl)−min(dk, dk+1, ..., dl)Where d is the distance between each point p and the cen-troid C of the circumference, the distance is calculated withthe equation:dk =√(x1 − x2)2 + (y1 − y2)2 (4)Figure 5. Calculating the distance of each point in the silhou-ette with respect to the centroid C.Then, for each circle segment j, a value vj is calculated as:vj = frange(pk, pk+1, ..., pl)/bk, ..., bl (5)which are then normalised:vj =vj∑nBi=1 vi(6)Finally, the feature vector V is formed by all the segmentcomponents:V = (v1, v2, .., vn) (7)3.3 Action RecognitionWe use supervised training using a one-against-one multiclassapproach with RBF (Radial Basis Function) SVM (SupportVector Machine) classifiers. RBF is known for being betterat capturing non-linear relations in the data. One-against-onedecomposes the problem of k classes into k(k − 1)/2 binaryproblems, where all the possible one-to-one combinations be-tween all classes are created. Each new data is evaluated for allcreated classifiers, obtaining one vote for each winning class ineach case, then the final result is the class with the most votes.In this case 136 binary classifiers are created that represent eachcombination of pairs of classes.No additional data normalisation is required, as that is doneby the feature extraction process described earlier.The RBF kernel needs two parameters, C and γ, which areapplication dependent. We use the well accepted exhaustivegrid approach that finds these parameters using cross-validationon a testing dataset and choose the combination that producesthe smallest mean classification error. The testing dataset (andtherefore the computation of optimal SVM parameters) de-pends on the evaluation being performed.34 ResultsWe follow the LOCO and LOAO tests suggested by [2], as de-scribed below.4.1 LOCO (Leave-one-camera-out)In this test, one camera is left out and the algorithm is trainedwith the remaining cameras and tested with the camera that wasleft out. This evaluates view dependence. This is done for allseven cameras (one camera in MuHAVi-uncut is not usable asit provides much less data).Table 2 shows the mean accuracy and standard deviation re-sults of the human actions recognition by means of the LOCOtest and compares it with the results reported in [2], which itoutperforms significantly in all cases (its overall average accu-racy is 85.52% vs. 49.28%), indicating its better resilience topoint of view. It is also noted that this method is less variablefrom one camera to another.Action Mean (%) St. dev.(%) [2]1 77,11 15,07 26,22 94,46 13,96 61,63 85,26 14,23 64,74 89,95 9,93 54,25 91,37 12,14 39,76 85,91 15,67 55,47 87,19 14,64 64,38 78,07 22,74 1,49 81,17 14,67 46,910 90,10 11,16 48,511 85,59 15,36 59,612 88,13 14,07 12,213 80,24 13,81 77,614 82,60 10,74 52,415 81,51 17,46 87,116 93,86 9,76 51,717 81,24 18,13 34,7Overall 85,52 - 49,28Table 2. LOCO Mean and standard deviation of the recognitionfor each of the 17 actions.4.2 LOAO (Leave-one-actor-out)In this test, one actor is left out and the algorithm is trainedwith the remaining actors and tested for the actor that was leftout. This evaluates actor dependence. This is done for all theactors.Table 3 shows the mean accuracy and standard deviationresults of the human actions recognition for the LOAO test andcompares it with the results reported in [2], which it outper-forms in many cases (its overall average accuracy is 93.52%vs. 83.11%). As expected, accuracy is higher and variationsmaller than for the LOCO test in all cases as this test is lessstringent (there is less variability between actors than betweencamera views).Action Mean (%) St. dev.(%) [2]1 92,67 3,77 90,52 94,37 4,16 98,23 95,87 4,30 83,04 92,14 3,66 60,75 94,21 3,65 65,66 91,73 4,10 71,97 94,63 4,51 95,78 93,60 4,96 95,29 93,52 4,89 76,910 93,10 4,24 98,011 92,80 3,96 87,512 94,45 4,09 34,713 93,27 3,89 91,814 95,10 4,37 87,715 93,32 4,26 97,316 93,13 4,65 82,317 91,85 4,19 95,9Overall 93,52 - 83,111Table 3. LOAO Mean and standard deviation of the recognitionfor each of the 17 actions.5 ConclusionIn this paper we have shown improvements on human actionrecognition on the MuHAVi-uncut dataset, particularly chal-lenging because of salt-pepper noise and shadows. We haveused silhouette features as proposed by [1] in combination witha multiclass one-against-one SVM classifier. This combinationhas proven particularly good compared to previous results [2],in the case of a LOCO test reaching an average accuracy of85.5% and of 93.5% for LOAO. Performance has also beenobserved to be more stable to changes than the compared al-gorithm. Although feature extraction is relatively fast, the pro-posed method is slow to fine-tune to find optimal SVM param-eters. Future work will look at datasets with more complexmulti-person actions and consider deep learning methods (al-though they might be tricky to work with relative small datasetsand so data augmentation techniques would need to be ex-plored). We also want to explore better techniques to extractsilhouettes that do not rely on foreground/background separa-tion so that they can be use in non-static cameras.AcknowledgementsSergio A Velastin has received funding from the UniversidadCarlos III de Madrid, the European Union’s Seventh Frame-work Programme for research, technological development anddemonstration under grant agreement no. 600371, el Ministe-rio de Economı´a, Industria y Competitividad (COFUND2013-51509) el Ministerio de Educacio´n, cultura y Deporte (CEI-15-17) and Banco Santander.4References[1] A. A. Chaaraoui and F. Flo´rez-Revuelta, “Adaptive hu-man action recognition with an evolving bag of keyposes,” IEEE Transactions on Autonomous Mental De-velopment, vol. 6, no. 2, pp. 139–152, 2014.[2] F. Murtaza, M. H. Yousaf, and S. A. Velastin, “Multi-view human action recognition using 2d motion templatesbased on mhis and their hog description,” Iet ComputerVision, vol. 10, no. 7, pp. 758–767, 2016.[3] S. Singh, S. A. Velastin, and H. Ragheb, “Muhavi: A mul-ticamera human action video dataset for the evaluation ofaction recognition methods,” in Proceedings of the 20107th IEEE International Conference on Advanced Videoand Signal Based Surveillance, AVSS ’10, (Washington,DC, USA), pp. 48–55, IEEE Computer Society, 2010.[4] J. Sepu´lveda and S. Velastin, “F1 Score Assesment ofGaussian Mixture Background Subtraction AlgorithmsUsing the MuHAVi Dataset,” 6th International Confer-ence on Imaging for Crime Prevention and Detection(ICDP-15), pp. 8 (6 .)–8 (6 .), 2015.[5] R. Poppe, “A survey on vision-based human action recog-nition,” Image and vision computing, vol. 28, no. 6,pp. 976–990, 2010.[6] J. M. Chaquet, E. J. Carmona, and A. Ferna´ndez-Caballero, “A survey of video datasets for human ac-tion and activity recognition,” Computer Vision and Im-age Understanding, vol. 117, no. 6, pp. 633–659, 2013.[7] S. Herath, M. Harandi, and F. Porikli, “Going deeper intoaction recognition: A survey,” Image and vision comput-ing, vol. 60, pp. 4–21, 2017.[8] A. A. Chaaraoui, P. Climent-Pe´rez, and F. Flo´rez-Revuelta, “An efficient approach for multi-view humanaction recognition based on bag-of-key-poses,” in HumanBehavior Understanding (A. A. Salah, J. Ruiz-del Solar,C¸. Meric¸li, and P.-Y. Oudeyer, eds.), (Berlin, Heidelberg),pp. 29–40, Springer Berlin Heidelberg, 2012.[9] A. A. Chaaraoui, “Vision-based Recognition of HumanBehaviour for Intelligent Environments,” 2014.[10] S. Suzuki and K. Abe, “Topological Structural Anal-ysis of Digitalized Binary Images by Border Follow-ing,” Computer Vision, Graphics, and Image Processing,vol. 30, no. 1, pp. 32–46, 1985.5

Silhouette-based human action recognition with a multi-class support vector machine

Abstract

Similar works

Full text

Available Versions

Crossref

Universidad Carlos III de Madrid e-Archivo