In GameStory: The 2019 Video Game Analytics Challenge, two main tasks are nominated to solve in the challenge, which are replay detection - multi-stream synchronization, and game story summarization. In this paper, we propose a data-driven based approach to solve the first task: replay detection - multi-stream synchronization. Our solution aims to determine the replays which lie between two logo-transitional endpoints and synchronize them with their sources by extracting frames from videos, then applying image processing and retrieval remedies. In detail, we use the Bag of Visual Words approach to detect the logo-transitional endpoints, which contains multiple replays in between, then employ an Image Signature Matching algorithm for multi-stream synchronization and replay boundaries refinement. The best configuration of our proposed solution manages to achieve the second-highest scores in all evaluation metrics of the challenge

Dang-Nguyen, Duc-Tien

Gurrin, Cathal

Le, Tu-Khiem

Ninh, Van-Tu

DCU Online Research Access Service

Replay Detection and Multi-stream Synchronization in CS:GOGame Streams Using Content-based Image Retrieval and ImageSignature MatchingVan-Tu Ninh1*, Tu-Khiem Le1*, Duc-Tien Dang-Nguyen2, Cathal Gurrin11 Dublin City University, Ireland2 University of Bergen, Norwaytu.ninhvan@adaptcentre.ie,tukhiem.le4@mail.dcu.ie,ductien.dangnguyen@uib.no,cathal.gurrin@dcu.ieABSTRACTIn GameStory: The 2019 Video Game Analytics Challenge, twomain tasks are nominated to solve in the challenge, which arereplay detection - multi-stream synchronization, and game storysummarization. In this paper, we propose a data-driven based ap-proach to solve the first task: replay detection - multi-stream syn-chronization. Our solution aims to determine the replays whichlie between two logo-transitional endpoints and synchronize themwith their sources by extracting frames from videos, then applyingimage processing and retrieval remedies. In detail, we use the Bagof Visual Words approach to detect the logo-transitional endpoints,which contains multiple replays in between, then employ an ImageSignature Matching algorithm for multi-stream synchronizationand replay boundaries refinement 1. The best configuration of ourproposed solution manages to achieve the second-highest scores inall evaluation metrics of the challenge.1 INTRODUCTIONGame analytics is a new research area in recent years that has be-come popular due to the explosion of both modern live streamingtechnologies (e.g., Twitch, Discord, Youtube) and e-sports devel-opments [2, 4]. Although e-sports has gained significant attentionfrom audiences across a variety of ages, not much research hasbeen conducted to analyse the game play to provide the spectatorwith either a quick look into a match (e.g., player’s performancestatistics, highlights, critical moments, summary) or a deep insightinto a game (e.g., whole game statistics, outstanding team’s playingstyle analysis, playing strategy trending analysis).In GameStory: The 2019 Video Game Analytics Challenge, theorganisers use the same dataset used in the previous year, whichwas provided by ZNIPE.tv [8]. More details of the data and taskdescriptions can be found in [8].2 RELATEDWORKSA lot of research in sports videos was conducted to analyse thehighlights, the replays, and the story of the match. Jinjun Wang,et.al. performed a shot classification followed by a proposed scenetransition structure analysis on the labels of the classified shots to1https://github.com/nvtu/gamestory_mediaeval2019* These two authors contributed equally.Copyright held by the owner/author(s).MediaEval’19, 27-29 October 2019, Sophia Antipolis, Francedetect replay scenes in sports [12]. Later, Bai Liang, Song-Yang Lao,et.al. proposed the Perception Concept Network-Petri Net (PCN-PN) model to search and locate interesting events in sports videos[5, 10], which is also a potential approach to detect replays as theyare usually critical events in the match. In 2019, Ali Javed, et.al. pro-posed a novel approach for key-event detection and summarizationusing Confined Elliptical Local Ternary Patterns (CE-LTPs) to ex-tract feature of motion history image for each key-event candidatelying between the beginning and end of a gradual transition, thenapplying Extreme Machine Learning (EML) to learn the pattern todetect the key-events in sports videos [3]. The results of this workcan be used for both replay detection and story summarization ofdifferent kinds of sports videos.3 APPROACHIn the spirit of previous efforts in the field of video event detectionand segmentation, we propose a straightforward method to detectreplays in videos which are bounded by Intel Extreme Masters logotransition scenes. The logo in these frames is shown in a plainblue background and occupies approximately 80% of its image. Wepropose to use a Bag-of-Visual-Words (BOVW) approach to detectthese two endpoints of each replay segment and use image signaturematching to synchronize multi-stream players’ views.3.1 Logo-bounded Video Detection3.1.1 Frame Extraction and Filtering: Firstly we use the ffmpegtool to extract frames from commentator stream. To capture theframes in which the Intel Extreme Masters logo is clearly visible,we perform frame extraction at fps=2. As the video length is long(approximately 12 hours), many redundant frames are generated.We reduce the number of extracted frames by using a two pointerstechnique to eliminate consecutive frames which have similar ORBfeatures [11] and the degree of similarity of a gray-scaled colorhistogram. For the ORB features, we only consider the top 500features in total for comparison. Specifically, at frame i , we iteratethrough its next consecutive frames until we reach a frame j (j > i)such that the number of matched ORB features does not exceedα and the L2 distance between the two corresponding histogramsis greater than or equal to β . In our implementation, we chooseα = 200 and β = 0.8 (β ≤ 1), which means that 60% of the ORBfeatures are different and the degree of similarity between the twocolor histograms is small. The ones between the frame i and j areredundant and do not provide any extra information for the nextsteps.MediaEval’19, 27-29 October 2019, Sophia Antipolis, France Van-Tu Ninh, Tu-Khiem Le, et al.3.1.2 Logo-bounded Video Detection using a Retrieval Approach:From the result in section 3.1.1, we use this filtered data to constructa dictionary of visual words (codebook). We run K-Means clustering[6] on the rootSIFT-feature data points [1, 7] extracted from im-ages in the same corpus. Before processing through the remainingBOVW’s steps, frames are center-cropped to reduce backgroundnoise. Another purpose is to force the model to focus on certainparts of the frame, which have a high probability of containing theIntel Extreme Masters logo only. After having retrieved the desiredframes, we can then select the proper endpoint pairs by assumingthat the replay’s length would not be more than 20 seconds.One problem in this stage is that we can only determine thelogo-bounded videos, which might contain many other replaysfrom either the same perspective or multiple perspectives. As thetransitions in these videos are smooth, we propose to use the resultsgenerated from the section 3.2 to refine the replay detection result.3.2 Multi-stream SynchronizationWith the result obtained from the section 3.1.2, we extract framesof these logo-bounded videos with their original fps, which is 59 forthe commentator stream. We apply the same process to the players’videos with their corresponding fps. The problem now is to find inthe indexed database the image which is a near duplication of thequery one. Therefore, the players frames are then encoded by using[13], added to the database, and then indexed for searching withElastic Search engine 2. Instead of searching the whole databasefor the nearly duplicate images, we determine the match and shape(roundness) of the logo-bounded videos based on providedmetadataand their time in the commentator stream. Thereby, we can bothnarrow down the search space and locate the start frame of eachreplay.3.3 Refine Replay Detection ResultAs stated in section 3.1.2, for a logo-bounded video with multi-player view, the problem could be solved easily using multi-streamsynchronization output. However, it would be harder for the onewith single-player view only as we now need to detect abrupt scenechanges in the video to split it into proper replays. By using thesearched rank list output from the section 3.2, for two consecu-tive frames, we choose the most similar frame from the indexeddatabase, compute the absolute time difference in the synchronisedplayer stream, and then set a threshold. Thereby, we could reusethe result of the previous step to detect abrupt scene transitionsbetween two replays in the stream with single-player view andrefine replay detection output based on the source time of similarframes in player streams set.4 RESULTS AND ANALYSISIt can be seen from the graph shown in figure 1 that the best runof our team (DCU-Computing) manages to gain a precision of73.17% and a recall of 68.18%, yielding F1-score of 70.59%, whichis the second-highest score among the submissions for evaluationat Jaccard threshold of 0.5. For evaluation at Jaccard thresholdof 0.75 shown in figure 2, all of our scores decrease significantly(approximately 23.53%), except for the average overlapping score2https://github.com/EdjoLabs/image-matchFigure 1: Evaluation of all team’s runs at Jaccard-Indexthreshold = 0.5Figure 2: Evaluation of all team’s runs at Jaccard-Indexthreshold = 0.75of multi-stream synchronization. The evaluation shows that ouralgorithm works well to retrieve correctly more than 50%, but lessthan 75% of a replay. Our result is lower than the best approach bythe AAU-Mekul team by around 20.98% [9]. As we detect replaysbased on two logo-transitional endpoint pairs, our algorithm cannothandle the case that misses one endpoint.There are many situations that we mistakenly split the replaysof a player due to wrong image signature matching results. Forinstance, in case that the player stands still and shoots, then movesa little bit and finally returns to the original point and shoots again,most of the frames will all find its source at one time point, whilea sudden player’s movement yields a different time point, whichcauses our algorithm to consider it as a scene change and split re-plays. Or when the player’s view is affected by smoke/blind grenade,the features of the white scene are not distinctive enough to deter-mine its source. These cases all lead to the wrong replay splits thatreduces our detected replays’ length, which decreases our score.Our approach works perfectly with logo-bounded streams contain-ing only one replay. For other cases, we can generate acceptableresults which might have few incorrect splits.ACKNOWLEDGMENTSThis publication has emanated from research supported by Sci-ence Foundation Ireland under grant numbers SFI/12/RC2289 and13/RC/2106.GameStory: Video Game Analytics Challenge MediaEval’19, 27-29 October 2019, Sophia Antipolis, FranceREFERENCES[1] R. Arandjelović and A. Zisserman. 2012. Three things everyone shouldknow to improve object retrieval. In 2012 IEEE Conference on ComputerVision and Pattern Recognition. 2911–2918. https://doi.org/10.1109/CVPR.2012.6248018[2] Nathan Edge. 2013. Evolution of the Gaming Experience: Live VideoStreaming and the Emergence of a NewWeb Community. Elon Journalof Undergraduate Research in Communications 4 (2013), 2153–5760.[3] Ali Javed, Aun Irtaza, Yasmeen Khaliq, Hafiz Malik, and Muham-mad Tariq Mahmood. 2019. Replay and key-events detection forsports video summarization using confined elliptical local ternarypatterns and extreme learning machine. Applied Intelligence 49, 8 (01Aug 2019), 2899–2917. https://doi.org/10.1007/s10489-019-01410-x[4] Mark Johnson and Jamie Woodcock. 2018. The impacts of live stream-ing and Twitch.tv on the video game industry. Media Culture Society(12 2018). https://doi.org/10.1177/0163443718818363[5] Bai Liang, Song-Yang Lao, Alan Smeaton, Noel O’Connor, DavidSadlier, and David Sinclair. 2009. Semantic Analysis of Field SportsVideo using a Petri-Net of Audio-Visual Concepts. Liang, Bai andLao, Songyang and Smeaton, Alan F. and O’Connor, Noel E. and Sadlier,David and Sinclair, David (2009) Semantic analysis of field sports videousing a petri-net of audio-visual concepts. The Computer Journal, 52(7). pp. 808-823. ISSN 0010-4620 52 (10 2009). https://doi.org/10.1093/comjnl/bxn058[6] S. Lloyd. 1982. Least squares quantization in PCM. IEEE Transactionson Information Theory 28, 2 (March 1982), 129–137. https://doi.org/10.1109/TIT.1982.1056489[7] David G. Lowe. 2004. Distinctive Image Features from Scale-InvariantKeypoints. International Journal of Computer Vision 60, 2 (01 Nov2004), 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94[8] Mathias Lux, Michael Riegler, Duc Tien Dang Nguyen, Marcus Lar-son, Martin Potthast, and Pål Halvorsen. 2019. GameStory Task atMediaEval 2019. In Proceedings of MediaEval 2019.[9] Kevin Mekul. 2019. Automated Replay Detection and Multi-StreamSynchronization. In Proceedings of MediaEval 2019.[10] Wolfgang Reisig. 1985. Petri Nets: An Introduction. Springer-Verlag,Berlin, Heidelberg.[11] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011.ORB: an efficient alternative to SIFT or SURF. Proceedings of theIEEE International Conference on Computer Vision, 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544[12] Jinjun Wang, Eng Chng, and Changsheng Xu. 2005. Soccer replaydetection using scene transition structure analysis. IEEE InternationalConference on Acoustics, Speech, and Signal Processing 2, ii/433 – ii/436Vol. 2. https://doi.org/10.1109/ICASSP.2005.1415434[13] Chi Wong, Marshall W. Bern, and David Goldberg. 2002. An imagesignature for any kind of image. Proceedings. International Conferenceon Image Processing 1 (2002), I–I.

Replay detection and multi-stream synchronization in CS:GO game streams using content-based Image retrieval and Image signature matching

Dang Nguyen, Duc Tien

Irish Universities

http://doras.dcu.ie/23953/1/MediaEval_GameStoryTask2019.pdf

Replay detection and multi-stream synchronization in CS:GO game streams using content-based Image retrieval and Image signature matching

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Irish Universities