Nicolaou, Mihalis

Ringer, Charles

English

Goldsmiths Research Online

Nicolaou, Mihalis and Ringer, Charles. 2018. ’Streaming Behaviour: Live Streaming as aParadigm for Multi-view Analysis of Emotional and Social Signals’. In: Foundations on DigitalGames 2018, Twitch Workshop. Malmo, Sweden. [Conference or Workshop Item]http://research.gold.ac.uk/id/eprint/24110/The version presented here may differ from the published, performed or presented work. Pleasego to the persistent GRO record above for more information.If you believe that any material held in the repository infringes copyright law, please contactthe Repository Team at Goldsmiths, University of London via the following email address:gro@gold.ac.uk.The item will be removed from the repository while any claim is being investigated. Formore information, please contact the GRO team: gro@gold.ac.ukStreaming Behaviour: Live Streaming as a Paradigm forMulti-view Analysis of Emotional and Social SignalsMihalis A. NicolaouDepartment of ComputingGoldsmiths, University of LondonLondon, United KingdomCharles RingerDepartment of ComputingGoldsmiths, University of LondonLondon, United KingdomABSTRACTWe argue that live video game streams constitute an invaluable par-adigm towards building multi-view, data-driven models of humanbehaviour. The interactive setting under which a stream operatesis enriched with social signals, conveyed between streamers andviewers via facial expressions, body movement, vocal cues, as wellas written language. We consider the data sources involved in atypical broadcast (e.g., camera stream, game footage, text) as data-views that carry inherent correlations, since they all describe eventsoccurring during a stream. We argue that this unique interactivesetting facilitates the joint, multi-view analysis of various hetero-geneous data sources involved, in a coherent and self-containedmanner. We elaborate on the emergence of social signals in thissetting, while discussing close links and potential research directi-ons related to areas such as affective computing, machine learning,computer vision and intelligent game design.1 INTRODUCTIONVideo game live streaming has become a dominant form of en-tertainment over the past years, with emerging communities ofmillions of broadcasters and viewers on popular platforms such asTWITCH.TV [5]. This paper is motivated by two relevant observati-ons, namely (i) the inherently interactive nature of streaming, wherestreamers may interact with each other as well as their viewers, and(ii) the rich, heterogeneous data that accompanies a broadcast (e.g.,game and webcam footage, speech, natural language). We arguethat the social, interactive setting that streamers find themselves inleads to the emergence of a multiset of complex emotional and so-cial signals (e.g., during co-operative play), that can be studied in thecontext of affective computing, where research in the past decade hasfocused on the machine analysis of social and emotional signals ba-sed on visual and auditory cues. The ‘streamer’s paradigm’ providesa rich source for vast amounts of heterogeneous data that encodesocial signals, as conveyed by facial expressions, speech emotion, aswell as viewer sentiment via live chat. As we discuss in what follows,this setting gives rise to a set of research problems that stronglyintersect with research undertaken in areas such as affective compu-ting, machine learning and computer vision, audio signal processingand natural language processing, as well as intelligent game design.2 RELATEDWORKMost related to the live stream setting are databases collected to-wards player experience modelling [15]. For instance, the Platfor-mer Experience Dataset (PED) [3] employs games as a means to eli-cit affect, and contains audio-visual recordings of players alongwithin-game data, context and self-reported experience rankings. Mainvariations with respect to the game stream setting include (i) thesocial interaction element, where players and viewers interact andtherefore are more likely to communicate social and emotional cues,and (ii) the uncontrolled settings under which streams take place.Other works analysing streamer behaviour include [10], wherewebcam and physiological data have been used for the analysis.Nevertheless, this work focuses on building a tool that utilizes ex-isting SDKs for facial analysis such as Affdex [8], while wristbandsare used for extracting physiological data - thus requiring accessto the streamer. Other related works include analysis of affect andbehaviour in co-operative and single player games, c.f. [1], wherephysiological and visual data from eight subjects is utilized. Finally,we refer the reader to recent articles such as [12, 13] for a review onrelated tasks such as audio-visual analysis of human behaviour andfacial expression analysis, while the interaction between machinelearning and intelligent game design is further explored in [16].3 STREAMING & MULTI-VIEW DATAIn this section, we discuss data sources emerging in a live streamsetting, while considering social interactions taking place. Duringa typical game stream, the streamer broadcasts webcam video, usu-ally capturing the streamer’s face and in some cases body, alongwith audio and the game footage. Social and behavioural cues arecommunicated to viewers and potentially other streamers withfacial expressions, body gestures, as well as via written and spo-ken language. Viewers communicate via chat, as well as by raisingvarious stream events (e.g., subscriptions). We expect that the in-teractive setting assumed in a typical game stream will give riseto more rich behavioural and emotional content. In settings wherestreamer-to-streamer interaction arises (e.g., co-operative play), thedata sources available extend to cover the broadcast of other playersparticipating. By considering both streams (including facial expres-sions and audio of both players), we can potentially better analysemore complex social behaviours such as conflict, agreement, andmimicry. In Fig. 1, we illustrate the discussed game stream conceptfrom a data-centric perspective. The audio-visual1 nature of thestream points to research in machine learning, computer vision,and acoustic signal processing, while data in the form of writtenlanguage to areas such as sentiment analysis and natural languageprocessing. By considering the uncontrolled settings under whichbroadcasts are taking place, the analysis of game streams becomesa very challenging and rich data paradigm.In Fig. 2, we show camera stills from data collected during atypical game stream of a popular game. In more detail, in the firstand second rows, frames indicate the variability in terms of facial1We note that fusing audio-visual data is in itself a challenging research direction [4].FDG 2018, Twitch Workshop, August 2018, Malmo, Sweeden M. A. Nicolaou and Charles RingerMulti-view GameStream DataStreamerCameraStreamAudioGameGameAudioGameFootageViewers Text ChatStreamEventsSocial Cues:.Body Gestures.Facial Expressions.Speech Emotion.Non-verbal cues.Written LanguageResearch Areas:.Machine Learning.Computer Vision.Affective Computing.Sentiment Analysis.Audio Signal Processing. Language Processing. Intelligent Game DesignStreamer2socialinteractionsocial interactioninteractionwatchingFigure 1: Illustration of main concepts discussed in the paper, including social interactions, social and emotional signals con-veyed, as well as related research areas. The streamer broadcast includes a video stream capturing face and/or body, along withaudio recorded from the streamer’smicrophone. This is accompanied by the footage of the game, including in-game audio. Thestreamer is interacting with viewers by video, voice, and text. Social signals can be extracted by analysing facial expressions,body gestures, speech emotion and non-verbal cues. The viewers interact with streamers and other viewers via text and stream-specific events, where analysis of the former is in the realms of sentiment and natural language processing. A frequent settingincludes the streamer interacting with other streamers via webcam video and voice (e.g., during a co-op game). This allows formodelling complex dyadic interactions via voice and facial expression analysis using data arising from both broadcasts.Figure 2: Camera stills from a typical game stream, inclu-ding various facial expressions, body gestures, head poses,as well as challenging frames with face occlusions.expressions and gestures respectively, as expressed by the streamerwhile communicating with viewers and other streamers. The thirdand fourth rows contain stills capturing head pose variability, aswell as randomly selected stills where parts of the face are occlu-ded by other body parts. This deems the computer vision analysisof facial data even more challenging, and is also on par with therecent developments in face analysis, where data captured underunconstrained settings (‘in-the-wild’) are commonly considered aschallenging benchmarks [6, 7].4 RESULTSWhile stream data is in abundance, providing annotations on whichmachine learning models can be trained is a tedious and laborioustask, that requires some form of human supervision and is parti-cularly expensive in case per-frame labels are to be provided. It istherefore reasonable to firstly approach this problem in the contextof unsupervised learning. In our recent work [9], we propose a deepunsupervised novelty-detection approach for detecting highlightsin game streams2, by utilizing multi-view data including streamerfacial expressions, audio, and game footage.The method consists of two convolutional autoencoder networksencoding game footage and facial expressions, with the facial ex-pression network following a VGG16-like architecture [14]. Audio2In this context, we consider that game stream highlights do not only depend on thegame footage, but also on the social and emotional behaviour of the streamer.Streaming Behaviour FDG 2018, Twitch Workshop, August 2018, Malmo, Sweedenfeatures are extracted by using the principal components of Fou-rier coefficients, while subsequently a recurrent layer is used forfusion. The premise is that a high reconstruction error points tonovelty, and novelty can be considered as a proxy for detectinginteresting parts of the stream. This also deems the method suit-able for segmenting a long stream into episodes that are likely tocontain emotional and social content, especially when utilizing thestreamer’s face and voice. Evaluation was carried out on over 5hours of footage broadcasted by two streamers playing a populargame. Results presented in [9] show that, as expected, analysis offacial expressions and audio significantly increases the precisionof detected episodes, compared to simply using the game footage.In Fig. 3, we show some sequences from detected episodes wheresocial and behavioural signals can be observed. In the first row, thesequence shows the player winning a game and reacting positivelyto this. In the second, the player is reacting to an in-game event.In the third row the player appears intrigued. Finally, in the fourthrow the streamer is interacting with viewers.Figure 3: Sequences from episodes extracted by applying [9]on face video, player audio, and game footage.5 RESEARCH DIRECTIONSFirstly, further studies should utilize a more rich and diverse set ofstreamers, while gathering enough game-specific data to facilitatemodelling game context. Furthermore, several research directionsspan from exploiting such a vast source of heterogeneous data. Forexample, while unsupervised methods such as [9] are suitable forsegmentation, supervised learning approaches can be employedto learn specific appraisals of the players emotional state directlyfrom the multi-view observed data (e.g., by using latent dimensionssuch as valence-arousal [11]). Extraction of video-based physiologi-cal measurements [2] can also potentially contribute towards this.Learning mappings from a subset of views leads to further inte-resting problems, e.g. predicting player behaviour and experiencedirectly by observing game footage raw pixels. Given appropriatedata, complex behaviours such as mimicry and conflict can be stu-died in an audio-visual manner, by exploiting recent advances inmachine and deep learning that scale to large datasets. Further-more, semi-supervised methods can be used to model game-contextas side-information, while creating platforms for recording othertypes of information (e.g., playtraces) can lead to further promisingresearch directions.6 CONCLUSIONSIn this paper, we discuss the underlying potential in utilizing thevast amounts of audio-visual data broadcasted during a typicalgame stream. We argue and show that the interactive setting un-der which a stream operates is enriched with emotional and socialsignals, thus providing a vast source of multi-view data that can beutilized towards tasks such as the machine analysis of social andemotional behaviour.ACKNOWLEDGMENTSWe thank the NVIDIA Corporation for providing a Titan X GPUused in this work. Charles Ringer’s work is supported by EPSRCgrant EP/L015846/1 (IGGI). The authors would like to thank MatsKathage (P4wnyhof) for permission to use footage fromhis streams.REFERENCES[1] Daniel Gábana Arellano, Laurissa Tokarchuk, and Hatice Gunes. 2016. Measuringaffective, physiological and behavioural differences in solo, competitive andcollaborative games. In International Conference on Intelligent Technologies forInteractive Entertainment. Springer, 184–193.[2] Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-Based Physiolo-gical Measurement Using Convolutional Attention Networks. arXiv preprintarXiv:1805.07888 (2018).[3] K. Karpouzis, G. N. Yannakakis, N. Shaker, and S. Asteriadis. 2015. The platformerexperience dataset. In 2015 International Conference on Affective Computing andIntelligent Interaction (ACII). 712–718. https://doi.org/10.1109/ACII.2015.7344647[4] Aggelos K Katsaggelos, Sara Bahaadini, and Rafael Molina. 2015. Audiovisualfusion: Challenges and new approaches. Proc. IEEE 103, 9 (2015), 1635–1653.[5] Mehdi Kaytoue, Arlei Silva, Loïc Cerf, Wagner Meira Jr, and Chedy Raïssi. 2012.Watch me playing, I am a professional: a first study on video game live streaming.In Proceedings of the 21st International Conference on World Wide Web. ACM.[6] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papai-oannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou.2018. Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, DeepArchitectures, and Beyond. arXiv preprint arXiv:1804.10938 (2018).[7] Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic. 2017.AFEW-VA database for valence and arousal estimation in-the-wild. Image andVision Computing 65 (2017), 23–36.[8] Daniel McDuff, Abdelrahman Mahmoud, Mohammad Mavadati, May Amr, JayTurcot, and Rana el Kaliouby. 2016. AFFDEX SDK: a cross-platform real-timemulti-face expression recognition toolkit. In Proceedings of the 2016 CHI Confe-rence Extended Abstracts on Human Factors in Computing Systems. ACM.[9] Charles Ringer and Mihalis A. Nicolaou. 2018. Deep Unsupervised Multi-ViewDetection of Video Game Stream Highlights. In International Conference on Foun-dations of Digital Games. ACM.[10] Raquel Robinson, Zachary Rubin, Elena Márquez Segura, and Katherine Isbister.2017. All the feels: designing a tool that reveals streamers’ biometrics to spectators.In Proceedings of the 12th International Conference on the Foundations of DigitalGames. ACM, 36.[11] James A Russell. 1980. A circumplex model of affect. Journal of personality andsocial psychology 39, 6 (1980), 1161.[12] Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro. 2015. Automaticanalysis of facial affect: A survey of registration, representation, and recognition.IEEE transactions on pattern analysis and machine intelligence 37, 6 (2015).[13] Björn W. Schuller. 2018. Speech Emotion Recognition: Two Decades in a Nutshell,Benchmarks, and Ongoing Trends. Commun. ACM 61, 5 (April 2018), 90–99.https://doi.org/10.1145/3129340[14] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networksfor large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).[15] Georgios N. Yannakakis and Julian Togelius. 2011. Experience-Driven ProceduralContent Generation. IEEE Transactions on Affective Computing 2, 3 (2011), 147–161. https://doi.org/10.1109/T-AFFC.2011.6[16] Georgios N. Yannakakis and Julian Togelius. 2018. Artificial Intelligence andGames. Springer. http://gameaibook.org.

Streaming Behaviour: Live Streaming as a Paradigm for Multi-view Analysis of Emotional and Social Signals

https://research.gold.ac.uk/id/eprint/24110/1/streaming_behaviour.pdf

Streaming Behaviour: Live Streaming as a Paradigm for Multi-view Analysis of Emotional and Social Signals

Abstract

Similar works

Full text

Available Versions

Goldsmiths Research Online