Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me

In this paper we describe the fundamentals of affective gaming from a physiological point of view, covering some of the origins of the genre, how affective videogames operate and current conceptual and technological capabilities. We ground this overview of the ongoing research by taking an in-depth look at one of our own early biofeedback-based affective games. Based on our analysis of existing videogames and our own experience with affective videogames, we propose a new approach to game design based on several high-level design heuristics: assist me, challenge me and emote me (ACE), a series of gameplay "tweaks" made possible through affective videogames

Analysing user physiological responses for affective video summarisation

This is the post-print version of the final paper published in Displays. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2009 Elsevier B.V.Video summarisation techniques aim to abstract the most significant content from a video stream. This is typically achieved by processing low-level image, audio and text features which are still quite disparate from the high-level semantics that end users identify with (the ‘semantic gap’). Physiological responses are potentially rich indicators of memorable or emotionally engaging video content for a given user. Consequently, we investigate whether they may serve as a suitable basis for a video summarisation technique by analysing a range of user physiological response measures, specifically electro-dermal response (EDR), respiration amplitude (RA), respiration rate (RR), blood volume pulse (BVP) and heart rate (HR), in response to a range of video content in a variety of genres including horror, comedy, drama, sci-fi and action. We present an analysis framework for processing the user responses to specific sub-segments within a video stream based on percent rank value normalisation. The application of the analysis framework reveals that users respond significantly to the most entertaining video sub-segments in a range of content domains. Specifically, horror content seems to elicit significant EDR, RA, RR and BVP responses, and comedy content elicits comparatively lower levels of EDR, but does seem to elicit significant RA, RR, BVP and HR responses. Drama content seems to elicit less significant physiological responses in general, and both sci-fi and action content seem to elicit significant EDR responses. We discuss the implications this may have for future affective video summarisation approaches

ELVIS: Entertainment-led video summaries

© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Multimedia Computing, Communications, and Applications, 6(3): Article no. 17 (2010) http://doi.acm.org/10.1145/1823746.1823751Video summaries present the user with a condensed and succinct representation of the content of a video stream. Usually this is achieved by attaching degrees of importance to low-level image, audio and text features. However, video content elicits strong and measurable physiological responses in the user, which are potentially rich indicators of what video content is memorable to or emotionally engaging for an individual user. This article proposes a technique that exploits such physiological responses to a given video stream by a given user to produce Entertainment-Led VIdeo Summaries (ELVIS). ELVIS is made up of five analysis phases which correspond to the analyses of five physiological response measures: electro-dermal response (EDR), heart rate (HR), blood volume pulse (BVP), respiration rate (RR), and respiration amplitude (RA). Through these analyses, the temporal locations of the most entertaining video subsegments, as they occur within the video stream as a whole, are automatically identified. The effectiveness of the ELVIS technique is verified through a statistical analysis of data collected during a set of user trials. Our results show that ELVIS is more consistent than RANDOM, EDR, HR, BVP, RR and RA selections in identifying the most entertaining video subsegments for content in the comedy, horror/comedy, and horror genres. Subjective user reports also reveal that ELVIS video summaries are comparatively easy to understand, enjoyable, and informative

Treat me well : affective and physiological feedback for wheelchair users

This work reports a electrocardiograph and skin conductivity hardware architecture, based on E-textile electrodes, attached to a wheelchair for affective and physiological computing. Appropriate conditioning circuits and a microcontroller platform that performs acquisition, primary processing, and communication using Bluetooth were designed and implemented. To increase the accuracy and repeatability of the skin conductivity measuring channel, force measurement sensors were attached to the system certifying measuring contact force on the electrode level. Advanced processing including Rwave peak detector, adaptive filtering and autonomic nervous system analysis based on wavelets transform was designed and implemented on a server. A central design of affective recognition and biofeedback system is described.Fundação para a Ciência e a Tecnologia (FCT

A Closed-Loop Perspective on Symbiotic Human-Computer Interaction

This paper is concerned with how people interact with an emergent form of technology that is capable of both monitoring and affecting the psychology and behaviour of the user. The current relationship between people and computer is characterised as asymmetrical and static. The closed-loop dynamic of physiological computing systems is used as an example of a symmetrical and symbiotic HCI, where the central nervous system of the user and an adaptive software controller are engaged in constant dialogue. This emergent technology offers several benefits such as: intelligent adaptation, a capacity to learn and an ability to personalise software to the individual. This paper argues that such benefits can only be obtained at the cost of a strategic reconfiguration of the relationship between people and technology - specifically users must cede a degree of control over their interaction with technology in order to create an interaction that is active, dynamic and capable of responding in a stochastic fashion. The capacity of the system to successfully translate human goals and values into adaptive responses that are appropriate and effective at the interface represents a particular challenge. It is concluded that technology can develop lifelike qualities (e.g. complexity, sentience, freedom) through sustained and symbiotic interaction with human beings. However, there are a number of risks associated with this strategy as interaction with this category of technology can subvert skills, self-knowledge and the autonomy of human user

Interpretation of Physiological Indicators of Motivation: Caveats and Recommendations

Motivation scientists employing physiological measures to gather information about motivation-related states are at risk of committing two fundamental errors: overstating the inferences that can be drawn from their physiological measures and circular reasoning. We critically discuss two complementary approaches, Cacioppo and colleagues’ model of psychophysiological relations and construct validation theory, to highlight the conditions under which these errors are committed and provide guidance on how to avoid them. In particular, we demonstrate that the direct inference from changes in a physiological measure to changes in a motivation-related state requires the demonstration that the measure is not related to other relevant psychological states. We also point out that circular reasoning can be avoided by separating the definition of the motivation-related state from the hypotheses that are empirically tested

Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (ACE)

[Jill] I don’t know what happened. [Chris] Barry. Where’s Barry? So opens the mansion scene to Capcom’s survival-horror Resident Evil (Capcom, 1996) – and with it one of the gaming world’s first tentative steps toward realisation of the emotionally-immersive, narrative cinematic experience. In this paper we describe the fundamentals of affective gaming; covering their origins, how they operate, some examples, an-in-depth analysis of one of our early affective games (Gilleade & Allanson, 2002), their current capabilities and the ongoing research to develop them further. We also explore a new approach to game design based on three high-level design heuristics: assist me, challenge me and emote me (ACE), a series of gameplay "tweaks" made possible through affective videogames. We are emotionally-creatures. If affect is not conveyed properly during game play (e.g. if Resident Evil’s ability to inspire fear in the player was non-existent), the player’s suspension of disbelief can be negatively affected and the movie-inspired immersive experience is spoiled. Advances in computation and memory capabilities mean that videogames are more than capable of conveying affect just as well as traditional media (e.g. film, books). As a result games are becoming more reliant on the imagination of game designers for their affective material rather than the constraints of the currently available technology. But the interactive nature of the videogame allows us to go one step further than traditional media. Unlike the latter; videogames are dynamic entities, they change according to how the player interacts with them. At the moment, these interactions are based purely on the input the player consciously decides to use in the game world (i.e. actions executed through the game controller). However these actions are not the only thing going on with the player during play; there are also the mostly unseen physiological responses that go on within the player’s body. Such responses are useful in identifying the current emotional state the player is in. If this information could be somehow collected and invested in the game dynamics; the affective bandwidth of future games could be increased (i.e. bi-directional, game affects player, player affects game and so on) allowing for the emotive "tweaking" of conventional gaming experiences or the creation of whole new ones. There are two ways in which physiological responses have been used in gaming so far. The most obvious are biofeedback games (sometimes referred to as affective feedback) such as the Media Lab relax-to-win racing game (Bersak et al, 2001); where players consciously try to control their biological responses of which they are not normally consciously aware (e.g. heartbeat, skin response, blood pressure). Such games use biological sensors to influence game play, thus the player effectively controls the game via their control of their own internal bodily functions. A variant of this is the skin-response based videogame created by Future University-Hakodate (Sakurazawa et al, 2004) where onlookers attempt to influence the physiological state of the player (i.e. provoking flight or fight responses through loud noises such as clapping) which then affects the game play (i.e. makes its more difficult, the player would attempt to exert conscious control over their biological responses to avoid getting into further difficulty). The other use of physiological data is for truly affective gaming, a derivation of Affective Computing (Picard, 1997). These games use the player’s own physiology to assess their current emotional state; this information is then used to manipulate gameplay in some prescribed manner in order to create more engaging and / or immersive entertainment experiences. The player may not even be aware that their physiological state is being sensed, the intention is to capture their normal affective reactions. In previous work on affective games (Gilleade & Allanson); we used the player’s heart rate to control the difficulty of a conventional videogame. Whenever game play was deemed too boring or overly exciting (i.e. represented as a decrease or increase in heartbeat rate respectively) the videogame would alter play to reverse the player’s affective state to keep within an optimum range. In the full paper we will describe these two classes in more detail and also introduce a more complete classification and discussion of affective gaming. Based on this analysis of other affective games and our own experience of the design of affective games, we propose several high level design heuristics for affective gaming, which we will explore further in the paper: • Assist me: Games that; identify player frustrations to which the game offers assistance through the current gaming context. • Challenge me: Games that; identify the player’s state of enjoyment in relation to the current challenge being offered to which the game compensates for if the challenge is to be found lacking. • Emote me: Games that; identify player responses to intentional emotional provoking content to which the game manipulates subsequent related content in respect to the recorded response. References: ----------- Gilleade, K., Allanson, J. (2003). A Toolkit for Exploring Affective Interface Adaptation in Videogames. Proceeding of HCI International 2003, volume 2. LEA, New Jersey, pages 370-374. Bersak, D., McDarby, G., Augenblick, N., McDarby, P., McDonnell, D., McDonal, B., Karkun, R. Biofeedback using an Immersive Competitive Environment. Online Proceedings for the Designing Ubiquitous Computing Games Workshop, Ubicomp 2001. Sakurazawa, S., Yoshida, N., Munekata, N. (2004). Entertainment Feature of a Game Using Skin Conductance Response. Proceedings of ACE 2004, Advances in Computer Entertainment Technology, ACM Press, pages 181-186. Picard, R. Affective Computing. MIT Press (1997)

Enhancing video game performance through an individualized biocybernetic system

Biocybernetic systems are physiological software systems that explicitly utilize physiological signals to control or adapt software functionality (Pope et al., 1995.) These systems have tremendous potential for innovation in human computer interaction by using physiological signals to infer a user\u27s emotional and mental states (Allanson & Fairclough, 2004; Fairclough, 2008). Nevertheless, development of these systems has been ultimately hindered by two fundamental challenges. First, these systems make generalizations about physiological indicators of cognitive states across populations when, in fact, relationships between physiological responses and cognitive states are specific to each individual (Andreassi, 2006). Second, they often employ largely inconsistent retrospective techniques to subjectively infer user\u27s mental state (Fairclough, 2008). An individualized biocybernetic system was developed to address the fundamental challenges of biocybernetic research. This system was used to adapt video game difficulty through real-time classifications of physiological responses to subjective appraisals. A study was conducted to determine the system\u27s ability to improve player\u27s performance. The results provide evidence of significant task performance increase and higher attained task difficulty when players interacted with the game using the system than without. This work offers researchers with an alternative method for software adaptation by conforming to the individual characteristics of each user

Towards a NeuroIS Research Methodology: Intensifying the Discussion on Methods, Tools, and Measurement

The genesis of the Neuro-Information Systems (NeuroIS) field took place in 2007. Since then, a considerable number of IS scholars and academics from related disciplines have started to use theories, methods, and tools from neuroscience and psychophysiology to better understand human cognition, emotion, and behavior in IS contexts, and to develop neuro-adaptive information systems (i.e., systems that recognize the physiological state of the user and that adapt, based on that information, in real-time). However, because the NeuroIS field is still in a nascent stage, IS scholars need to become familiar with the methods, tools, and measurements that are used in neuroscience and psychophysiology. Against the background of the increased importance of methodological discussions in the NeuroIS field, the Journal of the Association for Information Systems published a special issue call for papers entitled “Methods, tools, and measurement in NeuroIS research” in 2012. We, the special issue’s guest editors, accepted three papers after a stringent review process, which appear in this special issue. In addition to these three papers, we hope to intensify the discussion on NeuroIS research methodology, and to this end we present the current paper. Importantly, our observations during the review process (particularly with respect to methodology) and our own reading of the literature and the scientific discourse during conferences served as input for this paper. Specifically, we argue that six factors, among others that will become evident in future discussions, are critical for a rigorous NeuroIS research methodology; namely, reliability, validity, sensitivity, diagnosticity, objectivity, and intrusiveness of a measurement instrument. NeuroIS researchers—independent from whether their role is editor, reviewer, or author—should carefully give thought to these factors. We hope that the discussion in this paper instigates future contributions to a growing understanding towards a NeuroIS research methodology

Dynamic Threshold Selection for a Biocybernetic Loop in an Adaptive Video Game Context

Passive Brain-Computer interfaces (pBCIs) are a human-computer communication tool where the computer can detect from neurophysiological signals the current mental or emotional state of the user. The system can then adjust itself to guide the user toward a desired state. One challenge facing developers of pBCIs is that the system's parameters are generally set at the onset of the interaction and remain stable throughout, not adapting to potential changes over time such as fatigue. The goal of this paper is to investigate the improvement of pBCIs with settings adjusted according to the information provided by a second neurophysiological signal. With the use of a second signal, making the system a hybrid pBCI, those parameters can be continuously adjusted with dynamic thresholding to respond to variations such as fatigue or learning. In this experiment, we hypothesize that the adaptive system with dynamic thresholding will improve perceived game experience and objective game performance compared to two other conditions: an adaptive system with single primary signal biocybernetic loop and a control non-adaptive game. A within-subject experiment was conducted with 16 participants using three versions of the game Tetris. Each participant plays 15 min of Tetris under three experimental conditions. The control condition is the traditional game of Tetris with a progressive increase in speed. The second condition is a cognitive load only biocybernetic loop with the parameters presented in Ewing et al. (2016). The third condition is our proposed biocybernetic loop using dynamic threshold selection. Electroencephalography was used as the primary signal and automatic facial expression analysis as the secondary signal. Our results show that, contrary to our expectations, the adaptive systems did not improve the participants' experience as participants had more negative affect from the BCI conditions than in the control condition. We endeavored to develop a system that improved upon the authentic version of the Tetris game, however, our proposed adaptive system neither improved players' perceived experience, nor their objective performance. Nevertheless, this experience can inform developers of hybrid passive BCIs on a novel way to employ various neurophysiological features simultaneously