Physiopad: development of a non-invasive game controller toolkit to study physiological responses for Game User Research

Os jogos afectivos usam as respostas fisiológicas do jogador para criar um ambiente adequado ao estado emocional do utilizador. A investigação destes jogos tem sido explorada nos últimos anos. Estas experiências, contudo, ainda requerem sistemas complexos e difíceis de utilizar. Nesta dissertação, é proposta a construção de um dispositivo capaz de ler dados fisiológicos de forma não invasiva e que seja de fácil utilização. Este aparelho faz a leitura do ritmo cardíaco e dos níveis de excitação do jogador, além disso foi criado um software para interligar com o dispositivo. Utilizando um comando da PlayStation 3 e um BITalino, o dispositivo é capaz de fazer a aquisição do sinal PPG e sinal EDA durante o jogo. O software analisa os sinais do comando, calcula o ritmo cardíaco e mede os níveis de excitação em tempo real. Foi realizada uma experiência utilizando biofeedback positivo e negativo, com o objectivo de testar a integração entre o software e o hardware. Não será no imediato que os dispositivos deste género sejam disponibilizados comercialmente. Os resultados são, no entanto, promissores. O cálculo do ritmo cardíaco em tempo real tem apenas uma diferença de 5 batimentos por minuto em relação ao ritmo cardíaco real do jogador. Apesar de os testes com o EDA serem inconclusivos, pode-se verificar que foi possível construir um sistema para ler os dados fisiológicos sendo mais económico do que os seus pares, sem comprometer a fiabilidade dos dados.Affective games are a genre of games that use the physiological responses from the player to adapt the gameplay to a more enjoyable emotional state and experience. Physiological responses and affective games have been studied vastly over the years. However, the setups used in these interventions are very intrusive and are complex to set up. In this project, it is purposed to build a non-invasive and easy-to-set-up toolkit that records physiological data. This toolkit records the player's heart rate and arousal levels and was decomposed into software and hardware. Using a PS3 game controller replica and a BITalino, a physiological game controller which can record heart rate and arousal during gameplay was built. The software interfaces with the gamepad, processes the physiological signals and sends this information to the game. An experiment with a positive biofeedback condition and negative biofeedback condition was conducted. This experiment showed that even though more work must be done until these type of devices could be commercially available, the results are promising. This toolkit’s heart rate values, when compared with other more traditional acquisition devices, were very similar, being on average only 5 BMP lower than the actual heart rate, proving that is possible to build more affordable non-invasive physiological hardware without compromising the signal's accuracy

Machine learning methods for the study of cybersickness: a systematic review

This systematic review offers a world-first critical analysis of machine learning methods and systems, along with future directions for the study of cybersickness induced by virtual reality (VR). VR is becoming increasingly popular and is an important part of current advances in human training, therapies, entertainment, and access to the metaverse. Usage of this technology is limited by cybersickness, a common debilitating condition experienced upon VR immersion. Cybersickness is accompanied by a mix of symptoms including nausea, dizziness, fatigue and oculomotor disturbances. Machine learning can be used to identify cybersickness and is a step towards overcoming these physiological limitations. Practical implementation of this is possible with optimised data collection from wearable devices and appropriate algorithms that incorporate advanced machine learning approaches. The present systematic review focuses on 26 selected studies. These concern machine learning of biometric and neuro-physiological signals obtained from wearable devices for the automatic identification of cybersickness. The methods, data processing and machine learning architecture, as well as suggestions for future exploration on detection and prediction of cybersickness are explored. A wide range of immersion environments, participant activity, features and machine learning architectures were identified. Although models for cybersickness detection have been developed, literature still lacks a model for the prediction of first-instance events. Future research is pointed towards goal-oriented data selection and labelling, as well as the use of brain-inspired spiking neural network models to achieve better accuracy and understanding of complex spatio-temporal brain processes related to cybersickness

The influence of implicit and explicit biofeedback in first-person shooter games

To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing how biofeedback can enhance human–computer interaction. In our two-part study, (N=36) subjects first played eight different game stages with two implicit biofeedback conditions, with two simulation-based comparison and repetition rounds, then repeated the two biofeedback stages when given explicit information on the biofeedback. The biofeedback conditions were respiration and skin-conductance (EDA) adaptations. Adaptation targets were four balanced player avatar attributes. We collected data with psychophysiological measures (electromyography, respiration, and EDA), a game experience questionnaire, and game-play measures. According to our experiment, implicit biofeedback does not produce significant effects in player experience in an FPS game. In the explicit biofeedback conditions, players were more immersed and positively affected, and they were able to manipulate the game play with the biosignal interface. We recommend exploring the possibilities of using explicit biofeedback interaction in commercial games. Author Keywords Games, playing, affective computing, biosignals

Clinical Decision Support Systems with Game-based Environments, Monitoring Symptoms of Parkinson’s Disease with Exergames

Parkinson’s Disease (PD) is a malady caused by progressive neuronal degeneration, deriving in several physical and cognitive symptoms that worsen with time. Like many other chronic diseases, it requires constant monitoring to perform medication and therapeutic adjustments. This is due to the significant variability in PD symptomatology and progress between patients. At the moment, this monitoring requires substantial participation from caregivers and numerous clinic visits. Personal diaries and questionnaires are used as data sources for medication and therapeutic adjustments. The subjectivity in these data sources leads to suboptimal clinical decisions. Therefore, more objective data sources are required to better monitor the progress of individual PD patients. A potential contribution towards more objective monitoring of PD is clinical decision support systems. These systems employ sensors and classification techniques to provide caregivers with objective information for their decision-making. This leads to more objective assessments of patient improvement or deterioration, resulting in better adjusted medication and therapeutic plans. Hereby, the need to encourage patients to actively and regularly provide data for remote monitoring remains a significant challenge. To address this challenge, the goal of this thesis is to combine clinical decision support systems with game-based environments. More specifically, serious games in the form of exergames, active video games that involve physical exercise, shall be used to deliver objective data for PD monitoring and therapy. Exergames increase engagement while combining physical and cognitive tasks. This combination, known as dual-tasking, has been proven to improve rehabilitation outcomes in PD: recent randomized clinical trials on exergame-based rehabilitation in PD show improvements in clinical outcomes that are equal or superior to those of traditional rehabilitation. In this thesis, we present an exergame-based clinical decision support system model to monitor symptoms of PD. This model provides both objective information on PD symptoms and an engaging environment for the patients. The model is elaborated, prototypically implemented and validated in the context of two of the most prominent symptoms of PD: (1) balance and gait, as well as (2) hand tremor and slowness of movement (bradykinesia). While balance and gait affections increase the risk of falling, hand tremors and bradykinesia affect hand dexterity. We employ Wii Balance Boards and Leap Motion sensors, and digitalize aspects of current clinical standards used to assess PD symptoms. In addition, we present two dual-tasking exergames: PDDanceCity for balance and gait, and PDPuzzleTable for tremor and bradykinesia. We evaluate the capability of our system for assessing the risk of falling and the severity of tremor in comparison with clinical standards. We also explore the statistical significance and effect size of the data we collect from PD patients and healthy controls. We demonstrate that the presented approach can predict an increased risk of falling and estimate tremor severity. Also, the target population shows a good acceptance of PDDanceCity and PDPuzzleTable. In summary, our results indicate a clear feasibility to implement this system for PD. Nevertheless, long-term randomized clinical trials are required to evaluate the potential of PDDanceCity and PDPuzzleTable for physical and cognitive rehabilitation effects

An experimental protocol to access immersiveness in video games

In the video game industry, great importance is given to the experience that the user has while playing a game. In particular, this experience benefits from the players' perceived sense of being in the game or immersion. The level of user immersion depends not only on the game's content but also on how the game is displayed, thus on its User Interface (UI) and the Head's-Up Display (HUD). Another factor influencing immersiveness that has been found in the literature is the player's expertise: the more experience the user has with a specific game, the less they need information on the screen to be immersed in the game. Player's level of immersion can be accessed by using both questionnaires of their perceived experience and exploiting their behavioural and physiological responses while playing the target game. Therefore, in this paper, we propose an experimental protocol to access immersiveness of gamers while playing a third-person shooter (Fortnite) with UIs with a standard, a dietetic, and a proposed HUD. A subjective evaluation of the immersion will be provided by completing the Immersive Experience Questionnaire (IEQ), while objective indicators will be provided by face tracking, behaviour and physiological responses analyses. The ultimate goal of this study is to define guidelines for video game UI development that can enhance the players' immersion.Comment: Accepted at the Italian Workshop on Artificial Intelligence for Human Machine Interaction (AIxHMI 2023), November 06, 2023, Rome, Ital

Psychophysiological Measures of Cognitive Absorption

Cognitive absorption (CA) corresponds to a state of deep involvement with a software program. CA has widely been studied over the last decade in the IT literature using psychometric instruments. Measuring ongoing CA with psychometric tools requires interrupting a subject’s ongoing usage behavior to self-evaluate their level of absorption. Such interruptions may alter or contaminate the very CA state the researcher us attempting to measure. To circumvent this problem, we are investigating the effectiveness of psychophysiological measures of cognitive absorption. This paper reports preliminary results from an ongoing research project by looking at the correlation between electrodermal activity (EDA) and several dimensions of the CA construct

Understanding and designing avatar biosignal visualizations for social Virtual Reality entertainment

Visualizing biosignals can be important for social Virtual Reality (VR), where avatar non-verbal cues are missing. While several biosignal representations exist, designing effective visualizations and understanding user perceptions within social VR entertainment remains unclear. We adopt a mixed-methods approach to design biosignals for social VR entertainment. Using survey (N=54), context-mapping (N=6), and co-design (N=6) methods, we derive four visualizations. We then ran a within-subjects study (N=32) in a virtual jazz-bar to investigate how heart rate (HR) and breathing rate (BR) visualizations, and signal rate, influence perceived avatar arousal, user distraction, and preferences. Findings show that skeuomorphic visualizations for both biosignals allow differentiable arousal inference; skeuomorphic and particles were least distracting for HR, whereas all were similarly distracting for BR; biosignal perceptions often depend on avatar relations, entertainment type, and emotion inference of avatars versus spaces. We contribute HR and BR visualizations, and considerations for designing social VR entertainment biosignal visualizations

Total immersion : Designing for affective symbiosis in a virtual reality game with haptics, biosensors, and emotive agents

Peer reviewe