State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Towards Balancing Fun and Exertion in Exergames: Exploring the Impact of Movement-Based Controller Devices, Exercise Concepts, Game Adaptivity and Player Modes on Player Experience and Training Intensity in Different Exergame Settings

Physical inactivity remains one of the biggest societal challenges of the 21st century. The gaming industry and the fitness sector have responded to this alarming fact by introducing game-based or gamified training scenarios and thus established the promising trend of exergaming. Exergames – games controlled by active (whole) body movements – have been extolled as potential attractive and effective training tools. However, the majority of the exergames do not meet the required intensity or effectiveness, nor do they induce the intended training adherence or long-term motivation. One reason for this is that the evaluated exergames were often not co-designed with the user group to meet their specific needs and preferences, nor were they co-designed with an interdisciplinary expert team of game designers (to ensure a good gaming experience) and sports scientists (for a great training experience). Accordingly, the research results from studies with these exergames are rather limited. To fully exploit the potential of these innovative movement tools and to establish them as attractive and effective training approach, it is necessary to understand and explore both the underlying interdisciplinary theories and concepts as well as possible design approaches and their impact on the game and training experience. This dissertation aims to contribute to a better understanding of well-balanced exergame design. It explores and evaluates how different movement-based control devices, exercise concepts, game adaptations, and player modes influence the attractiveness and effectiveness of exergames. The work provides theoretical and practical contributions to the problem area of effective and attractive exergames. For this purpose, a research and development (R&D) approach with iterative phases was followed. As preliminary work for the contributions of this dissertation, exergames were approached from a theoretical perspective. Underlying multidisciplinary theories and concepts of exergames from relevant fields were analyzed and a generic framework was built, which structured the findings based on three interdependent dimensions: the player, the game controller, and the virtual game scenario. Some commercially available exergames were explored to verify the theory-based assumption that the interposition of technology brings specific transformations in the coupling of perception and action that do not occur in real sports situations. Among other things, the comparative pilot study showed that two different controllers (one gesture-based and one haptic device), which allowed for different physical input, were likely to induce diverse gameplay experiences (e.g., higher feeling of flow and self-location when playing with the haptic device) with differently skilled players. However, certain design-specific differences in the two exergame conditions meant that these results could only be interpreted as a first trend. To overcome the limitations of this preliminary study approach (e.g., unequal game design of the commercial exergames and very sports-specific movement concept), Plunder Planet, an adaptive exergame environment, was iteratively designed with and for children and allowed for a single- and cooperative multiplayer experience with two different controller devices. The user-centered design was further informed by insights from the growing body of related R&D work in the field of exergames. The first study presented in this dissertation compared the subjectively experienced attractiveness and effectiveness of Plunder Planet when played with different motion-based controllers. Besides a generally great acceptance of the exergame, it was found that the haptic full-body motion controller provided physical guidance and a more cognitively and coordinatively challenging workout, which was more highly rated by experienced gamers with fewer athletic skills. The gesture-based Kinect sensor felt more natural, allowed more freedom of movement, and provided a rather physically intense but cognitively less challenging workout, which was more highly rated by athletic players with less gameplay experience. Furthermore, experiments were made with an exploratory adaptive algorithm that enabled the cognitive and the physical challenge of the exergame to be manually adapted in real-time based on the player’s fitness and gaming skills. The first and the second study also compared an adaptive with a non-adaptive single player version of Plunder Planet. It could be shown that the (well-balanced) adaptive version of the exergame was better valued than the non-adaptive version with regard to the experienced and measured attractiveness (motivation, game flow, spatial presence experience, balance of cognitive and physical challenge) and effectiveness (heart rate, physical exertion, balance of cognitive and physical challenge) by differently skilled players. Finally, and contrary to the findings from related work, the results of the third study proved that the specifically designed controller technology could be used as an “enabler”, “supporter” and “shaper” of bodily interplay in social exergaming. Based on these promising findings, the goal became to further explore the effectiveness of exergames, refine the adaptive game difficulty algorithm, and explore further attractiveness- and motivation-boosting design approaches. Therefore, the ExerCube, a physically immersive and adaptive fitness game setting, was developed. It was iteratively designed with and for adults and allowed for cooperatively and competitive exergame experiences. With its physically immersive game setup, the ExerCube combines a mixed version of the advantages of both previously tested controllers. A coordinatively and cognitively challenging functional workout protocol with scalable intensity (moderate to high) was developed and the subjective experience of the ExerCube training was compared with a conventional functional training with a personal trainer. The fourth study showed that the game-based training gave signs of reaching a similar intensity to the personal training, but was more highly rated for flow, motivation, and enjoyment. Based on this exploratory comparison of the ExerCube with a personal trainer session, valuable avenues for further design could be identified. Among other things, it could be proved that the player’s focus during the ExerCube session was more on the game than on the own body. Players experienced stronger physical exertion and social pressure with the personal trainer and a stronger cognitive exertion and involvement with the ExerCube. Furthermore, a refined version of the previously tested adaptive game difficulty algorithm was implemented and automated for the first time for purpose of this study. Again it was shown that the adaptive version had benefits with regard to subjectively experienced attractiveness (motivation, game flow, balance of cognitive and physical challenge) and effectiveness (physical exertion, balance of cognitive and physical challenge) compared to the non-adaptive version. In order to further enhance the gaming experience, experiments were also conducted with sound designs and an adaptive audio design with adaptive background music and sound feedback was implemented. It was found to be a promising and beneficial add-on for a user-centered attractive exergame design. To inform the design of a multiplayer version of the ExerCube, different social play mechanics were explored in the fifth study. This resulted in differently balanced experiences of fun, and in physical as well as cognitive exertion. As the preliminary comparative evaluation of the subjectively experienced effectiveness and attractiveness of an ExerCube session and a personal trainer session could prove the general feasibility of the concept and revealed the first indications of the intensity of the ExerCube’s training concept, the objectively measured effectiveness of a single ExerCube session with a functional high-intensity interval training (fHIIT) with a personal trainer was compared in a final sixth study, and after another design iteration. Again, the subjectively experienced attractiveness of both conditions was assessed. It could be shown that the ExerCube is a feasible training device for training at fHIIT-level. While physical exertion was slightly lower than in the conventional fHIIT condition, the ExerCube condition’s average heart rate values reached the fHIIT threshold and also yielded significantly better results for flow, enjoyment, and motivation. The ExerCube training also resulted in a subjectively experienced higher cognitive load (dual-domain training). To sum up, it can be stated that this dissertation provides valuable and fundamental research contributions to the promising field of exergames as attractive and effective training tools. Furthermore, important contributions to design questions in this field could be developed. Since this field is still relatively unexplored, the work presented creates a sound basis for future R&D work in this area

State of the Art of Audio- and Video-Based Solutions for AAL

It is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters. Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals. Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely lifelogging and self-monitoring, remote monitoring of vital signs, emotional state recognition, food intake monitoring, activity and behaviour recognition, activity and personal assistance, gesture recognition, fall detection and prevention, mobility assessment and frailty recognition, and cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed

Proceedings of the 9th international conference on disability, virtual reality and associated technologies (ICDVRAT 2012)

The proceedings of the conferenc