Measurement with Persons: A European Network

The European ‘Measuring the Impossible’ Network MINET promotes new research activities in measurement dependent on human perception and/or interpretation. This includes the perceived attributes of products and services, such as quality or desirability, and societal parameters such as security and well-being. Work has aimed at consensus about four ‘generic’ metrological issues: (1) Measurement Concepts & Terminology; (2) Measurement Techniques: (3) Measurement Uncertainty; and (4) Decision-making & Impact Assessment, and how these can be applied specificallyto the ‘Measurement of Persons’ in terms of ‘Man as a Measurement Instrument’ and ‘Measuring Man.’ Some of the main achievements of MINET include a research repository with glossary; training course; book; series of workshops;think tanks and study visits, which have brought together a unique constellation of researchers from physics, metrology,physiology, psychophysics, psychology and sociology. Metrology (quality-assured measurement) in this area is relativelyunderdeveloped, despite great potential for innovation, and extends beyond traditional physiological metrology in thatit also deals with measurement with all human senses as well as mental and behavioral processes. This is particularlyrelevant in applications where humans are an important component of critical systems, where for instance health andsafety are at stake

Kognition arviointi : Hajautetun tiedon sovelluksia

The amount of information collected by personal health records, smartphone ecosystems, and other cloud services has increased enormously in recent years. This has, for instance, led to new ways of automated physical exercise assessment, but this also introduces the potential for novel methods and applications in the automated evaluation of various mental factors such as cognitive state and stress. Extracting such latent variables holds considerable promise, in particular in group-level analysis. Furthermore, the current initiatives and research programs collecting masses of health data from large cohorts create opportunities for developing the methodology. The lack of targeted research is partially hindering the development of the analysis of obscure factors, progress of machine learning and other algorithmic solutions, and the evolution of novel applications and technologies. As described in this introduction, various features inherent in biosignals increase the complexity in the research. In this thesis I provide an introduction to the emerging ubiquitous recording of physiological signals, its effects on the industry, opportunities for organizations and management, and data analytics and measurement techniques. The aim is to seek the future prospects of systemic scenarios and large-scale initiatives. The original publications reviewed in this thesis seek biosignals for features responsive for cognitive states such as stress and, more interestingly, second-order factors derived from inter-individual responses and activations. By introducing more complex features to psychophysiological research, group analytics can be further developed. Second-order analyses provide robust signal features and may lead to advanced applications in assessing well-being and performance. The original publications consist of three research articles and a primer review. The experiments include recordings of magnetoencephalography (MEG), heart rate variability (HRV), and electrodermal activity (EDA), and they exemplify systemic use cases of biosignals. The included primer review discusses general methods in psychophysiological research in human–computer interaction (HCI). Together with this introduction, my experimental results provide evidence that taking psychophysiological measurements from the laboratory to ecologically valid environments is plausible and effective. The literature suggests that consumer-grade devices and personal internet of things will revolutionize myriad sectors, e.g., organizational management. Together with an exponentially increasing data collection and novel applications, the industry will have large economical impacts.Henkilökohtaisen terveystiedon kerääminen ja tallennus on lisääntynyt valtavasti viime vuosina. Monet käyttävät tietoa esimerkiksi fyysisen harjoittelun tukena. Tämän lisäksi mitattua tietoa on alettu hyödyntää esimerkiksi stressitilojen tunnistamisessa. Tällaista fysiologisten signaalien arviointia kutsutaan psykofysiologiaksi. Jatkokehityksen avulla tällaiset piirteet sopivat varsinkin ryhmäanalyyseihin ja suurempien joukkojen arvioimiseen. Menetelmien kehitystä tukevat useat suuret väestötason tutkimusavaukset. Toisaalta juuri kohdennetun tutkimuksen puute osaltaan hidastaa tallennetusta tiedosta eristettävien piilevien piirteiden hyödyntämisen yleistymistä uusissa algoritmeissa ja sovel- luksissa. Tässä yhteenvedossa esittelen, mitkä asiat vaikuttavat osaltaan tähän kehitykseen. Esittelen fysiologisten signaalien mittaamisen taustoja, sekä mittausmenetelmien kehitystä. Lisäksi pohdin kaupallisten sovellusten mahdollisuuksia ja muita tulevaisuuden näkymiä. Johdanto-osuus toimii siten taustamateriaalina soveltavalle osiolle ja liitetyille osajulkaisuille. Osajulkaisut tutkivat kohdennetummin biosignaalien soveltuvuutta kognitiivisen toim- intakyvyn arvioimisessa. Jäljemmät julkaisut keskittyvät useiden yksilöiden biosignaalien kovarianssia hyödyntäviin menetelmiin. Tällaiset menetelmät luovat pohjaa kehittyneem- mille analyysitavoille ja signaalien yhä tehokkaammalle hyödyntämiselle hyvinvoinnin ja toimintakyvyn arvioinnissa. Kolme ensimmäistä osajulkaisua ovat kokeellisia tutkimusar- tikkeleita ja viimeinen on katsaus olemassa olevaan tutkimukseen. Tutkimusasetelmissa hyödynnetyt fysiologiset menetelmät ovat magnetoenkefalografia (MEG), sykevälivaihtelu (HRV) ja ihosähköinen vaste (EDA). Katsaus toisaalta tarkastelee psykofysiologian hyödyn- tämistä tietokoneen käyttöliittymätutkimuksessa (HCI). Yhdessä tämän yhteenvedon kanssa tutkimustulokset edistävät mittausmenetelmien hyödynnettävyyttä luonnollisissa ympäristöissä, sekä psykofysiologisten signaalien käyttöä vaihtelevissa ja kontrolloimattomissa olosuhteissa. Kirjallisuudesta löytyy viitteitä kulutta- jalaitteiden ja esineiden internetin kasvusta ja potentiaalista mullistaa useita sektoreita, kuten organisaatioiden ohjaus. Lähteet ennustavat myös markkinoiden kasvua. Yhdessä kaikkialle levittyvä tiedon kerääminen ja uudet sovellukset sekä datalähtöiset analyysimenetelmät voivat johtaa suuriin muutoksiin

Applications of Affective Computing in Human-Robot Interaction: state-of-art and challenges for manufacturing

The introduction of collaborative robots aims to make production more flexible, promoting a greater interaction between humans and robots also from physical point of view. However, working closely with a robot may lead to the creation of stressful situations for the operator, which can negatively affect task performance. In Human-Robot Interaction (HRI), robots are expected to be socially intelligent, i.e., capable of understanding and reacting accordingly to human social and affective clues. This ability can be exploited implementing affective computing, which concerns the development of systems able to recognize, interpret, process, and simulate human affects. Social intelligence is essential for robots to establish a natural interaction with people in several contexts, including the manufacturing sector with the emergence of Industry 5.0. In order to take full advantage of the human-robot collaboration, the robotic system should be able to perceive the psycho-emotional and mental state of the operator through different sensing modalities (e.g., facial expressions, body language, voice, or physiological signals) and to adapt its behaviour accordingly. The development of socially intelligent collaborative robots in the manufacturing sector can lead to a symbiotic human-robot collaboration, arising several research challenges that still need to be addressed. The goals of this paper are the following: (i) providing an overview of affective computing implementation in HRI; (ii) analyzing the state-of-art on this topic in different application contexts (e.g., healthcare, service applications, and manufacturing); (iii) highlighting research challenges for the manufacturing sector

Neural Network Driven Eye Tracking Metrics and Data Visualization in Metaverse and Virtual Reality Maritime Safety Training

Understand the human brain, predict human performance, and proactively plan, strategize and act based on such information initiated a scientific multidisciplinary alliance to address modern management challenges. This paper integrates numerous advanced information technologies such as eye tracking, virtual reality and neural networks for cognitive task analysis leading to behavioral analysis on humans that perform specific activities. The technology developed and presented in this paper has been tested on a maritime safety training application for command bridge communication and procedures for collision avoidance. The technology integrates metaverse and virtual reality environments with eye tracking for the collection of behavioral data which are analyzed by a neural network to indicate the mental and physical state, attention and readiness of a seafarer to perform such a critical task. The paper demonstrates the technology architecture, data collection process, indicative results, and areas for further research

PhysioVR: a novel mobile virtual reality framework for physiological computing

Virtual Reality (VR) is morphing into a ubiquitous technology by leveraging of smartphones and screenless cases in order to provide highly immersive experiences at a low price point. The result of this shift in paradigm is now known as mobile VR (mVR). Although mVR offers numerous advantages over conventional immersive VR methods, one of the biggest limitations is related with the interaction pathways available for the mVR experiences. Using physiological computing principles, we created the PhysioVR framework, an Open-Source software tool developed to facilitate the integration of physiological signals measured through wearable devices in mVR applications. PhysioVR includes heart rate (HR) signals from Android wearables, electroencephalography (EEG) signals from a low cost brain computer interface and electromyography (EMG) signals from a wireless armband. The physiological sensors are connected with a smartphone via Bluetooth and the PhysioVR facilitates the streaming of the data using UDP communication protocol, thus allowing a multicast transmission for a third party application such as the Unity3D game engine. Furthermore, the framework provides a bidirectional communication with the VR content allowing an external event triggering using a real-time control as well as data recording options. We developed a demo game project called EmoCat Rescue which encourage players to modulate HR levels in order to successfully complete the in-game mission. EmoCat Rescue is included in the PhysioVR project which can be freely downloaded. This framework simplifies the acquisition, streaming and recording of multiple physiological signals and parameters from wearable consumer devices providing a single and efficient interface to create novel physiologically-responsive mVR applications.info:eu-repo/semantics/publishedVersio

Sensing with Earables: A Systematic Literature Review and Taxonomy of Phenomena

Earables have emerged as a unique platform for ubiquitous computing by augmenting ear-worn devices with state-of-the-art sensing. This new platform has spurred a wealth of new research exploring what can be detected on a wearable, small form factor. As a sensing platform, the ears are less susceptible to motion artifacts and are located in close proximity to a number of important anatomical structures including the brain, blood vessels, and facial muscles which reveal a wealth of information. They can be easily reached by the hands and the ear canal itself is affected by mouth, face, and head movements. We have conducted a systematic literature review of 271 earable publications from the ACM and IEEE libraries. These were synthesized into an open-ended taxonomy of 47 different phenomena that can be sensed in, on, or around the ear. Through analysis, we identify 13 fundamental phenomena from which all other phenomena can be derived, and discuss the different sensors and sensing principles used to detect them. We comprehensively review the phenomena in four main areas of (i) physiological monitoring and health, (ii) movement and activity, (iii) interaction, and (iv) authentication and identification. This breadth highlights the potential that earables have to offer as a ubiquitous, general-purpose platform

Sensing with Earables: A Systematic Literature Review and Taxonomy of Phenomena

Earables have emerged as a unique platform for ubiquitous computing by augmenting ear-worn devices with state-of-the-art sensing. This new platform has spurred a wealth of new research exploring what can be detected on a wearable, small form factor. As a sensing platform, the ears are less susceptible to motion artifacts and are located in close proximity to a number of important anatomical structures including the brain, blood vessels, and facial muscles which reveal a wealth of information. They can be easily reached by the hands and the ear canal itself is affected by mouth, face, and head movements. We have conducted a systematic literature review of 271 earable publications from the ACM and IEEE libraries. These were synthesized into an open-ended taxonomy of 47 different phenomena that can be sensed in, on, or around the ear. Through analysis, we identify 13 fundamental phenomena from which all other phenomena can be derived, and discuss the different sensors and sensing principles used to detect them. We comprehensively review the phenomena in four main areas of (i) physiological monitoring and health, (ii) movement and activity, (iii) interaction, and (iv) authentication and identification. This breadth highlights the potential that earables have to offer as a ubiquitous, general-purpose platform

Brain Computer Interfaces and Emotional Involvement: Theory, Research, and Applications

This reprint is dedicated to the study of brain activity related to emotional and attentional involvement as measured by Brain–computer interface (BCI) systems designed for different purposes. A BCI system can translate brain signals (e.g., electric or hemodynamic brain activity indicators) into a command to execute an action in the BCI application (e.g., a wheelchair, the cursor on the screen, a spelling device or a game). These tools have the advantage of having real-time access to the ongoing brain activity of the individual, which can provide insight into the user’s emotional and attentional states by training a classification algorithm to recognize mental states. The success of BCI systems in contemporary neuroscientific research relies on the fact that they allow one to “think outside the lab”. The integration of technological solutions, artificial intelligence and cognitive science allowed and will allow researchers to envision more and more applications for the future. The clinical and everyday uses are described with the aim to invite readers to open their minds to imagine potential further developments

An Intra-Subject Approach Based on the Application of HMM to Predict Concentration in Educational Contexts from Nonintrusive Physiological Signals in Real-World Situations

Previous research has proven the strong influence of emotions on student engagement and motivation. Therefore, emotion recognition is becoming very relevant in educational scenarios, but there is no standard method for predicting students’ affects. However, physiological signals have been widely used in educational contexts. Some physiological signals have shown a high accuracy in detecting emotions because they reflect spontaneous affect-related information, which is fresh and does not require additional control or interpretation. Most proposed works use measuring equipment for which applicability in real-world scenarios is limited because of its high cost and intrusiveness. To tackle this problem, in this work, we analyse the feasibility of developing low-cost and nonintrusive devices to obtain a high detection accuracy from easy-to-capture signals. By using both inter-subject and intra-subject models, we present an experimental study that aims to explore the potential application of Hidden Markov Models (HMM) to predict the concentration state from 4 commonly used physiological signals, namely heart rate, breath rate, skin conductance and skin temperature. We also study the effect of combining these four signals and analyse their potential use in an educational context in terms of intrusiveness, cost and accuracy. The results show that a high accuracy can be achieved with three of the signals when using HMM-based intra-subject models. However, inter-subject models, which are meant to obtain subject-independent approaches for affect detection, fail at the same task.This research was partly supported by Spanish Ministry of Science, Innovation and Universities through projects PGC2018-096463-B-I00 and PGC2018-102279-B-I00 (MCIU/AEI/FEDER, UE)