Prerequisites for Affective Signal Processing (ASP) - Part V: A response to comments and suggestions

In four papers, a set of eleven prerequisites for affective signal processing (ASP) were identified (van den Broek et al., 2010): validation, triangulation, a physiology-driven approach, contributions of the signal processing community, identification of users, theoretical specification, integration of biosignals, physical characteristics, historical perspective, temporal construction, and real-world baselines. Additionally, a review (in two parts) of affective computing was provided. Initiated by the reactions on these four papers, we now present: i) an extension of the review, ii) a post-hoc analysis based on the eleven prerequisites of Picard et al.(2001), and iii) a more detailed discussion and illustrations of temporal aspects with ASP

Ubiquitous emotion-aware computing

Emotions are a crucial element for personal and ubiquitous computing. What to sense and how to sense it, however, remain a challenge. This study explores the rare combination of speech, electrocardiogram, and a revised Self-Assessment Mannequin to assess people’s emotions. 40 people watched 30 International Affective Picture System pictures in either an office or a living-room environment. Additionally, their personality traits neuroticism and extroversion and demographic information (i.e., gender, nationality, and level of education) were recorded. The resulting data were analyzed using both basic emotion categories and the valence--arousal model, which enabled a comparison between both representations. The combination of heart rate variability and three speech measures (i.e., variability of the fundamental frequency of pitch (F0), intensity, and energy) explained 90% (p < .001) of the participants’ experienced valence--arousal, with 88% for valence and 99% for arousal (ps < .001). The six basic emotions could also be discriminated (p < .001), although the explained variance was much lower: 18–20%. Environment (or context), the personality trait neuroticism, and gender proved to be useful when a nuanced assessment of people’s emotions was needed. Taken together, this study provides a significant leap toward robust, generic, and ubiquitous emotion-aware computing

Virtuality Supports Reality for e-Health Applications

Strictly speaking the word “virtuality” or the expression “virtual reality” refers to an application for things simulated or created by the computer, which not really exist. More and more often such things are becoming equally referred with the adjective “virtual” or “digital” or mentioned with the prefixes “e-” or “cyber-”. So we know, for instance, of virtual or digital or e- or cyber- community, cash, business, greetings, books .. till even pets. The virtuality offers interesting advantages with respect to the “simple” reality, since it can reproduce, augment and even overcome the reality. The reproduction is not intended as it has been so far that a camera films a scenario from a fixed point of view and a player shows it, but today it is possible to reproduce the scene dynamically moving the point of view in practically any directions, and “real” becomes “realistic”. The virtuality can augment the reality in the sense that graphics are pulled out from a television screen (or computer/laptop/palm display) and integrated with the real world environments. In this way useful, and often in somehow essentials, information are added for the user. As an example new apps are now available even for iphone users who can obtain graphical information overlapped on camera played real scene surroundings, so directly reading the height of mountains, names of streets, lined up of satellites .., directly over the real mountains, the real streets, the real sky. But the virtuality can even overcome reality, since it can produce and make visible the hidden or inaccessible or old reality and even provide an alternative not real world. So we can virtually see deeply into the matter till atomic dimensions, realize a virtual tour in a past century or give visibility to hypothetical lands otherwise difficult or impossible to simple describe. These are the fundamental reasons for a naturally growing interest in “producing” virtuality. So here we will discuss about some of the different available methods to “produce” virtuality, in particular pointing out some steps necessary for “crossing” reality “towards” virtuality. But between these two parallel worlds, as the “real” and the “virtual” ones are, interactions can exist and this can lead to some further advantages. We will treat about the “production” and the “interaction” with the aim to focus the attention on how the virtuality can be applied in biomedical fields, since it has been demonstrated that virtual reality can furnish important and relevant benefits in e-health applications. As an example virtual tomography joins together 3D imaging anatomical features from several CT (Computerized axial Tomography) or MRI (Magnetic Resonance Imaging) images overlapped with a computer-generated kinesthetic interface so to obtain a useful tool in diagnosis and healing. With the new endovascular simulation possibilities, a head mounted display superimposes 3D images on the patient’s skin so to furnish a direction for implantable devices inside blood vessels. Among all, we chose to investigate the fields where we believe the virtual applications can furnish the meaningful advantages, i.e. in surgery simulation, in cognitive and neurological rehabilitation, in postural and motor training, in brain computer interface. We will furnish to the reader a necessary partial but at the same time fundamental view on what the virtual reality can do to improve possible medical treatment and so, at the end, resulting a better quality of our life

A Flexible Multiring Concentric Electrode for Non-Invasive Identification of Intestinal Slow Waves

[EN] Developing new types of optimized electrodes for specific biomedical applications can substantially improve the quality of the sensed signals. Concentric ring electrodes have been shown to provide enhanced spatial resolution to that of conventional disc electrodes. A sensor with different electrode sizes and configurations (monopolar, bipolar, etc.) that provides simultaneous records would be very helpful for studying the best signal-sensing arrangement. A 5-pole electrode with an inner disc and four concentric rings of different sizes was developed and tested on surface intestinal myoelectrical recordings from healthy humans. For good adaptation to a curved body surface, the electrode was screen-printed onto a flexible polyester substrate. To facilitate clinical use, it is self-adhesive, incorporates a single connector and can perform dry or wet (with gel) recordings. The results show it to be a versatile electrode that can evaluate the optimal configuration for the identification of the intestinal slow wave and reject undesired interference. A bipolar concentric record with an outer ring diameter of 30 mm, a foam-free adhesive material, and electrolytic gel gave the best results.Grant from the Ministerio de Economia y Competitividad y del Fondo Europeo de Desarrollo Regional. DPI2015-68397-R (MINECO/FEDER).Zena-Giménez, VF.; Garcia Casado, FJ.; Ye Lin, Y.; Garcia-Breijo, E.; Prats-Boluda, G. (2018). A Flexible Multiring Concentric Electrode for Non-Invasive Identification of Intestinal Slow Waves. Sensors. 18(2):396-412. https://doi.org/10.3390/s18020396S39641218

Unveiling the impact of neuromotor disorders on speech: a structured approach combining biomechanical fundamentals and statistical machine learning

Speech has been shown to convey clinically useful information in the study of Neurodegenerative Disorders (NDs), such as Parkinson’s Disease (PD). Traditionally the use of speech as an exploratory tool in People with Parkinson’s (PwP) has focused on the estimation of acoustic characteristics and their study at face value, analysing the physio-acoustical markers and using them as features for the differentiation between Healthy Controls (HC) and PwP. The present work takes a step further, given the intricate interoperation between neuromotor activity, responsible for both planning and driving the system, and the production of the acoustic speech signal; by the study of speech, this relationship may be properly exploited and analysed, providing a non-invasive method for the diagnosis, analysis, and observation of NDs. This work aims to introduce a working model that is capable of linking both domains and serves as a projection tool to provide insights about a speaker’s neuromotor state. This is based on a review of the neurophysiological background of the structure and function of the nervous system, and a review of the main nervous system dysfunctions involved in PD and other related neuromotor disorders. The role of the respiratory, phonatory, and articulatory systems is reviewed in the production of voice and speech under normal and pathological circumstances. This setting might allow for speech to be considered a useful trait within the precision medicine framework, as it provides a personal biometric marker that is innate and easy to elicit, can be recorded remotely with inexpensive equipment, is non-invasive, cost-effective, and easy to process. The problem can be divided into two main categories: firstly, a binary detection task distinguishing between healthy controls and individuals with NDs based on the projection model and phonatory estimates; secondly, a progression and tracking task providing a set of quantitative indices that enable clinically interpretable scores. This study aims to define a set of features and models that help to characterise hypokinetic dysarthria (HD). These incorporate the neuroscientific knowhow semantically and quantitatively to be used in clinical decision support tools that provide mechanistic insight on the processes involved in speech production, incorporating into the algorithmic element neuromotor considerations that add to better interpretability, consequently leading to improved clinical decisions and diagnosis. An overview of the acoustic signal processing algorithms for use in speech articulation and phonation system inversion regarding neuromotor disorder assessment is provided. An algorithmic methodology for model inversion and exploration has been proposed for the functional characterization and system inversion of each subsystem involved under the neuro-biomechanical foundations exposed before. A description of the vocal fold biomechanics using the glottal source, and formant dynamics provides the base for specific mapping to articulation kinematics. The statistical methods used in performance evaluation are based on three-way comparisons and transversal and longitudinal assessment by classical hypothesis testing. Three related experimental studies are shown to empirically illustrate the potential of phonation and articulation analysis: the characterization of PD from glottal biomechanics based on the amplitude distributions of the glottal flow and on the vocal fold body stiffness in assessing the efficiency of transcranial magnetic stimulation, and the description of PD dysarthria through an articulation projection model. The results from the biomechanical analysis of phonation showed that the behaviour of glottal source amplitude distributions from PD and healthy controls using three-way comparisons and hierarchical clustering were essentially distinguishable from those from normative young participants with the best accuracy scores produced by SVM classifiers of 94.8% (males) and 92.2% (females). Nevertheless, PD participants were barely separable from age-matched controls, possibly pointing to confounding factors due to age. The outcomes from using vocal fold stiffness in assessing the efficiency of transcranial magnetic stimulation showed mixed results, as some PD participants reflected clear improvements in phonation stability after stimulation, whereas some others did not. Some cases of sham controls experienced also minor improvements of unknown origin, possibly expressing a placebo effect. The overall results on the efficiency of stimulation showed an accuracy global score of 67% over the 18 cases studied. The results from articulation projection modelling showed the possibility of formulating personalised models for PD and control participants to transform acoustic formant dynamics into articulation kinematics. This might open the possibility of characterising PD dysarthria based on speech audio records. The most remarkable findings of the study include the determination of the glottal source amplitude distribution behaviour of normative and PD participants; the impact of age effects in phonation as a confounding factor in neuromotor disorder characterization; the importance of ensuring that the classification of speech dysarthria is based on principles that can be explained and interpreted; the need of taking into account the effects of medication when framing new classification experiments; the potential of using EEG-band decomposition to analyse vocal fold stiffness correlates, as well as the possibility of using these descriptions in longitudinal monitoring of treatment efficiency; the feasibility of establishing a relationship between acoustic and kinematic variables by projection model inversion; and the potential of these descriptions for estimating neuromotor activities in midbrain related to phonation and articulation activity. The most important outcome to be brought forth from the thesis is that the methodology used throughout the project uses a bottom-up approach based on speech model inversion at the acoustical, biomechanical, and neuromotor levels allowing to estimate glottal signals, biomechanical correlates, and neuromotor activity from speech alone, establishing a common neuromechanical characterisation framework on its own

Políticas de Copyright de Publicações Científicas em Repositórios Institucionais: O Caso do INESC TEC

A progressiva transformação das práticas científicas, impulsionada pelo desenvolvimento das novas Tecnologias de Informação e Comunicação (TIC), têm possibilitado aumentar o acesso à informação, caminhando gradualmente para uma abertura do ciclo de pesquisa. Isto permitirá resolver a longo prazo uma adversidade que se tem colocado aos investigadores, que passa pela existência de barreiras que limitam as condições de acesso, sejam estas geográficas ou financeiras. Apesar da produção científica ser dominada, maioritariamente, por grandes editoras comerciais, estando sujeita às regras por estas impostas, o Movimento do Acesso Aberto cuja primeira declaração pública, a Declaração de Budapeste (BOAI), é de 2002, vem propor alterações significativas que beneficiam os autores e os leitores. Este Movimento vem a ganhar importância em Portugal desde 2003, com a constituição do primeiro repositório institucional a nível nacional. Os repositórios institucionais surgiram como uma ferramenta de divulgação da produção científica de uma instituição, com o intuito de permitir abrir aos resultados da investigação, quer antes da publicação e do próprio processo de arbitragem (preprint), quer depois (postprint), e, consequentemente, aumentar a visibilidade do trabalho desenvolvido por um investigador e a respetiva instituição. O estudo apresentado, que passou por uma análise das políticas de copyright das publicações científicas mais relevantes do INESC TEC, permitiu não só perceber que as editoras adotam cada vez mais políticas que possibilitam o auto-arquivo das publicações em repositórios institucionais, como também que existe todo um trabalho de sensibilização a percorrer, não só para os investigadores, como para a instituição e toda a sociedade. A produção de um conjunto de recomendações, que passam pela implementação de uma política institucional que incentive o auto-arquivo das publicações desenvolvidas no âmbito institucional no repositório, serve como mote para uma maior valorização da produção científica do INESC TEC.The progressive transformation of scientific practices, driven by the development of new Information and Communication Technologies (ICT), which made it possible to increase access to information, gradually moving towards an opening of the research cycle. This opening makes it possible to resolve, in the long term, the adversity that has been placed on researchers, which involves the existence of barriers that limit access conditions, whether geographical or financial. Although large commercial publishers predominantly dominate scientific production and subject it to the rules imposed by them, the Open Access movement whose first public declaration, the Budapest Declaration (BOAI), was in 2002, proposes significant changes that benefit the authors and the readers. This Movement has gained importance in Portugal since 2003, with the constitution of the first institutional repository at the national level. Institutional repositories have emerged as a tool for disseminating the scientific production of an institution to open the results of the research, both before publication and the preprint process and postprint, increase the visibility of work done by an investigator and his or her institution. The present study, which underwent an analysis of the copyright policies of INESC TEC most relevant scientific publications, allowed not only to realize that publishers are increasingly adopting policies that make it possible to self-archive publications in institutional repositories, all the work of raising awareness, not only for researchers but also for the institution and the whole society. The production of a set of recommendations, which go through the implementation of an institutional policy that encourages the self-archiving of the publications developed in the institutional scope in the repository, serves as a motto for a greater appreciation of the scientific production of INESC TEC

Complexity and Entropy in Physiological Signals (CEPS): Resonance Breathing Rate Assessed Using Measures of Fractal Dimension, Heart Rate Asymmetry and Permutation Entropy

Background: As technology becomes more sophisticated, more accessible methods of interpretating Big Data become essential. We have continued to develop Complexity and Entropy in Physiological Signals (CEPS) as an open access MATLAB® GUI (graphical user interface) providing multiple methods for the modification and analysis of physiological data. Methods: To demonstrate the functionality of the software, data were collected from 44 healthy adults for a study investigating the effects on vagal tone of breathing paced at five different rates, as well as self-paced and un-paced. Five-minute 15-s recordings were used. Results were also compared with those from shorter segments of the data. Electrocardiogram (ECG), electrodermal activity (EDA) and Respiration (RSP) data were recorded. Particular attention was paid to COVID risk mitigation, and to parameter tuning for the CEPS measures. For comparison, data were processed using Kubios HRV, RR-APET and DynamicalSystems.jl software. We also compared findings for ECG RR interval (RRi) data resampled at 4 Hz (4R) or 10 Hz (10R), and non-resampled (noR). In total, we used around 190–220 measures from CEPS at various scales, depending on the analysis undertaken, with our investigation focused on three families of measures: 22 fractal dimension (FD) measures, 40 heart rate asymmetries or measures derived from Poincaré plots (HRA), and 8 measures based on permutation entropy (PE). Results: FDs for the RRi data differentiated strongly between breathing rates, whether data were resampled or not, increasing between 5 and 7 breaths per minute (BrPM). Largest effect sizes for RRi (4R and noR) differentiation between breathing rates were found for the PE-based measures. Measures that both differentiated well between breathing rates and were consistent across different RRi data lengths (1–5 min) included five PE-based (noR) and three FDs (4R). Of the top 12 measures with short-data values consistently within ± 5% of their values for the 5-min data, five were FDs, one was PE-based, and none were HRAs. Effect sizes were usually greater for CEPS measures than for those implemented in DynamicalSystems.jl. Conclusion: The updated CEPS software enables visualisation and analysis of multichannel physiological data using a variety of established and recently introduced complexity entropy measures. Although equal resampling is theoretically important for FD estimation, it appears that FD measures may also be usefully applied to non-resampled data