Fear Classification using Affective Computing with Physiological Information and Smart-Wearables

Mención Internacional en el título de doctorAmong the 17 Sustainable Development Goals proposed within the 2030 Agenda and adopted by all of the United Nations member states, the fifth SDG is a call for action to effectively turn gender equality into a fundamental human right and an essential foundation for a better world. It includes the eradication of all types of violence against women. Focusing on the technological perspective, the range of available solutions intended to prevent this social problem is very limited. Moreover, most of the solutions are based on a panic button approach, leaving aside the usage and integration of current state-of-the-art technologies, such as the Internet of Things (IoT), affective computing, cyber-physical systems, and smart-sensors. Thus, the main purpose of this research is to provide new insight into the design and development of tools to prevent and combat Gender-based Violence risky situations and, even, aggressions, from a technological perspective, but without leaving aside the different sociological considerations directly related to the problem. To achieve such an objective, we rely on the application of affective computing from a realist point of view, i.e. targeting the generation of systems and tools capable of being implemented and used nowadays or within an achievable time-frame. This pragmatic vision is channelled through: 1) an exhaustive study of the existing technological tools and mechanisms oriented to the fight Gender-based Violence, 2) the proposal of a new smart-wearable system intended to deal with some of the current technological encountered limitations, 3) a novel fear-related emotion classification approach to disentangle the relation between emotions and physiology, and 4) the definition and release of a new multi-modal dataset for emotion recognition in women. Firstly, different fear classification systems using a reduced set of physiological signals are explored and designed. This is done by employing open datasets together with the combination of time, frequency and non-linear domain techniques. This design process is encompassed by trade-offs between both physiological considerations and embedded capabilities. The latter is of paramount importance due to the edge-computing focus of this research. Two results are highlighted in this first task, the designed fear classification system that employed the DEAP dataset data and achieved an AUC of 81.60% and a Gmean of 81.55% on average for a subjectindependent approach, and only two physiological signals; and the designed fear classification system that employed the MAHNOB dataset data achieving an AUC of 86.00% and a Gmean of 73.78% on average for a subject-independent approach, only three physiological signals, and a Leave-One-Subject-Out configuration. A detailed comparison with other emotion recognition systems proposed in the literature is presented, which proves that the obtained metrics are in line with the state-ofthe- art. Secondly, Bindi is presented. This is an end-to-end autonomous multimodal system leveraging affective IoT throughout auditory and physiological commercial off-theshelf smart-sensors, hierarchical multisensorial fusion, and secured server architecture to combat Gender-based Violence by automatically detecting risky situations based on a multimodal intelligence engine and then triggering a protection protocol. Specifically, this research is focused onto the hardware and software design of one of the two edge-computing devices within Bindi. This is a bracelet integrating three physiological sensors, actuators, power monitoring integrated chips, and a System- On-Chip with wireless capabilities. Within this context, different embedded design space explorations are presented: embedded filtering evaluation, online physiological signal quality assessment, feature extraction, and power consumption analysis. The reported results in all these processes are successfully validated and, for some of them, even compared against physiological standard measurement equipment. Amongst the different obtained results regarding the embedded design and implementation within the bracelet of Bindi, it should be highlighted that its low power consumption provides a battery life to be approximately 40 hours when using a 500 mAh battery. Finally, the particularities of our use case and the scarcity of open multimodal datasets dealing with emotional immersive technology, labelling methodology considering the gender perspective, balanced stimuli distribution regarding the target emotions, and recovery processes based on the physiological signals of the volunteers to quantify and isolate the emotional activation between stimuli, led us to the definition and elaboration of Women and Emotion Multi-modal Affective Computing (WEMAC) dataset. This is a multimodal dataset in which 104 women who never experienced Gender-based Violence that performed different emotion-related stimuli visualisations in a laboratory environment. The previous fear binary classification systems were improved and applied to this novel multimodal dataset. For instance, the proposed multimodal fear recognition system using this dataset reports up to 60.20% and 67.59% for ACC and F1-score, respectively. These values represent a competitive result in comparison with the state-of-the-art that deal with similar multi-modal use cases. In general, this PhD thesis has opened a new research line within the research group under which it has been developed. Moreover, this work has established a solid base from which to expand knowledge and continue research targeting the generation of both mechanisms to help vulnerable groups and socially oriented technology.Programa de Doctorado en Ingeniería Eléctrica, Electrónica y Automática por la Universidad Carlos III de MadridPresidente: David Atienza Alonso.- Secretaria: Susana Patón Álvarez.- Vocal: Eduardo de la Torre Arnan

Embedded platform for ECG biometric recognition

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia de Electrónica e TelecomunicaçõesMuitas das tarefas diárias do ser humano requerem processos que validem a identidade do utilizador. Cartões de identificação, chapas de identificação militar, senhas e códigos são as estratégias mais usuais no campo da validação e identificação de utilizador. Apesar do amplo uso de tais mecanismos, os meios de autenticação baseados na entidade ou no conhecimento do sujeito levantam graves problemas de segurança no que diz respeito ao risco de fraude e roubo de identidade. O uso de características físicas ou comportamentais dos seres vivos como forma de identificálos unicamente, é o tópico da Biometria [bio (vida) + metria (medida)]. A principal vantagem destes sistemas é a dependência completa no individuo, não existindo nenhuma sujeição a objetos ou à memorização de códigos, como ocorre nas estratégias tradicionais. Isto leva a uma maior utilização de sistemas biométricos a fim de aumentar a dificuldade de falsificação de credencias, visto este ser a principal falha dos sistemas de reconhecimento pessoal tradicionais. Por exemplo, uma foto pode fingir um rosto, a íris do olho pode ser falsificada por lentes de contacto e até mesmo a impressão digital pode ser trocada por um dedo de gel. Este trabalho propõe um sistema de reconhecimento biométrico baseado em sinais electrocardiográficos (ECG). As diferenças de potencial podem ser obtidas pela colocação de elétrodos sobre a superfície do corpo e medindo a tensão entre eles. O sistema de aquisição dos sinais ECG deste trabalho é constituído por dois elétrodos colocados um em cada membro superior do utilizador, preferencialmente nas mãos, para um aquisição mais cómoda. A fim de tornar esta solução móvel e facilmente transportável para qualquer local, é proposto um sistema embebido autónomo para autenticação humana baseada no ECG de cada indivíduo. Normalmente, os sistemas baseados em ECG usam hardware capaz de adquirir o sinal e um micro-controlador para enviar os sinais para um computador onde é realizado o tratamento dos dados. Em alternativa, o objetivo deste trabalho é conceber de um prototipo capaz de adquirir e processar o sinal ECG de diferentes indivíduos e, através de um algoritmo de extração e classificação de padrões, autenticar ou identificar as pessoas em questão. Este prototipo será baseado num sistema embebidos capaz de autenticar ou identificar indivíduos em tempo real sem recurso a um computador pessoal ou a qualquer plataforma de processamento externa. O problema do reconhecimento humano com base em biometria, é tipicamente dividido em várias fases (independentemente do tipo de biométrica) típicas de um sistema de reconhecimento de padrões: • Aquisição do sinal biométrico por sensores; • Pré-processamento do sinal de acordo com o sistema adotado; • Extração das características necessárias ao reconhecimento; • Seleção das características mais discriminativas do sujeito; • Classificação das características escolhidas e decisão de qual a correspondência da mesma na base de dados. Nesta abordagem o sinal ECG obtido é segmentado por batimento cardíaco, usando o pico R (complexo R, incluído no complexo QRS, nomes dados às ondas características constituintes do ECG) como ponto de pesquisa para a segmentação. Dois algoritmos de segmentação são estudados, Hamilton e Slope Sum Function (SSF), sendo o SSF a solução implementada no sistema embebido pelas suas propriedades de execução em tempo real. Outra particularidade da implementação do algoritmo SSF é que este foi desenvolvido para detetar pulsos de pressão arterial e é aqui adaptado para a deteção do complexo R, constituinte do sinal ECG. A extração de características do sinal ECG segmentado é baseada na análise do sinal no domínio da frequência e segue um algoritmo proposto por Odinaka. Cada batimento cardíaco é analisado por troços obtidos pela segmentação do sinal em várias janelas sobrepostas. É calculada a transformada de Fourier de cada janela segmentada (em que normalmente é usada uma janela de Hamming para melhor estimativa) e estimada uma distribuição de gaussiana (caracterizada por uma média e desvio padrão) para cada intervalo de frequência que caracteriza o batimento cardíaco típico em analise. Posteriormente, para treino, é estimada a distribuição gaussiana para as janelas extraídas de cada segmento e calculada a sua média entre todos os segmentos. São estas as características guardadas na base de dados para futura comparação com novas aquisições para se efetuar o reconhecimento dos batimentos. A cada nova aquisição, é confirmada a veracidade do utilizador, e é feita uma atualização dos valores da base de dados com os novos adquiridos, através de uma média ponderada. Com este método é possível contornar o efeito temporal nos sinais ECG. É de conhecimento comum que com a evolução da idade do individuo, os seus sinais fisiológicos sofrem pequenas alterações e o ECG não é exceção. Para a transformada de Fourier foi aumentado a dimensão do array para aumentar a definição nas baixas frequências, onde a informação requerida para o reconhecimento pessoal é preponderante. Neste trabalho, o sistema foi implementado para garantir uma execução em tempo real. As amostras do sinal ECG não podem ser perdidas e o processo de autenticação tem que ser realizado de forma muito eficiente de modo a permitir o funcionamento em tempo real. Para isto é necessária a escolha de hardware capaz de concretizar este objetivo. A possibilidade do uso de um microprocessador foi descartada pela sua baixa versatilidade e alto custo de desenvolvimento. Os sistema ASIC e FPGA, também foram descartados pelos elevados custos de desenvolvimento e aquisição. Foi escolhido então, um sistema de desenvolvimento baseado num micro-controlador (MCU) com arquitetura ARM Cortex 4. O MCU escolhido, STM32F4-Discovery, conta com uma grande versatilidade, baixo consumo de energia (100mA), grande velocidade de processamento (168MHz), integração de DSP e unidade de virgula flutuante. Memoria interna não volátil também é necessária, a fim de conservar as características de treino de cada individuo. O sistema é projetado para ser autónomo, não-intrusivo e fácil de usar em diferentes cenários. Isto é conseguido combinando a facil utilização de apenas dois elétrodos, um em cada membro superior, com um sistema embebido alimentado por bateria com processamento em tempo real e capacidade de visualização de resultados. O sistema foi validado em duas fases. Em primeiro lugar os algoritmos foram validados usando uma base de dados já testada em estudos anteriores, e foi comprovado que o sistema tem uma percentagem de identificação de 89% e 10% de taxa de erros em autenticação. Finalmente foram realizadas novas aquisições que comprovaram a eficiência do sistema. Com 11 sujeitos na base de dados o sistema conta com uma taxa de identificação de 100% e um taxa de erros de autenticação de 9.3%. Utilizando as propriedades dos sinais ECG, este sistema torna-se um plataforma fiável, eficaz e eficiente. Problemas cardíacos humanos, como arritmias são um problema que fazem descer o rendimento do sistema. O sistema realizado é uma prova de conceito que ilustra como os sistemas embebidos podem mudar o mundo dos sistemas de autenticação pois proporcionam segurança e uma utilização muito fácil para toda a população.Abstract: Traditional strategies for authentication are either entity-based or knowledge-based, like PIN numbers, passwords and ID cards. This raises serious security problems, concerning the risk of identity theft as these mechanisms are widely spread. They are a part of many daily tasks and they are dependent on objects or memories. This work prompts to change these mechanisms for a secure and ubiquitous biometric reckoning system based on the electrocardiographic (ECG) signal. It includes the study of all the steps required for the development of a biometric system, namely: acquisition, processing and classification. In the acquisition, the ECG signal is obtained from two electrodes placed at each limb to a electronic device that filters and amplifies the raw signal to be able to be converted to digital in the microcontroller, using the internal Analogue-to-Digital Converter; In the processing phase, the signal is digitally filtered and segmented in heartbeats. Features are selected and extracted using one algorithm created by Odinaka and herein modified to increase performance in low bandwidth ECG signals; In classification, extracted features are compared, using nearest neighbour algorithm, with data stored in the database in order to classify each heartbeat. The work develop and implement a working prototype based on an embedded system (ARMBased Cortex4 32 bit RISC STM32F407VGT6). Acquisition modules, processing units and algorithms are studied and developed on a prototype for identification and authentication mobile system based on the ECG. The lack of mobile real-time reckoning systems makes this thesis a challenging and self-motivated work. Unique, continuous acquisition and non-intrusive are the main characteristics of the ECG signals. These properties make ECG based reckoning system a reliable and effective platform. Preliminary evaluation showed a 100% identification rate and a 9.3% equal error rate at the authentication procedure. These results came form an acquired database of 11 subjects, with test and train sequences acquired in different procedures. Human heart problems, like arrhythmias are a challenging problem that drop the reckoning performance of the system. This kind of embedded solutions can change the world of authentication systems in order to provide security and be easy-to-use for the general population

Intelligent Subgrouping of Multitrack Audio

Subgrouping facilitates the simultaneous manipulation of a number of audio tracks and is a central aspect of mix engineering. However, the decision process of subgrouping is a poorly documented technique. This research sheds light on this ubiquitous but poorly de ned mix practice, provides rules and constraints on how it should be approached as well as demonstrates its bene t to an automatic mixing system. I rst explored the relationship that subgrouping has with perceived mix quality by examining a number of mix projects. This was in order to decipher the actual process of creating subgroups and to see if any of the decisions made were intrinsically linked to mix quality. I found mix quality to be related to the number of subgroups and type of subgroup processing used. This subsequently led me to interviewing distinguished professionals in the audio engineering eld, with the intention of gaining a deeper understanding of the process. The outcome of these interviews and the previous analyses of mix projects allowed me to propose rules that could be used for real life mixing and automatic mixing. Some of the rules I established were used to research and develop a method for the automatic creation of subgroups using machine learning techniques. I also investigated the relationship between music production quality and human emotion. This was to see if music production quality had an emotional e ect on a particular type of listener. The results showed that the emotional impact of mixing only really mattered to those with critical listening skills. This result is important for automatic mixing systems in general, as it would imply that quality only really matters to a minority of people. I concluded my research on subgrouping by conducting an experiment to see if subgrouping would bene t the perceived clarity and quality of a mix. The results of a subjective listening test showed this to be true

Aspects of room acoustics, vision and motion in the human auditory perception of space

The human sense of hearing contributes to the awareness of where sound-generating objects are located in space and of the environment in which the hearing individual is located. This auditory perception of space interacts in complex ways with our other senses, can be both disrupted and enhanced by sound reflections, and includes safety mechanisms which have evolved to protect our lives, but can also mislead us. This dissertation explores some selected topics from this wide subject area, mostly by testing the abilities and subjective judgments of human listeners in virtual environments. Reverberation is the gradually decaying persistence of sounds in an enclosed space which results from repeated sound reflections at surfaces. The first experiment (Chapter 2) compared how strongly people perceived reverberation in different visual situations: when they could see the room and the source which generated the sound; when they could see some room and some sound source, but the image did not match what they heard; and when they could not see anything at all. There were no indications that the visual image had any influence on this aspect of room-acoustical perception. The potential benefits of motion for judging the distance of sound sources were the focus of the second study (Chapter 3), which consists of two parts. In the first part, loudspeakers were placed at different depths in front of sitting listeners who, on command, had to either remain still or move their upper bodies sideways. This experiment demonstrated that humans can exploit motion parallax (the effect that closer objects appear faster to a moving observer than farther objects) with their ears and not just with their eyes. The second part combined a virtualisation of such sound sources with a motion platform to show that the listeners’ interpretation of this auditory motion parallax was better when they performed this lateral movement by themselves, rather than when they were moved by the apparatus or were not actually in motion at all. Two more experiments were concerned with the perception of sounds which are perceived as becoming louder over time. These have been called “looming”, as the source of such a sound might be on a collision course. One of the studies (Chapter 4) showed that western diamondback rattlesnakes (Crotalus atrox) increase the vibration speed of their rattle in response to the approach of a threatening object. It also demonstrated that human listeners perceive (virtual) snakes which engage in this behaviour as especially close, causing them to keep a greater margin of safety than they would otherwise. The other study (section 5.6) was concerned with the well-known looming bias of the sound localisation system, a phenomenon which leads to a sometimes exaggerated, sometimes more accurate perception of approaching compared to receding sounds. It attempted to find out whether this bias is affected by whether listeners hear such sounds in a virtual enclosed space or in an environment with no sound reflections. While the results were inconclusive, this experiment is noteworthy as a proof of concept: It was the first study to make use of a new real-time room-acoustical simulation system, liveRAZR, which was developed as part of this dissertation (Chapter 5). Finally, while humans have been more often studied for their unique abilities to communicate with each other and bats for their extraordinary capacity to locate objects by sound, this dissertation turns this setting of priorities on its head with the last paper (Chapter 6): Based on recordings of six pale spear-nosed bats (Phyllostomus discolor), it is a survey of the identifiably distinct vocalisations observed in their social interactions, along with a description of the different situations in which they typically occur.Das menschliche Gehör trägt zum Bewusstsein dafür bei, wo sich schallerzeugende Objekte im Raum befinden und wie die Umgebung beschaffen ist, in der sich eine Person aufhält. Diese auditorische Raumwahrnehmung interagiert auf komplexe Art und Weise mit unseren anderen Sinnen, kann von Schallreflektionen sowohl profitieren als auch durch sie behindert werden, und besitzt Mechanismen welche evolutionär entstanden sind, um unser Leben zu schützen, uns aber auch irreführen können. Diese Dissertation befasst sich mit einigen ausgewählten Themen aus diesem weiten Feld und stützt sich dabei meist auf die Testung von Wahrnehmungsfähigkeiten und subjektiver Einschätzungen menschlicher Hörer/-innen in virtueller Realität. Beim ersten Experiment (Kapitel 2) handelte es sich um einen Vergleich zwischen der Wahrnehmung von Nachhall, dem durch wiederholte Reflexionen an Oberflächen hervorgerufenen, sukzessiv abschwellenden Verbleib von Schall in einem umschlossenen Raum, unter verschiedenen visuellen Umständen: wenn die Versuchsperson den Raum und die Schallquelle sehen konnte; wenn sie irgendeinen Raum und irgendeine Schallquelle sehen konnte, dieses Bild aber vom Schalleindruck abwich; und wenn sie gar kein Bild sehen konnte. Dieser Versuch konnte keinen Einfluss eines Seheindrucks auf diesen Aspekt der raumakustischen Wahrnehmung zu Tage fördern. Mögliche Vorteile von Bewegung für die Einschätzung der Entfernung von Schallquellen waren der Schwerpunkt der zweiten Studie (Kapitel 3). Diese bestand aus zwei Teilen, wovon der erste zeigte, dass Hörer/-innen, die ihren Oberkörper relativ zu zwei in unterschiedlichen Abständen vor ihnen aufgestellten Lautsprechern auf Kommando entweder stillhalten oder seitlich bewegen mussten, im letzteren Falle von der Bewegungsparallaxe (dem Effekt, dass sich der nähere Lautsprecher relativ zum sich bewegenden Körper schneller bewegte als der weiter entfernte) profitieren konnten. Der zweite Teil kombinierte eine Simulation solcher Schallquellen mit einer Bewegungsplattform, wodurch gezeigt werden konnte, dass die bewusste Eigenbewegung für die Versuchspersonen hilfreicher war, als durch die Plattform bewegt zu werden oder gar nicht wirklich in Bewegung zu sein. Zwei weitere Versuche gingen auf die Wahrnehmung von Schallen ein, deren Ursprungsort sich nach und nach näher an den/die Hörer/-in heranbewegte. Derartige Schalle werden auch als „looming“ („anbahnend“) bezeichnet, da eine solche Annäherung bei bedrohlichen Signalen nichts Gutes ahnen lässt. Einer dieser Versuche (Kapitel 4) zeigte zunächst, dass Texas-Klapperschlangen (Crotalus atrox) die Vibrationsgeschwindigkeit der Schwanzrassel steigern, wenn sich ein bedrohliches Objekt ihnen nähert. Menschliche Hörer/-innen nahmen (virtuelle) Schlangen, die dieses Verhalten aufweisen, als besonders nahe wahr und hielten einen größeren Sicherheitsabstand ein, als sie es sonst tun würden. Der andere Versuch (Abschnitt 5.6) versuchte festzustellen, ob die wohlbekannte Neigung unserer Schallwahrnehmung, näherkommende Schalle manchmal übertrieben und manchmal genauer einzuschätzen als sich entfernende, durch Schallreflektionen beeinflusst werden kann. Diese Ergebnisse waren unschlüssig, jedoch bestand die Besonderheit dieses Versuchs darin, dass er erstmals ein neues Echtzeitsystem zur Raumakustiksimulation (liveRAZR) nutzte, welches als Teil dieser Dissertation entwickelt wurde (Kapitel 5). Abschließend (Kapitel 6) wird die Schwerpunktsetzung auf den Kopf gestellt, nach der Menschen öfter auf ihre einmaligen Fähigkeiten zur Kommunikation miteinander untersucht werden und Fledermäuse öfter auf ihre außergewöhnliches Geschick, Objekte durch Schall zu orten: Anhand von Aufnahmen von sechs Kleinen Lanzennasen (Phyllostomus discolor) fasst das Kapitel die klar voneinander unterscheidbaren Laute zusammen, die diese Tiere im sozialen Umgang miteinander produzieren, und beschreibt, in welchen Situationen diese Lauttypen typischerweise auftreten

Determining Additional Modulus of Subgarde Reaction Based on Tolerable Settlement for the Nailed-slab System Resting on Soft Clay.

Abstract—Nailed-slab System is a proposed alternative solution for rigid pavement problem on soft soils. Equivalent modulus of subgrade reaction (k’) can be used in designing of nailed-slab system. This modular is the cumulative of modulus of subgrade reaction from plate load test (k) and additional modulus of subgrade reaction due to pile installing (∆∆∆∆k). A recent method has used reduction of pile resistance approach in determining ∆∆∆∆k. The relative displacement between pile and soils, and reduction of pile resistance has been identified. In fact, determining of reduction of pile resistance is difficult. This paper proposes an approach by considering tolerable settlement of rigid pavement. Validation is carried out with respect to a loading test of nailed-slab models. The models are presented as strip section of rigid pavement. The theory of beams on elastic foundation is used to calculate the slab deflection by using k’. Proposed approach can results in deflection prediction close to observed one. In practice, the Nailed-slab System would be constructed by multiple-row piles. Designing this system based on one-pile row analysis will give more safety design and will consume less time