Techniques of EMG signal analysis: detection, processing, classification and applications

Electromyography (EMG) signals can be used for clinical/biomedical applications, Evolvable Hardware Chip (EHW) development, and modern human computer interaction. EMG signals acquired from muscles require advanced methods for detection, decomposition, processing, and classification. The purpose of this paper is to illustrate the various methodologies and algorithms for EMG signal analysis to provide efficient and effective ways of understanding the signal and its nature. We further point up some of the hardware implementations using EMG focusing on applications related to prosthetic hand control, grasp recognition, and human computer interaction. A comparison study is also given to show performance of various EMG signal analysis methods. This paper provides researchers a good understanding of EMG signal and its analysis procedures. This knowledge will help them develop more powerful, flexible, and efficient applications

An autopoietic approach to the development of speech recognition (pendekatan autopoietic dalam pembangunan pengecaman suara)

The focus of research here is on the implementation of speech recognition through an autopoietic approach. The work done here has culminated in the introduction of a neural network architecture named Homunculus Network. This network was used in the development of a speech recognition system for Bahasa Melayu. The speech recognition system is an isolated-word, phoneme-level speech recognizer that is speaker independent and has a vocabulary of 15 words. The research done has identified some issues worth further work later. These issues are also the basis for the design and the development of the new autopoietic speech recognition system

A Review of Deep Learning Techniques for Speech Processing

The field of speech processing has undergone a transformative shift with the advent of deep learning. The use of multiple processing layers has enabled the creation of models capable of extracting intricate features from speech data. This development has paved the way for unparalleled advancements in speech recognition, text-to-speech synthesis, automatic speech recognition, and emotion recognition, propelling the performance of these tasks to unprecedented heights. The power of deep learning techniques has opened up new avenues for research and innovation in the field of speech processing, with far-reaching implications for a range of industries and applications. This review paper provides a comprehensive overview of the key deep learning models and their applications in speech-processing tasks. We begin by tracing the evolution of speech processing research, from early approaches, such as MFCC and HMM, to more recent advances in deep learning architectures, such as CNNs, RNNs, transformers, conformers, and diffusion models. We categorize the approaches and compare their strengths and weaknesses for solving speech-processing tasks. Furthermore, we extensively cover various speech-processing tasks, datasets, and benchmarks used in the literature and describe how different deep-learning networks have been utilized to tackle these tasks. Additionally, we discuss the challenges and future directions of deep learning in speech processing, including the need for more parameter-efficient, interpretable models and the potential of deep learning for multimodal speech processing. By examining the field's evolution, comparing and contrasting different approaches, and highlighting future directions and challenges, we hope to inspire further research in this exciting and rapidly advancing field

Brains in interaction

Wanneer twee mensen met elkaar communiceren, dan ontstaat er een soort tijdelijke verbinding tussen hen. Deze verbinding bestaat uit een keten van gebeurtenissen en begint bijvoorbeeld bij de hersenactiviteit in de motorische cortex van de ene persoon. Deze activiteit leidt tot gedrag, bijvoorbeeld het maken van een gebaar, dat wordt gezien door de andere persoon. Deze bekijkt en interpreteert dit gebaar wat leidt tot activiteit in de visuele, sensorische en associatieve cortices. Dit kan dan weer leiden tot hersenactiviteit in de motorische cortex, tot een ander gebaar en zo verder. Dit proefschrift beschrijft een hersenonderzoek naar zo’n dergelijke indirecte verbinding tussen twee mensen. Voordat ik in ga op het onderzoek wat we hebben uitgevoerd, is het belangrijk om iets te weten over de achtergrond en inspiratie waarop dit onderzoek is gebaseerd. De afgelopen jaren zijn er twee belangrijke ideeën ontwikkelt over hoe mensen elkaar begrijpen en met elkaar kunnen communiceren: het idee van een spiegelsysteem en een ‘redeneersysteem’. Deze twee ideeën vormen de basis van dit onderzoek en worden beschreven in de volgende paragraven. Verder heeft de ontwikkeling van ‘Granger causaliteit’, een analysemethode om verbindingen tussen hersengebieden vast te stellen ook een belangrijke rol gespeeld, deze wordt hierna beschreven. HE T S P I EGE L S Y S T E EM Het idee van spiegelen is dat ons brein de handelingen van andere mensen ‘nabootst’. Aan de basis van dit idee staat de ontdekking van spiegelneuronen (‘mirror neurons’) in de jaren negentig (Gallese et al., 1996; Pellegrino et al., 1992). Deze spiegelneuronen zijn min of meer per toeval ontdekt in een lab in Parma tijdens het meten van neuronen in het gebied F5 (ventrale premotorische cortex) van deMakaak aap. Een onderzoeker merkte op dat deze neuronen niet alleen reageerden op het moment dat de aap zelf een pinda oppakte, maar ook op het moment dat de aap naar de onderzoeker keek terwijl deze een pinda oppakte. Het was bekend dat deze neuronen betrokken zijn bij het uitvoeren van doelgerichte handelingen met de handen en met de mond. Maar nu werd opeens duidelijk dat deze gebieden ook sensorische eigenschappen bezitten (Kurata and Tanji, 1986; Rizzolatti et al., 1988). Deze neuronen representeren hiermee zowel het uitvoeren van een handeling als de waarneming van die handeling. De ontdekking van spiegelneuronen had een grote impact, omdat hiermee het vermoeden werd bevestigd dat waarnemen en handelen sterk aan elkaar gekoppeld zijn. Dit idee speelde al langer een rol in psychologische theorieën. James Gibson beweerde bijvoorbeeld dat perceptie bestaat uit het direct waarnemen van handelingsmogelijkheden (Gibson, 1986). Kort na de eerste ontdekking van spiegelneuronen wilde men weten of de menselijke hersenen ook zo’n dergelijk mechanisme bezitten. Omdat het meten van een individuele neuron vrijwel niet mogelijk is zonder een brein te beschadigen, richtten onderzoeken zich op de vraag of er wellicht hersengebieden bestaan die activiteit laten zien tijdens zowel het uitvoeren als het waarnemen van een handeling (Buccino et al., 2001; Grafton et al., 1996; Grèzes et al., 1998; Grèzes and Decety, 2001; Grèzes et al., 2003; Nishitani and Hari, 2000, 2002; Perani et al., 2001; Gazzola et al., 2007b,a; Gazzola and Keysers, 2008). Dat blijkt inderdaad zo te zijn en de gebieden met deze eigenschap vormen samen het menselijke spiegelsysteem (Keysers and Gazzola, 2009). Deze gebieden zijn de ventrale en dorsale premotorische cortex, de inferieure parietale cortex en de middelste superieure temporele gyrus (zie Figuur 3). Er bestaan overigens niet alleen spiegelgebieden die een overlap in activiteit laten zien voor het uitvoeren en waarnemen van handelingen, maar ook voor het ervaren en het waarnemen van emoties en sensaties, zoals walging, aanraking en pijn (Wicker et al., 2003; Keysers and Perrett, 2004; Singer et al., 2004; Bastiaansen et al., 2009). Innovatieve experimenten, die bijvoorbeeld gebruikmaken van ‘cross-modal repetition suppression’, hebben inmiddels wetenschappelijk bewijs geleverd voor het bestaan van individuele spiegelneuronen in de menselijke hersenen (Kilner et al., 2009; Lingnau et al., 2009; Chong et al., 2008;Mukamel et al., 2010). Doordat spiegelneuronen een directe link leggen tussen de handelingen die we zelf uitvoeren en de handelingen die we anderen zien doen, wordt aangenomen dat spiegelneuronen een functie hebben in het begrijpen van wat de ander aan het doen is (zie Rizzolatti and Sinigaglia, 2010, voor een recent overzicht van de literatuur). Bij het zien van een handeling van iemand anders wordt de motorische representatie van deze handeling in de eigen hersenen actief, alsof deze handeling zelf wordt uitgevoerd. Dit idee vormt de kern van de simulatietheorie: we begrijpen wat een ander doet doordat we deze handeling als het ware simuleren in onze eigen hersenen (Goldman, 1992; Gibson, 1986; Gallese, 2003). Belangrijk voor het onderzoek in dit proefschrift is dat de simulatietheorie een voorspellingmaakt over spiegelneuronen. Deze theorie beweert namelijk dat spiegelneuronen in het brein van degene die een handeling waarneemt resoneren met de spiegelneuronen van degene die de handeling uitvoert. De term‘resonantie’ wordt hier losjes gebruikt en er wordt mee bedoeld dat de pieken en dalen in de hersenactiviteit van het motorsysteem van de ene persoon overeenkomstige pieken en dalen veroorzaakt in de hersenactiviteit van het motorsysteem in de andere persoon (Gallese and Goldman, 1998; Gallese et al., 2004; Rizzolatti et al., 2001). In Hoofdstuk 4 van dit proefschrift wordt deze bewering over resonantie onderzocht. HE T REDENE ERS Y S T E EM Naast dit spiegelmechanisme waarmee we anderen begrijpen, bezitten we ook een meer reflectief vermogen om na te denken over wat er in anderen omgaat. Denk bijvoorbeeld aan een typische scene uit een soap, zoals The Bold and the Beautiful: Taylor and Ridge staan op het punt om met elkaar in het huwelijk te treden. Zonder dat Taylor dit weet, staat Brooke op het punt om te vertellen dat ze zwanger is van Ridge, hopende dat ze hiermee de bruiloft kan verhinderen. Om zo’n situatie te kunnen begrijpen en te kunnen waarderen, moeten we in staat zijn om bij te houden wat de verschillende personen wel en niet weten en wat ze zullen denken op het moment dat ze het te horen zullen krijgen. Dit soort bewuste denkprocessen wordt in de literatuur wel ‘Theory of Mind’ (ToM) genoemd (Premack andWoodruff, 1978; Wimmer and Perner, 1983) en vindt plaats in andere gebieden dan de spiegelgebieden (Frith and Frith, 1999, 2006). Het zijn de ‘redeneergebieden’ (zie Figuur 3), die actief zijn tijdens bijvoorbeeld het interpreteren van (strip)verhalen en het nadenken over jezelf en anderen (Amodio and Frith, 2006; Fletcher et al., 1995; Frith and Frith, 2006, 2003; Gallagher et al., 2000; Gusnard et al., 2001). De belangrijkste twee gebieden van dit redeneersysteem zijn de ventrale mediale prefrontale cortex en de temporeelparietale junctie. Over the decades, two important networks in the brain have been identified about how people interact: the mirror system and the mentalizing network. This thesis investigates how these networks work together during social interaction. We performed an experiment in which brain activity of two persons was measured while they engaged in a social communication game (Charades). Results showed that the mirror system is highly involved during the game, while the main mentalizing area does not show any involvement. We then extended a connectivity analysis, Granger causality, which is usually applied within one brain, to a between-brain analysis. With this method, we used brain activity of the gesturer to map regions in the brain of the guesser, whose brain activity has a Granger-causal relation to that of the gesturer. The mirror system of the gesturer shows a Granger-causal relation to the mirror system of the guesser, but also to the main mentalizing area of the guesser. This means that, even while this mentalizing area does not show involvement when analyzed using a classic method, it does show a temporal relationship with the brain activity of the gesturer. We furthermore performed simulations to investigate a possible confound of Granger causality: inter- and intrasubject variability in hemodynamic responses. Results show high sensitivity and accuracy for Granger causality between-brains, while sensitivity of within-brain Granger causality remains low. However, if a Grangercausality is found, this indicates the correct underlying direction in 80% of the cases. Finally, we used within-brain Granger causality to investigate how areas in the mirror system influence each other during gesturing and guessing.

음성언어 이해에서의 중의성 해소

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2022. 8. 김남수.언어의 중의성은 필연적이다. 그것은 언어가 의사 소통의 수단이지만, 모든 사람이 생각하는 어떤 개념이 완벽히 동일하게 전달될 수 없는 것에 기인한다. 이는 필연적인 요소이기도 하지만, 언어 이해에서 중의성은 종종 의사 소통의 단절이나 실패를 가져오기도 한다. 언어의 중의성에는 다양한 층위가 존재한다. 하지만, 모든 상황에서 중의성이 해소될 필요는 없다. 태스크마다, 도메인마다 다른 양상의 중의성이 존재하며, 이를 잘 정의하고 해소될 수 있는 중의성임을 파악한 후 중의적인 부분 간의 경계를 잘 정하는 것이 중요하다. 본고에서는 음성 언어 처리, 특히 의도 이해에 있어 어떤 양상의 중의성이 발생할 수 있는지 알아보고, 이를 해소하기 위한 연구를 진행한다. 이러한 현상은 다양한 언어에서 발생하지만, 그 정도 및 양상은 언어에 따라서 다르게 나타나는 경우가 많다. 우리의 연구에서 주목하는 부분은, 음성 언어에 담긴 정보량과 문자 언어의 정보량 차이로 인해 중의성이 발생하는 경우들이다. 본 연구는 운율(prosody)에 따라 문장 형식 및 의도가 다르게 표현되는 경우가 많은 한국어를 대상으로 진행된다. 한국어에서는 다양한 기능이 있는(multi-functional한) 종결어미(sentence ender), 빈번한 탈락 현상(pro-drop), 의문사 간섭(wh-intervention) 등으로 인해, 같은 텍스트가 여러 의도로 읽히는 현상이 발생하곤 한다. 이것이 의도 이해에 혼선을 가져올 수 있다는 데에 착안하여, 본 연구에서는 이러한 중의성을 먼저 정의하고, 중의적인 문장들을 감지할 수 있도록 말뭉치를 구축한다. 의도 이해를 위한 말뭉치를 구축하는 과정에서 문장의 지향성(directivity)과 수사성(rhetoricalness)이 고려된다. 이것은 음성 언어의 의도를 서술, 질문, 명령, 수사의문문, 그리고 수사명령문으로 구분하게 하는 기준이 된다. 본 연구에서는 기록된 음성 언어(spoken language)를 충분히 높은 일치도(kappa = 0.85)로 주석한 말뭉치를 이용해, 음성이 주어지지 않은 상황에서 중의적인 텍스트를 감지하는 데에 어떤 전략 혹은 언어 모델이 효과적인가를 보이고, 해당 태스크의 특징을 정성적으로 분석한다. 또한, 우리는 텍스트 층위에서만 중의성에 접근하지 않고, 실제로 음성이 주어진 상황에서 중의성 해소(disambiguation)가 가능한지를 알아보기 위해, 텍스트가 중의적인 발화들만으로 구성된 인공적인 음성 말뭉치를 설계하고 다양한 집중(attention) 기반 신경망(neural network) 모델들을 이용해 중의성을 해소한다. 이 과정에서 모델 기반 통사적/의미적 중의성 해소가 어떠한 경우에 가장 효과적인지 관찰하고, 인간의 언어 처리와 어떤 연관이 있는지에 대한 관점을 제시한다. 본 연구에서는 마지막으로, 위와 같은 절차로 의도 이해 과정에서의 중의성이 해소되었을 경우, 이를 어떻게 산업계 혹은 연구 단에서 활용할 수 있는가에 대한 간략한 로드맵을 제시한다. 텍스트에 기반한 중의성 파악과 음성 기반의 의도 이해 모듈을 통합한다면, 오류의 전파를 줄이면서도 효율적으로 중의성을 다룰 수 있는 시스템을 만들 수 있을 것이다. 이러한 시스템은 대화 매니저(dialogue manager)와 통합되어 간단한 대화(chit-chat)가 가능한 목적 지향 대화 시스템(task-oriented dialogue system)을 구축할 수도 있고, 단일 언어 조건(monolingual condition)을 넘어 음성 번역에서의 에러를 줄이는 데에 활용될 수도 있다. 우리는 본고를 통해, 운율에 민감한(prosody-sensitive) 언어에서 의도 이해를 위한 중의성 해소가 가능하며, 이를 산업 및 연구 단에서 활용할 수 있음을 보이고자 한다. 본 연구가 다른 언어 및 도메인에서도 고질적인 중의성 문제를 해소하는 데에 도움이 되길 바라며, 이를 위해 연구를 진행하는 데에 활용된 리소스, 결과물 및 코드들을 공유함으로써 학계의 발전에 이바지하고자 한다.Ambiguity in the language is inevitable. It is because, albeit language is a means of communication, a particular concept that everyone thinks of cannot be conveyed in a perfectly identical manner. As this is an inevitable factor, ambiguity in language understanding often leads to breakdown or failure of communication. There are various hierarchies of language ambiguity. However, not all ambiguity needs to be resolved. Different aspects of ambiguity exist for each domain and task, and it is crucial to define the boundary after recognizing the ambiguity that can be well-defined and resolved. In this dissertation, we investigate the types of ambiguity that appear in spoken language processing, especially in intention understanding, and conduct research to define and resolve it. Although this phenomenon occurs in various languages, its degree and aspect depend on the language investigated. The factor we focus on is cases where the ambiguity comes from the gap between the amount of information in the spoken language and the text. Here, we study the Korean language, which often shows different sentence structures and intentions depending on the prosody. In the Korean language, a text is often read with multiple intentions due to multi-functional sentence enders, frequent pro-drop, wh-intervention, etc. We first define this type of ambiguity and construct a corpus that helps detect ambiguous sentences, given that such utterances can be problematic for intention understanding. In constructing a corpus for intention understanding, we consider the directivity and rhetoricalness of a sentence. They make up a criterion for classifying the intention of spoken language into a statement, question, command, rhetorical question, and rhetorical command. Using the corpus annotated with sufficiently high agreement on a spoken language corpus, we show that colloquial corpus-based language models are effective in classifying ambiguous text given only textual data, and qualitatively analyze the characteristics of the task. We do not handle ambiguity only at the text level. To find out whether actual disambiguation is possible given a speech input, we design an artificial spoken language corpus composed only of ambiguous sentences, and resolve ambiguity with various attention-based neural network architectures. In this process, we observe that the ambiguity resolution is most effective when both textual and acoustic input co-attends each feature, especially when the audio processing module conveys attention information to the text module in a multi-hop manner. Finally, assuming the case that the ambiguity of intention understanding is resolved by proposed strategies, we present a brief roadmap of how the results can be utilized at the industry or research level. By integrating text-based ambiguity detection and speech-based intention understanding module, we can build a system that handles ambiguity efficiently while reducing error propagation. Such a system can be integrated with dialogue managers to make up a task-oriented dialogue system capable of chit-chat, or it can be used for error reduction in multilingual circumstances such as speech translation, beyond merely monolingual conditions. Throughout the dissertation, we want to show that ambiguity resolution for intention understanding in prosody-sensitive language can be achieved and can be utilized at the industry or research level. We hope that this study helps tackle chronic ambiguity issues in other languages ​​or other domains, linking linguistic science and engineering approaches.1 Introduction 1 1.1 Motivation 2 1.2 Research Goal 4 1.3 Outline of the Dissertation 5 2 Related Work 6 2.1 Spoken Language Understanding 6 2.2 Speech Act and Intention 8 2.2.1 Performatives and statements 8 2.2.2 Illocutionary act and speech act 9 2.2.3 Formal semantic approaches 11 2.3 Ambiguity of Intention Understanding in Korean 14 2.3.1 Ambiguities in language 14 2.3.2 Speech act and intention understanding in Korean 16 3 Ambiguity in Intention Understanding of Spoken Language 20 3.1 Intention Understanding and Ambiguity 20 3.2 Annotation Protocol 23 3.2.1 Fragments 24 3.2.2 Clear-cut cases 26 3.2.3 Intonation-dependent utterances 28 3.3 Data Construction . 32 3.3.1 Source scripts 32 3.3.2 Agreement 32 3.3.3 Augmentation 33 3.3.4 Train split 33 3.4 Experiments and Results 34 3.4.1 Models 34 3.4.2 Implementation 36 3.4.3 Results 37 3.5 Findings and Summary 44 3.5.1 Findings 44 3.5.2 Summary 45 4 Disambiguation of Speech Intention 47 4.1 Ambiguity Resolution 47 4.1.1 Prosody and syntax 48 4.1.2 Disambiguation with prosody 50 4.1.3 Approaches in SLU 50 4.2 Dataset Construction 51 4.2.1 Script generation 52 4.2.2 Label tagging 54 4.2.3 Recording 56 4.3 Experiments and Results 57 4.3.1 Models 57 4.3.2 Results 60 4.4 Summary 63 5 System Integration and Application 65 5.1 System Integration for Intention Identification 65 5.1.1 Proof of concept 65 5.1.2 Preliminary study 69 5.2 Application to Spoken Dialogue System 75 5.2.1 What is 'Free-running' 76 5.2.2 Omakase chatbot 76 5.3 Beyond Monolingual Approaches 84 5.3.1 Spoken language translation 85 5.3.2 Dataset 87 5.3.3 Analysis 94 5.3.4 Discussion 95 5.4 Summary 100 6 Conclusion and Future Work 103 Bibliography 105 Abstract (In Korean) 124 Acknowledgment 126박