141 research outputs found
On the Utility of Representation Learning Algorithms for Myoelectric Interfacing
Electrical activity produced by muscles during voluntary movement is a reflection of the firing patterns of relevant motor neurons and, by extension, the latent motor intent driving the movement. Once transduced via electromyography (EMG) and converted into digital form, this activity can be processed to provide an estimate of the original motor intent and is as such a feasible basis for non-invasive efferent neural interfacing. EMG-based motor intent decoding has so far received the most attention in the field of upper-limb prosthetics, where alternative means of interfacing are scarce and the utility of better control apparent. Whereas myoelectric prostheses have been available since the 1960s, available EMG control interfaces still lag behind the mechanical capabilities of the artificial limbs they are intended to steer—a gap at least partially due to limitations in current methods for translating EMG into appropriate motion commands. As the relationship between EMG signals and concurrent effector kinematics is highly non-linear and apparently stochastic, finding ways to accurately extract and combine relevant information from across electrode sites is still an active area of inquiry.This dissertation comprises an introduction and eight papers that explore issues afflicting the status quo of myoelectric decoding and possible solutions, all related through their use of learning algorithms and deep Artificial Neural Network (ANN) models. Paper I presents a Convolutional Neural Network (CNN) for multi-label movement decoding of high-density surface EMG (HD-sEMG) signals. Inspired by the successful use of CNNs in Paper I and the work of others, Paper II presents a method for automatic design of CNN architectures for use in myocontrol. Paper III introduces an ANN architecture with an appertaining training framework from which simultaneous and proportional control emerges. Paper Iv introduce a dataset of HD-sEMG signals for use with learning algorithms. Paper v applies a Recurrent Neural Network (RNN) model to decode finger forces from intramuscular EMG. Paper vI introduces a Transformer model for myoelectric interfacing that do not need additional training data to function with previously unseen users. Paper vII compares the performance of a Long Short-Term Memory (LSTM) network to that of classical pattern recognition algorithms. Lastly, paper vIII describes a framework for synthesizing EMG from multi-articulate gestures intended to reduce training burden
Multikernel convolutional neural network for sEMG based hand gesture classification
openIl riconoscimento dei gesti della mano è un argomento ampiamente discusso in letteratura, dove vengono analizzate diverse tecniche sia in termini di tipi di segnale in ingresso che di algoritmi. Tra i più utilizzati ci sono i segnali elettromiografici (sEMG), già ampiamente sfruttati nelle applicazioni di interazione uomo-macchina (HMI). Determinare come decodificare le informazioni contenute nei segnali EMG in modo robusto e accurato è un problema chiave per il quale è urgente trovare una soluzione.
Recentemente, molti incarichi di riconoscimento dei pattern EMG sono stati affrontati utilizzando metodi di deep learning. Nonostante le elevate prestazioni di questi ultimi, le loro capacitĂ di generalizzazione sono spesso limitate dall'elevata eterogeneitĂ tra i soggetti, l'impedenza cutanea, il posizionamento dei sensori, ecc.
Inoltre, poiché questo progetto è focalizzato sull'applicazione in tempo reale di protesi, ci sono maggiori vincoli sui tempi di risposta del sistema che riducono la complessità dei modelli. In questa tesi è stata testata una rete neurale convoluzionale multi-kernel su diversi dataset pubblici per verificare la sua generalizzabilità . Inoltre, è stata analizzata la capacità del modello di superare i limiti inter-soggetto e inter-sessione in giorni diversi, preservando i vincoli legati a un sistema embedded. I risultati confermano le difficoltà incontrate nell'estrazione di informazioni dai segnali emg; tuttavia, dimostrano la possibilità di ottenere buone prestazioni per un uso robusto di mani prostetiche. Inoltre, è possibile ottenere prestazioni migliori personalizzando il modello con tecniche di transfer learning e di adattamento al dominio.Hand gesture recognition is a widely discussed topic in the literature, where different techniques are analyzed in terms of both input signal types and algorithms. Among the most widely used are electromyographic signals (sEMG), which are already widely exploited in human-computer interaction (HMI) applications. Determining how to decode the information contained in EMG signals robustly and accurately is a key problem for which a solution is urgently needed.
Recently, many EMG pattern recognition tasks have been addressed using deep learning methods. Despite their high performance, their generalization capabilities are often limited by high heterogeneity among subjects, skin impedance, sensor placement, etc.
In addition, because this project is focused on the real-time application of prostheses, there are greater constraints on the system response times that reduce the complexity of the models. In this thesis, a multi-kernel convolutional neural network was tested on several public datasets to verify its generalizability. In addition, the model's ability to overcome inter-subject and inter-session constraints on different days while preserving the constraints associated with an embedded system was analyzed. The results confirm the difficulties encountered in extracting information from emg signals; however, they demonstrate the possibility of achieving good performance for robust use of prosthetic hands. In addition, better performance can be achieved by customizing the model with transfer learning and domain-adaptationtechniques
Changes in surface electromyography characteristics and foot-tapping rate of force development as measures of spasticity in patients with multiple sclerosis
Spasticity is a common symptom experienced by individuals with upper motor neuron lesions such as those with stroke, spinal cord injury, traumatic brain injury, cerebral palsy, amyotrophic lateral sclerosis, and multiple sclerosis. Although the etiology and progression of spasticity differs between these clinical populations, it shares many of the same consequences: muscle pain, weakness, fatigue, increased disability, depression, medication side effects, and a reduced quality of life. For this reason, there has been increased interest in the measurement and treatment of spasticity symptoms.
Subjective measures of spasticity like the Modified Ashworth Scale (MAS) and Tardieu Scale have shown questionable validity/reliability and poorly correlate to functional outcome measures but continue to be used in clinical and research settings. Objective measures like myotonometry, electrogoniometry, and inertial sensors on the other hand provide much more reliable measures but at the expense of increased costs, time, and equipment. Therefore, to properly assess and treat spasticity symptoms, a timelier and cost-effective objective measure of spasticity is needed. PURPOSE: To reexamine a previously collected dataset from a sample of patients with multiple sclerosis before and after dry-needling and functional electrically stimulated walking spasticity treatments. Specifically, we wished to know whether there were: 1.) Acute (within visit) and chronic (between visit) changes in sEMG and Foot-tapping rate of force development measures after treatment, 2.) Between leg differences before and after treatments, 3.) significant correlations between EMG, foot-tapping, and functional outcome measures. METHODS: 16 MS patients (10 relapsing-remitting and 6 progressive MS) participated in the original study. The study consisted of 14 visits: 2 pre/post visits, 4 visits of dry-needling + functional electrically stimulated walking (FESW), and 8 visits with FESW only. The more spastic leg (involved leg) was given the treatment, making the other the control. Dry-needling was performed on the involved leg’s gastrocnemius medial and lateral heads by inserting monofilament needles and electrically stimming the muscles until visible twitches occurred. Dry-needling was done 30 seconds on and 30 seconds off for a total of 90 seconds of treatment. FESW was performed on the involved leg by attaching electrodes to the tibialis anterior and gastrocnemius muscles. Patients walked 20-minutes at a self-selected pace while the involved leg was stimmed. sEMG was collected before and after each treatment by having the patient perform a single maximal heel raise. Foot-tapping ability was assessed using the 10-second foot-tapping test (FTT) and a small force plate. Functional measures also included the 25-foot walk test (25FWT) 6-minute walk test (6MWT), modified fatigue impact score (MFIS), and number of heel raises performed. RESULTS: No significant between leg differences were noted for any of the sEMG measures (p>0.05). No significant chronic changes occurred in any of the sEMG measures. For the Dry-needling + FESW visits, sEMG sample entropy was significantly increased in the involved leg at post-needling (p = 0.035) and post-FESW (p = 0.027). The non-involved leg’s sample entropy was significantly higher at post-FESW only (p = 0.017). The non-involved leg’s, mean frequency was significantly higher at post-FESW compared pre-needling (p = 0.033) and post-needling (p = 0.032). For the FESW only visits, there were no significant changes in the involved leg. The Non-involved leg’s mean frequency was significantly higher at Post-FESW (p = 0.006). Median frequency was significantly higher at Post-FESW (p = 0.009). The number of foot-taps was significantly increased from Pre to Post-intervention in both the Involved (p = 0.006) and Non-involved legs (p 0.002). There was a significantly higher number of foot-taps in the Non-involved leg compared to the Involved leg at both Pre (p =0.008) and Post (p = 0.015) timepoints. AUC was significantly higher in the Involved leg at Post-treatment (p = 0.030). Time to peak was found to be higher in the Involved leg compared to the Non-involved leg at both Pre (p = 0.037) and Post-intervention (p = 0.019). Time to base was higher in the Involved leg compared to the Non-involved leg at both Pre (p = 0.031) and Post-intervention (p = 0.004). Total tap time was higher in the Involved leg at both Pre (p = 0.010) and Post-intervention (p = 0.007). Percent time to peak was significantly lower in the involved limb at Pre-intervention (p = 0.026) and Post intervention (p = 0.037). Percent time to base was significantly higher in the Involved leg at Pre-intervention (p = 0.026) and Post intervention (p = 0.037). The sEMG measures tended to poorly or non-significantly correlate with the functional outcome measures. The foot-tapping measures, especially the involved leg, tended to exhibit stronger correlations with the functional outcome measures. CONCLUSION: sEMG Sample entropy and foot-tapping ability are significantly improved by dry-needling treatments and walking. sEMG measures did not tend to correlate well with functional outcome measures but the foot-tapping measures did. This suggests that foot-tapping rate and related measures may be a useful measure of spasticity and treatment effects
Robust and reliable hand gesture recognition for myoelectric control
Surface Electromyography (sEMG) is a physiological signal to record the electrical activity of muscles by electrodes applied to the skin. In the context of Muscle Computer Interaction (MCI), systems are controlled by transforming myoelectric signals into interaction commands that convey the intent of user movement, mostly for rehabilitation purposes. Taking the myoeletric hand prosthetic control as an example, using sEMG recorded from the remaining muscles of the stump can be considered as the most natural way for amputees who lose their limbs to perform activities of daily living with the aid of prostheses. Although the earliest myoelectric control research can date back to the 1950s, there still exist considerable challenges to address the significant gap between academic research and industrial applications. Most recently, pattern recognition-based control is being developed rapidly to improve the dexterity of myoelectric prosthetic devices due to the recent development of machine learning and deep learning techniques. It is clear that the performance of Hand Gesture Recognition (HGR) plays an essential role in pattern recognition-based control systems. However, in reality, the tremendous success in achieving very high sEMG-based HGR accuracy (≥ 90%) reported in scientific articles produced only limited clinical or commercial impact. As many have reported, its real-time performance tends to degrade significantly as a result of many confounding factors, such as electrode shift, sweating, fatigue, and day-to-day variation. The main interest of the present thesis is, therefore, to improve the robustness of sEMG-based HGR by taking advantage of the most recent advanced deep learning techniques to address several practical concerns. Furthermore, the challenge of this research problem has been reinforced by only considering using raw sparse multichannel sEMG signals as input. Firstly, a framework for designing an uncertainty-aware sEMG-based hand gesture classifier is proposed. Applying it allows us to quickly build a model with the ability to make its inference along with explainable quantified multidimensional uncertainties. This addresses the black-box concern of the HGR process directly. Secondly, to fill the gap of lacking consensus on the definition of model reliability in this field, a proper definition of model reliability is proposed. Based on it, reliability analysis can be performed as a new dimension of evaluation to help select the best model without relying only on classification accuracy. Our extensive experimental results have shown the efficiency of the proposed reliability analysis, which encourages researchers to use it as a supplementary tool for model evaluation. Next, an uncertainty-aware model is designed based on the proposed framework to address the low robustness of hand grasp recognition. This offers an opportunity to investigate whether reliable models can achieve robust performance. The results show that the proposed model can improve the long-term robustness of hand grasp recognition by rejecting highly uncertain predictions. Finally, a simple but effective normalisation approach is proposed to improve the robustness of inter-subject HGR, thus addressing the clinical challenge of having only a limited amount of data from any individual. The comparison results show that better performance can be obtained by it compared to a state-of-the-art (SoA) transfer learning method when only one training cycle is available. In summary, this study presents promising methods to pursue an accurate, robust, and reliable classifier, which is the overarching goal for sEMG-based HGR. The direction for future work would be the inclusion of these in real-time myoelectric control applications
Application of wearable sensors in actuation and control of powered ankle exoskeletons: a Comprehensive Review
Powered ankle exoskeletons (PAEs) are robotic devices developed for gait assistance, rehabilitation, and augmentation. To fulfil their purposes, PAEs vastly rely heavily on their sensor systems. Human–machine interface sensors collect the biomechanical signals from the human user to inform the higher level of the control hierarchy about the user’s locomotion intention and requirement, whereas machine–machine interface sensors monitor the output of the actuation unit to ensure precise tracking of the high-level control commands via the low-level control scheme. The current article aims to provide a comprehensive review of how wearable sensor technology has contributed to the actuation and control of the PAEs developed over the past two decades. The control schemes and actuation principles employed in the reviewed PAEs, as well as their interaction with the integrated sensor systems, are investigated in this review. Further, the role of wearable sensors in overcoming the main challenges in developing fully autonomous portable PAEs is discussed. Finally, a brief discussion on how the recent technology advancements in wearable sensors, including environment—machine interface sensors, could promote the future generation of fully autonomous portable PAEs is provided
A Silent-Speech Interface using Electro-Optical Stomatography
Sprachtechnologie ist eine große und wachsende Industrie, die das Leben von technologieinteressierten Nutzern auf zahlreichen Wegen bereichert. Viele potenzielle Nutzer werden jedoch ausgeschlossen: Nämlich alle Sprecher, die nur schwer oder sogar gar nicht Sprache produzieren können.
Silent-Speech Interfaces bieten einen Weg, mit Maschinen durch ein bequemes sprachgesteuertes Interface zu kommunizieren ohne dafür akustische Sprache zu benötigen. Sie können außerdem prinzipiell eine Ersatzstimme stellen, indem sie die intendierten Äußerungen, die der Nutzer nur still artikuliert, künstlich synthetisieren. Diese Dissertation stellt ein neues Silent-Speech Interface vor, das auf einem neu entwickelten Messsystem namens Elektro-Optischer Stomatografie und einem neuartigen parametrischen Vokaltraktmodell basiert, das die Echtzeitsynthese von Sprache basierend auf den gemessenen Daten ermöglicht. Mit der Hardware wurden Studien zur Einzelworterkennung durchgeführt, die den Stand der Technik in der intra- und inter-individuellen Genauigkeit erreichten und übertrafen. Darüber hinaus wurde eine Studie abgeschlossen, in der die Hardware zur Steuerung des Vokaltraktmodells in einer direkten Artikulation-zu-Sprache-Synthese verwendet wurde. Während die Verständlichkeit der Synthese von Vokalen sehr hoch eingeschätzt wurde, ist die Verständlichkeit von Konsonanten und kontinuierlicher Sprache sehr schlecht. Vielversprechende Möglichkeiten zur Verbesserung des Systems werden im Ausblick diskutiert.:Statement of authorship iii
Abstract v
List of Figures vii
List of Tables xi
Acronyms xiii
1. Introduction 1
1.1. The concept of a Silent-Speech Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Structure of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Fundamentals of phonetics 7
2.1. Components of the human speech production system . . . . . . . . . . . . . . . . . . . 7
2.2. Vowel sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3. Consonantal sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4. Acoustic properties of speech sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5. Coarticulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6. Phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7. Summary and implications for the design of a Silent-Speech Interface (SSI) . . . . . . . 21
3. Articulatory data acquisition techniques in Silent-Speech Interfaces 25
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2. Scope of the literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3. Video Recordings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.4. Ultrasonography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.5. Electromyography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.6. Permanent-Magnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.7. Electromagnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.8. Radio waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.9. Palatography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.10.Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4. Electro-Optical Stomatography 55
4.1. Contact sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2. Optical distance sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3. Lip sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.4. Sensor Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.5. Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.6. Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5. Articulation-to-Text 99
5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.2. Command word recognition pilot study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.3. Command word recognition small-scale study . . . . . . . . . . . . . . . . . . . . . . . . 102
6. Articulation-to-Speech 109
6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.2. Articulatory synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.3. The six point vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4. Objective evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 116
6.5. Perceptual evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 120
6.6. Direct synthesis using EOS to control the vocal tract model . . . . . . . . . . . . . . . . 125
6.7. Pitch and voicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7. Summary and outlook 145
7.1. Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
A. Overview of the International Phonetic Alphabet 151
B. Mathematical proofs and derivations 153
B.1. Combinatoric calculations illustrating the reduction of possible syllables using phonotactics
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
B.2. Signal Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
B.3. Effect of the contact sensor area on the conductance . . . . . . . . . . . . . . . . . . . . 155
B.4. Calculation of the forward current for the OP280V diode . . . . . . . . . . . . . . . . . . 155
C. Schematics and layouts 157
C.1. Schematics of the control unit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
C.2. Layout of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
C.3. Bill of materials of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
C.4. Schematics of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
C.5. Layout of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
C.6. Bill of materials of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
D. Sensor unit assembly 169
E. Firmware flow and data protocol 177
F. Palate file format 181
G. Supplemental material regarding the vocal tract model 183
H. Articulation-to-Speech: Optimal hyperparameters 189
Bibliography 191Speech technology is a major and growing industry that enriches the lives of technologically-minded people in a number of ways. Many potential users are, however, excluded: Namely, all speakers who cannot easily or even at all produce speech. Silent-Speech Interfaces offer a way to communicate with a machine by a convenient speech recognition interface without the need for acoustic speech. They also can potentially provide a full replacement voice by synthesizing the intended utterances that are only silently articulated by the user. To that end, the speech movements need to be captured and mapped to either text or acoustic speech. This dissertation proposes a new Silent-Speech Interface based on a newly developed measurement technology called Electro-Optical Stomatography and a novel parametric vocal tract model to facilitate real-time speech synthesis based on the measured data. The hardware was used to conduct command word recognition studies reaching state-of-the-art intra- and inter-individual performance. Furthermore, a study on using the hardware to control the vocal tract model in a direct articulation-to-speech synthesis loop was also completed. While the intelligibility of synthesized vowels was high, the intelligibility of consonants and connected speech was quite poor. Promising ways to improve the system are discussed in the outlook.:Statement of authorship iii
Abstract v
List of Figures vii
List of Tables xi
Acronyms xiii
1. Introduction 1
1.1. The concept of a Silent-Speech Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Structure of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Fundamentals of phonetics 7
2.1. Components of the human speech production system . . . . . . . . . . . . . . . . . . . 7
2.2. Vowel sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3. Consonantal sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4. Acoustic properties of speech sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5. Coarticulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6. Phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7. Summary and implications for the design of a Silent-Speech Interface (SSI) . . . . . . . 21
3. Articulatory data acquisition techniques in Silent-Speech Interfaces 25
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2. Scope of the literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3. Video Recordings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.4. Ultrasonography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.5. Electromyography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.6. Permanent-Magnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.7. Electromagnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.8. Radio waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.9. Palatography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.10.Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4. Electro-Optical Stomatography 55
4.1. Contact sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2. Optical distance sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3. Lip sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.4. Sensor Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.5. Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.6. Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5. Articulation-to-Text 99
5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.2. Command word recognition pilot study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.3. Command word recognition small-scale study . . . . . . . . . . . . . . . . . . . . . . . . 102
6. Articulation-to-Speech 109
6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.2. Articulatory synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.3. The six point vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4. Objective evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 116
6.5. Perceptual evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 120
6.6. Direct synthesis using EOS to control the vocal tract model . . . . . . . . . . . . . . . . 125
6.7. Pitch and voicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7. Summary and outlook 145
7.1. Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
A. Overview of the International Phonetic Alphabet 151
B. Mathematical proofs and derivations 153
B.1. Combinatoric calculations illustrating the reduction of possible syllables using phonotactics
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
B.2. Signal Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
B.3. Effect of the contact sensor area on the conductance . . . . . . . . . . . . . . . . . . . . 155
B.4. Calculation of the forward current for the OP280V diode . . . . . . . . . . . . . . . . . . 155
C. Schematics and layouts 157
C.1. Schematics of the control unit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
C.2. Layout of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
C.3. Bill of materials of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
C.4. Schematics of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
C.5. Layout of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
C.6. Bill of materials of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
D. Sensor unit assembly 169
E. Firmware flow and data protocol 177
F. Palate file format 181
G. Supplemental material regarding the vocal tract model 183
H. Articulation-to-Speech: Optimal hyperparameters 189
Bibliography 19
Intelligent Sensors for Human Motion Analysis
The book, "Intelligent Sensors for Human Motion Analysis," contains 17 articles published in the Special Issue of the Sensors journal. These articles deal with many aspects related to the analysis of human movement. New techniques and methods for pose estimation, gait recognition, and fall detection have been proposed and verified. Some of them will trigger further research, and some may become the backbone of commercial systems
Intelligent Biosignal Processing in Wearable and Implantable Sensors
This reprint provides a collection of papers illustrating the state-of-the-art of smart processing of data coming from wearable, implantable or portable sensors. Each paper presents the design, databases used, methodological background, obtained results, and their interpretation for biomedical applications. Revealing examples are brain–machine interfaces for medical rehabilitation, the evaluation of sympathetic nerve activity, a novel automated diagnostic tool based on ECG data to diagnose COVID-19, machine learning-based hypertension risk assessment by means of photoplethysmography and electrocardiography signals, Parkinsonian gait assessment using machine learning tools, thorough analysis of compressive sensing of ECG signals, development of a nanotechnology application for decoding vagus-nerve activity, detection of liver dysfunction using a wearable electronic nose system, prosthetic hand control using surface electromyography, epileptic seizure detection using a CNN, and premature ventricular contraction detection using deep metric learning. Thus, this reprint presents significant clinical applications as well as valuable new research issues, providing current illustrations of this new field of research by addressing the promises, challenges, and hurdles associated with the synergy of biosignal processing and AI through 16 different pertinent studies. Covering a wide range of research and application areas, this book is an excellent resource for researchers, physicians, academics, and PhD or master students working on (bio)signal and image processing, AI, biomaterials, biomechanics, and biotechnology with applications in medicine
- …