A silent speech system based on permanent magnet articulography and direct synthesis

In this paper we present a silent speech interface (SSI) system aimed at restoring speech communication for individuals who have lost their voice due to laryngectomy or diseases affecting the vocal folds. In the proposed system, articulatory data captured from the lips and tongue using permanent magnet articulography (PMA) are converted into audible speech using a speaker-dependent transformation learned from simultaneous recordings of PMA and audio signals acquired before laryngectomy. The transformation is represented using a mixture of factor analysers, which is a generative model that allows us to efficiently model non-linear behaviour and perform dimensionality reduction at the same time. The learned transformation is then deployed during normal usage of the SSI to restore the acoustic speech signal associated with the captured PMA data. The proposed system is evaluated using objective quality measures and listening tests on two databases containing PMA and audio recordings for normal speakers. Results show that it is possible to reconstruct speech from articulator movements captured by an unobtrusive technique without an intermediate recognition step. The SSI is capable of producing speech of sufficient intelligibility and naturalness that the speaker is clearly identifiable, but problems remain in scaling up the process to function consistently for phonetically rich vocabularies

Direct Speech Reconstruction From Articulatory Sensor Data by Machine Learning

This paper describes a technique that generates speech acoustics from articulator movements. Our motivation is to help people who can no longer speak following laryngectomy, a procedure that is carried out tens of thousands of times per year in the Western world. Our method for sensing articulator movement, permanent magnetic articulography, relies on small, unobtrusive magnets attached to the lips and tongue. Changes in magnetic field caused by magnet movements are sensed and form the input to a process that is trained to estimate speech acoustics. In the experiments reported here this “Direct Synthesis” technique is developed for normal speakers, with glued-on magnets, allowing us to train with parallel sensor and acoustic data. We describe three machine learning techniques for this task, based on Gaussian mixture models, deep neural networks, and recurrent neural networks (RNNs). We evaluate our techniques with objective acoustic distortion measures and subjective listening tests over spoken sentences read from novels (the CMU Arctic corpus). Our results show that the best performing technique is a bidirectional RNN (BiRNN), which employs both past and future contexts to predict the acoustics from the sensor data. BiRNNs are not suitable for synthesis in real time but fixed-lag RNNs give similar results and, because they only look a little way into the future, overcome this problem. Listening tests show that the speech produced by this method has a natural quality that preserves the identity of the speaker. Furthermore, we obtain up to 92% intelligibility on the challenging CMU Arctic material. To our knowledge, these are the best results obtained for a silent-speech system without a restricted vocabulary and with an unobtrusive device that delivers audio in close to real time. This work promises to lead to a technology that truly will give people whose larynx has been removed their voices back

Silent Speech Interfaces for Speech Restoration: A Review

This work was supported in part by the Agencia Estatal de Investigacion (AEI) under Grant PID2019-108040RB-C22/AEI/10.13039/501100011033. The work of Jose A. Gonzalez-Lopez was supported in part by the Spanish Ministry of Science, Innovation and Universities under Juan de la Cierva-Incorporation Fellowship (IJCI-2017-32926).This review summarises the status of silent speech interface (SSI) research. SSIs rely on non-acoustic biosignals generated by the human body during speech production to enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the first case and present latest SSI research aimed at providing new alternative and augmentative communication methods for persons with severe speech disorders. SSIs can employ a variety of biosignals to enable silent communication, such as electrophysiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking of articulator movements using imaging techniques. Depending on the disorder, some sensing techniques may be better suited than others to capture speech-related information. For instance, EMG and imaging techniques are well suited for laryngectomised patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for severely paralysed individuals. From the biosignals, SSIs decode the intended message, using automatic speech recognition or speech synthesis algorithms. Despite considerable advances in recent years, most present-day SSIs have only been validated in laboratory settings for healthy users. Thus, as discussed in this paper, a number of challenges remain to be addressed in future research before SSIs can be promoted to real-world applications. If these issues can be addressed successfully, future SSIs will improve the lives of persons with severe speech impairments by restoring their communication capabilities.Agencia Estatal de Investigacion (AEI) PID2019-108040RB-C22/AEI/10.13039/501100011033Spanish Ministry of Science, Innovation and Universities under Juan de la Cierva-Incorporation Fellowship IJCI-2017-3292

Speech reconstruction from articulatory movement for laryngectomees

Laryngectomees are individuals who have their larynx surgically removed due to the treatment of laryngeal cancer. They lost their ability to vocalize speech but can still articulate after the surgery. As a result, they rely on alternative methods for communication (e.g., alaryngeal speech). However, alaryngeal speech generates unnatural-sounding voice, which discourages their willingness to speak and causes social isolation and even depression. Silent speech interfaces (SSIs) convert non-audio human bio-signals (e.g., tongue and lip movement) to speech, which have the potential to reconstruct speech with natural-sounding voice and even speaker identity. Although the concept of SSI and its feasibility has been demonstrated in the field, SSI development still faces a few major challenges, including small data size, lack of algorithms for laryngectomees, and lack of wearable devices for daily use. A series of studies were conducted to address these challenges. This dissertation contributed to the field from different aspects, including novel algorithms and approaches for articulation-to-speech mapping, new knowledge to improve the design of SSI, and evaluation of newly developed wearable devices.Electrical and Computer Engineerin

A silent speech system based on permanent magnet articulography and direct synthesis

In this paper we present a silent speech interface (SSI) system aimed at restoring speech communication for individuals who have lost their voice due to laryngectomy or diseases affecting the vocal folds. In the proposed system, articulatory data captured from the lips and tongue using permanent magnet articulography (PMA) are converted into audible speech using a speaker-dependent transformation learned from simultaneous recordings of PMA and audio signals acquired before laryngectomy. The transformation is represented using a mixture of factor analysers, which is a generative model that allows us to efficiently model non-linear behaviour and perform dimensionality reduction at the same time. The learned transformation is then deployed during normal usage of the SSI to restore the acoustic speech signal associated with the captured PMA data. The proposed system is evaluated using objective quality measures and listening tests on two databases containing PMA and audio recordings for normal speakers. Results show that it is possible to reconstruct speech from articulator movements captured by an unobtrusive technique without an intermediate recognition step. The SSI is capable of producing speech of sufficient intelligibility and naturalness that the speaker is clearly identifiable, but problems remain in scaling up the process to function consistently for phonetically rich vocabularies

Voice restoration after laryngectomy based on magnetic sensing of articulator movement and statistical articulation-to-speech conversion

© Springer International Publishing AG 2017.In this work, we present a silent speech system that is able to generate audible speech from captured movement of speech articulators. Our goal is to help laryngectomy patients, i.e. patients who have lost the ability to speak following surgical removal of the larynx most frequently due to cancer, to recover their voice. In our system, we use a magnetic sensing technique known as Permanent Magnet Articulography (PMA) to capture the movement of the lips and tongue by attaching small magnets to the articulators and monitoring the magnetic field changes with sensors close to the mouth. The captured sensor data is then transformed into a sequence of speech parameter vectors from which a time-domain speech signal is finally synthesised. The key component of our system is a parametric transformation which represents the PMA-tospeech mapping. Here, this transformation takes the form of a statistical model (a mixture of factor analysers, more specifically) whose parameters are learned from simultaneous recordings of PMA and speech signals acquired before laryngectomy. To evaluate the performance of our system on voice reconstruction, we recorded two PMA-and-speech databases with different phonetic complexity for several non-impaired subjects. Results show that our system is able to synthesise speech that sounds as the original voice of the subject and also is intelligible. However, more work still need to be done to achieve a consistent synthesis for phonetically-rich vocabularies

EMG-to-Speech: Direct Generation of Speech from Facial Electromyographic Signals

The general objective of this work is the design, implementation, improvement and evaluation of a system that uses surface electromyographic (EMG) signals and directly synthesizes an audible speech output: EMG-to-speech

Reconstruction of intelligible audio speech from visual speech information

The aim of the work conducted in this thesis is to reconstruct audio speech signals using information which can be extracted solely from a visual stream of a speaker's face, with application for surveillance scenarios and silent speech interfaces. Visual speech is limited to that which can be seen of the mouth, lips, teeth, and tongue, where the visual articulators convey considerably less information than in the audio domain, leading to the task being difficult. Accordingly, the emphasis is on the reconstruction of intelligible speech, with less regard given to quality. A speech production model is used to reconstruct audio speech, where methods are presented in this work for generating or estimating the necessary parameters for the model. Three approaches are explored for producing spectral-envelope estimates from visual features as this parameter provides the greatest contribution to speech intelligibility. The first approach uses regression to perform the visual-to-audio mapping, and then two further approaches are explored using vector quantisation techniques and classification models, with long-range temporal information incorporated at the feature and model-level. Excitation information, namely fundamental frequency and aperiodicity, is generated using artificial methods and joint-feature clustering approaches. Evaluations are first performed using mean squared error analyses and objective measures of speech intelligibility to refine the various system configurations, and then subjective listening tests are conducted to determine word-level accuracy, giving real intelligibility scores, of reconstructed speech. The best performing visual-to-audio domain mapping approach, using a clustering-and-classification framework with feature-level temporal encoding, is able to achieve audio-only intelligibility scores of 77 %, and audiovisual intelligibility scores of 84 %, on the GRID dataset. Furthermore, the methods are applied to a larger and more continuous dataset, with less favourable results, but with the belief that extensions to the work presented will yield a further increase in intelligibility

Técnicas para la mejora de la inteligibilidad en voces patológicas

229 p.Los laringectomizados son personas cuya laringe ha sido extirpada quirúrgicamente, normalmente comoconsecuencia de un tumor. Al tratarse éste de un órgano fundamental para la producción de la voz,pierden la capacidad de hablar. Sin embargo, muchas de ellas consiguen re-aprender a hablar de unamanera distinta. Este tipo de habla se conoce como voz esofágica y es bastante distinta de la voz sana. Sunaturalidad e inteligibilidad es menor hasta el punto de que ciertos oyentes tienen que hacer un esfuerzopara comprender lo que se les está diciendo.Esto supone un perjuicio en la calidad de vida de los laringectomizados ya que sus capacidadescomunicativas se ven afectadas, no solo en las interacciones entre personas sino también en las interfaceshombre-máquina controladas por la voz. En esta tesis se abordan diferentes métodos para la mejora de lainteligibilidad de las voces alaríngeas de manera que palíen estos problemas.Un aspecto importante ha sido analizar las características propias de la voz esofágica. No es fácilencontrar el material necesario para hacer este análisis y los recursos disponibles son escasos. Esta tesisha querido llenar este vacío mediante la grabación de una base de datos paralela de locutores esofágicos.Esta base de datos ha sido caracterizada acústicamente. Con este objetivo se ha comprobado los efectosque tiene el método de extracción de la frecuencia fundamental a la hora de analizar las características delas señales esofágicas. Se ha propuesto utilizar el análisis del residuo glotal ya que capta mejor laspeculiaridades de este tipo de voces.Es necesario también disponer de algún método para evaluar de manera objetiva el impacto que tienen losmétodos propuestos para mejorar la inteligibilidad. Con este propósito se ha implementado unreconocedor cuyas características y particularidades se recogen en este documento. Este ASR se validóparticipando en una evaluación de detección de términos hablados organizada por la Red Temática enTecnologías del Habla.Para la mejora de la inteligibilidad de las voces esofágicas primero se han analizado diferentes algoritmosbasados en las técnicas de conversión de voz existentes aplicadas a voces sanas. Se ha evaluado tanto elcomportamiento de técnicas clásicas basadas en mezclas de Gaussianas como el de técnicas deconversión basadas en aprendizaje profundo.Por último, se han adaptado con éxito estas técnicas de conversión a las voces esofágicas. Estasconversiones se han evaluado de manera objetiva mediante el ASR construido, y subjetivamentemediante tests de preferencia. Aunque los resultados de las pruebas subjetivas exponen que para losoyentes no hay diferencias significativas entre las voces convertidas y las esofágicas originales, losresultados del reconocimiento automático muestran que las técnicas de conversión aplicadas a este tipode voces consiguen disminuir la tasa de error obtenida