Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception

Hearing-impaired people often struggle to follow the speech stream of an individual talker in noisy environments. Recent studies show that the brain tracks attended speech and that the attended talker can be decoded from neural data on a single-trial level. This raises the possibility of “neuro-steered” hearing devices in which the brain-decoded intention of a hearing-impaired listener is used to enhance the voice of the attended speaker from a speech separation front-end. So far, methods that use this paradigm have focused on optimizing the brain decoding and the acoustic speech separation independently. In this work, we propose a novel framework called brain-informed speech separation (BISS)1 in which the information about the attended speech, as decoded from the subject’s brain, is directly used to perform speech separation in the front-end. We present a deep learning model that uses neural data to extract the clean audio signal that a listener is attending to from a multi-talker speech mixture. We show that the framework can be applied successfully to the decoded output from either invasive intracranial electroencephalography (iEEG) or non-invasive electroencephalography (EEG) recordings from hearing-impaired subjects. It also results in improved speech separation, even in scenes with background noise. The generalization capability of the system renders it a perfect candidate for neuro-steered hearing-assistive devices

Sintesi vocale attraverso speech BCI invasive: nuove prospettive verso un parlato intelligibile

Ogni anno milioni di persone risultano affette da numerose patologie neurodegenerative o traumatiche che comportano la perdita della capacità di parlare. Gli strumenti che permettono il ripristino della comunicazione sono le BCI (Brain Computer Interface), cioè delle interfacce che collegano l’attività celebrale ad un computer che ne registra e ne interpreta le variazioni. Una caratteristica comune alla maggior parte di tali strumenti è che la comunicazione permessa dalle BCIs risulta solitamente essere molto lenta rispetto alla capacità comunicativa del linguaggio naturale poiché si riescono a riprodurre solo 5/6 parole al minuto. Per questo motivo negli ultimi dieci anni la ricerca si è concentrata su altre possibili soluzioni in cui le tecniche di BCIs fossero in grado di controllare un sintetizzatore vocale in tempo reale al fine di ripristinare una comunicazione fluente. In questo contesto si inserisce l’obiettivo della tesi che consiste nel confrontare tre differenti sistemi di speech BCI in grado di riprodurre un discorso fluente e intelligibile. I metodi di BCI confrontati nel presente elaborato sintetizzano il parlato attraverso la rilevazione invasiva dell’attività cerebrale misurata tramite elettrocorticografia (ECoG) da specifiche aree del cervello deputate al linguaggio. Tutti i metodi descritti decodificano l’attività cerebrale attraverso delle reti neurali, e utilizzano l’attività cerebrale direttamente collegata alla produzione del linguaggio per controllare il sistema di speech BCI, attuando quindi una comunicazione diretta. Ad oggi questi innovativi sistemi di speech BCI rappresentano la soluzione più promettente per risolvere problemi di comunicazione per persone affette da SLA e LIS poiché permettono di ripristinare una capacità comunicativa molto simile a quella del linguaggio naturale migliorando la qualità della vita del paziente

Identification of brain epileptiform discharges from electroencephalograms

Brain interictal epileptiform discharges (IEDs), as the fundamental indicators of seizure, are transient events occurring between two or before seizure onsets, captured using electroencephalogram (EEG). For epilepsy diagnosis and localization of seizure sources, both interictal and ictal recordings are extremely informative. Accurate detection of IEDs from over the scalp helps faster diagnosis of epilepsy. The scalp EEG (sEEG) suffers from a low signal-to-noise ratio and high attenuation of IEDs due to the high skull electrical impedance. On the other hand, the intracranial EEG (iEEG) recorded using implanted electrodes enjoys high temporal-spatial resolution and enables capturing most IEDs. Therefore, in this thesis, the focus is on the identification of IEDs from the concurrent scalp and intracranial EEGs. Multi-way analysis provides an opportunity to jointly analyse the data in different domains. IEDs may share some features within and between the segments. We have developed methods based on multi-way analysis and tensor factorization to detect the IEDs from the concurrent sEEG in both segmented and real-time approaches. The diversities in IED morphology, strength, and source location within the brain cause a great deal of uncertainty in their labeling by clinicians. We have exploited and incorporated this uncertainty (the probability of the waveform being an IED) in an IED detection system. Furthermore, IEDs are naturally sparse. We have benefited from the sparsity of IED waveforms in developing an algorithm to exploit sparse common features among the IED segments, referred to as sparse common feature analysis. By mapping sEEG to iEEG, the sEEG quality is improved. In this thesis, the proposed tensor factorization maps the time-frequency features of sEEG to those of iEEG to detect the IEDs from over the scalp with high sensitivity. We have concatenated time, frequency, and channel modes of iEEG recordings into a tensor. After decomposing the tensor into temporal, spectral, and spatial components, the EEG time-frequency features have been extracted and projected onto the temporal components. Furthermore, we have developed two novel algorithms based on generative adversarial networks to map the raw sEEG to iEEG. As a result of this work, the visibility of IEDs from sEEG has over 4-fold improvement. Additionally, the outcome paves the path for future research in epilepsy prediction, seizure source localisation, and modeling the brain seizure pathways

A Unique Method of Using Information Entropy to Evaluate the Reliability of Deep Neural Network Predictions on Intracranial Electroencephalogram

Deep Neural networks (DNN) are fundamentally information processing machines, which synthesize the complex patterns in input to arrive at solutions, with applications in various fields. One major question when working with the DNN is, which features in the input lead to a specific decision by DNN. One of the common methods of addressing this question involve generation of heatmaps. Another pertinent question is how effectively DNN has captured the entire information presented in the input, which can potentially be addressed with complexity measures of the inputs. In the case of patients with intractable epilepsy, appropriate clinical decision making depends on the interpretation of the brain signals, as recorded in the form of Electroencephalogram (EEG), which in most of the cases will be recorded through intracranial monitoring (iEEG)). In current clinical settings, the iEEG is visually inspected by the clinicians to arrive at decisions regarding the location of the epileptogenic zones which is used in the determination of surgical planning. Visual inspection and decision making is a very tedious and potentially error prone approach, given the massive amount of data that need to be evaluated in a limited amount of time. We developed a DNN model to evaluate iEEG to classify signals arising from epileptic and non-epileptic zones. One of the challenges of incorporating the deep neural network tools in the medical decision making is the black box nature of these tools. To further analyze the underlying reasons for DNN\u27s decision regarding iEEG, we used heatmapping and signal processing tools to better understand the decision-making process of DNN. We were able to demonstrate that the energy rich regions, as captured by analytical signals, is identified by DNN as potentially epileptogenic, when arriving at decisions. We explored the DNN\u27s ability to capture the details of the signal with information theoretical approaches. We introduced a measure of confidence of DNN predictions, named certainty index, which is calculated based on the overall outputs in the penultimate layer of the network. We employed the method of Sample Entropy (SampEn) and were able to demonstrate that the DNN\u27s prediction certainty is related to how effectively the heatmap is correlated to the SampEn of the entire signal. We explored the parameter space of the SampEn calculation and demonstrate that the relationship between SampEn and certainty of DNN predictions hold even on changing the estimation parameters. Further we were able to demonstrate that the rate of change of relationship between the DNN output and activation map, as a function of the sequential DNN layers, is related to the SampEn of the signal. This observation suggests that the speed at which DNN captures the results is directly proportional to the information content in the signal

Machine Learning and Deep Learning Approaches for Brain Disease Diagnosis : Principles and Recent Advances

This work was supported in part by the National Research Foundation of Korea-Grant funded by the Korean Government (Ministry of Science and ICT) under Grant NRF 2020R1A2B5B02002478, and in part by Sejong University through its Faculty Research Program under Grant 20212023.Peer reviewedPublisher PD