Decoding of human identity by computer vision and neuronal vision

Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells—neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons’ sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series “24”. First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare “computer vision” with “neuronal vision”—footprints associated with each character present in the activity of a subset of neurons—and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants’ subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL

Learning characteristic neuronal dynamics of neurological disorders

The understanding of the origins of neurological disorders,such as Epilepsy, Alzheimer’s disease (AD), represents one of the most urgent and challenging areas of current scientific enquiry. In USA alone, $20\%$ of the general population fall into one of these categories, thus creating an enormous need for medical intervention. Neurological disorders lead to system-level deficits which can cause disruptions in structural connectivity, functional organization, and information processing across various brain regions. Characterizing these system-level deficits from neuronal dynamics perspective is crucial for developing targeted interventions aimed at restoring normal brain function. Fueled by the rapid advancement in neural recording technologies both at the single and population level, we develop a data driven framework to characterize the neuronal dynamics in neurological disorders that can advance our understanding, diagnosis, and treatment of neurological disorders, ultimately improving patient outcomes and quality of life.In this thesis, we study one of the most prevalent and debilitating neurological disorders called Epilepsy: approximately 65 million people suffer from it globally. In patients with epilepsy, the normal signaling mechanism in the brain is disrupted by sudden and synchronized bursts of electrical pulses, leading to recurrent episodes of seizures. Epileptic seizures can be broadly classified into two types: generalized seizures which involve multiple cross-hemisphere epileptic foci and focal seizures where the epileptic focus is localized to a specific brain region. About one-third of patients with focal seizures, cannot be treated with anti-seizure medications and they need to undergo a resective surgery for the removal of Epileptogenic zone (EZ), which is the site of the cortex responsible for generating seizures. In the pre-surgical stage, the patient is placed under intracranial EEG (iEEG) monitoring in the hospital leading to iEEG recordings during actual seizures, referred to as ictal segments. Synchronous electrical signals recorded in the ictal segments have been modeled as network/collective dynamics involving all the channels leading to automated identifications of channels that drive the observed seizures. These channels are referred to as seizure onset zones (SOZs) as they constitute parts of the EZ active during observed seizures. Another correlate of this synchronous activity are short duration oscillatory field potential, known as High Frequency Oscillations (HFO), that are observed at the level of individual electrodes of iEEG. SOZ channels have distinctly higher rates of HFOs during the ictal segment, allowing neurologists to identify SOZs without any explicit network modeling. Once the SOZ has been identified then the surgeons resect the SOZ if possible. However, this approach has led to success in about only $60-70\%$ of the patients because there might be parts of EZ that did not participate in generating seizures during the limited observation window; such unobserved parts of EZ are known as potential seizure onset zones (PSOZs). The inability of SOZ to completely encapsulate EZ in many patients, along with the hardships and risks associated with lengthy hospitalization period -often lasting two weeks or more- has prompted the need to find accurate physiological biomarkers of EZ during the interictal period, i.e, the majority of the time when patients do not have seizures.HFOs observed during ictal periods have also been observed to be present at higher rates in SOZ channels (determined from ictal periods) during interictal periods, leading to the hope that resection of channels with high interictal HFO rates would lead to seizure freedom. However, the presence of HFOs arising from cognitive processes (physiological HFOs) during interictal periods have diminished the predictive power of interictal HFO rate in the context of surgical outcome prediction. In the first part of the thesis, we develop a weakly supervised deep learning model to filter out the physiological HFOs and thus extract the pathological cluster of HFOs: epileptogenic HFOs (eHFO). In retrospective validation on a patient cohort of 15 patients, the eHFO cluster was found to be a better biomarker of EZ compared to Real HFO cluster (HFO cluster after filtering for artifacts) as it was able to correctly predict the post surgical outcomes of patients with an F1 score of $87\%$ in comparison to Real HFO cluster's $72\%$. However, when tested on a much larger patient cohort (159 patients), we found that a significant percentage of patients ($\sim 25\%$) did not have enough HFO detections and as a result the eHFO resection ratio was not able to correctly predict the surgical outcomes of those patients. Therefore, there is a need to look beyond HFOs in the space of potential interictal biomarkers of EZ.Recently, there is a growing interest in determining whether synchronization effects can be observed in the interictal period and their temporal dynamics can be leveraged to delineate EZ. The problem of studying such effects and using them for better surgical outcome prediction is still open and we address this in the second part of the thesis. In particular, we use Power-Phase coupling amongst channels to construct a sequence of directed weighted networks from interictal segments. We leverage the topological dynamics of the network, both local and global, to train a machine learning model to identify SOZ (ground truth obtained from ictal data) from purely interictal segments. The model identifies the SOZ with over $95\%$ accuracy. One of the hallmarks of the constructed networks is that they occasionally transition into a state of hyper-synchrony with SOZ and PSOZ nodes being the hub of these hyper-synchronous states. We hypothesize that these hyper-synchronous states are 'mini seizures' in the interictal phase and our machine learning model is able to identify them and use them for not only accurate SOZ identification but also identify PSOZ. The only way to validate whether our model has truly identified PSOZ and hence EZ is through surgical outcome prediction. In the third part of thesis, we construct a set of features from SOZ model prediction scores along with the constructed network flow dynamics to propose a network based biomarker of EZ. In retrospective validation on a patient cohort of 159 patients, the network based biomarker was able to correctly predict the post surgical outcomes of patients with an F1 score of $89\%$. Finally, we develop an integrated framework to exploit the interplay between the constructed network and pathological HFO cluster to propose a novel biomarker of EZ. At the heart of this framework is a regression model which predicts the pathological HFO rate using a mixture of local and global properties of the constructed network. The summary statistics of the $R^2$ of the regression model along with the previously computed features (SOZ model prediction scores and epileptic network flow dynamics) is proposed as a novel biomarker of EZ. In retrospective validation on a patient cohort of 159 patients, the novel biomarker was able to correctly predict the post surgical outcomes of patients with an F1 score of $98\%$. A closer inspection into the summary statistics of the regression model for non-seizure free patients reveals a cluster of pathological HFO whose rate cannot be explained by the constructed network thus revealing the presence of a brain region capable of generating seizures outside of the one sampled by iEEG

Optimizing detection and deep learning-based classification of pathological high-frequency oscillations in epilepsy.

OBJECTIVE: This study aimed to explore sensitive detection methods for pathological high-frequency oscillations (HFOs) to improve seizure outcomes in epilepsy surgery. METHODS: We analyzed interictal HFOs (80-500 Hz) in 15 children with medication-resistant focal epilepsy who underwent chronic intracranial electroencephalogram via subdural grids. The HFOs were assessed using the short-term energy (STE) and Montreal Neurological Institute (MNI) detectors and examined for spike association and time-frequency plot characteristics. A deep learning (DL)-based classification was applied to purify pathological HFOs. Postoperative seizure outcomes were correlated with HFO-resection ratios to determine the optimal HFO detection method. RESULTS: The MNI detector identified a higher percentage of pathological HFOs than the STE detector, but some pathological HFOs were detected only by the STE detector. HFOs detected by both detectors had the highest spike association rate. The Union detector, which detects HFOs identified by either the MNI or STE detector, outperformed other detectors in predicting postoperative seizure outcomes using HFO-resection ratios before and after DL-based purification. CONCLUSIONS: HFOs detected by standard automated detectors displayed different signal and morphological characteristics. DL-based classification effectively purified pathological HFOs. SIGNIFICANCE: Enhancing the detection and classification methods of HFOs will improve their utility in predicting postoperative seizure outcomes

Decoding of human identity by computer vision and neuronal vision

Abstract Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells—neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons’ sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series “24”. First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare “computer vision” with “neuronal vision”—footprints associated with each character present in the activity of a subset of neurons—and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants’ subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL

Refining epileptogenic high-frequency oscillations using deep learning: a reverse engineering approach.

Intracranially recorded interictal high-frequency oscillations have been proposed as a promising spatial biomarker of the epileptogenic zone. However, its visual verification is time-consuming and exhibits poor inter-rater reliability. Furthermore, no method is currently available to distinguish high-frequency oscillations generated from the epileptogenic zone (epileptogenic high-frequency oscillations) from those generated from other areas (non-epileptogenic high-frequency oscillations). To address these issues, we constructed a deep learning-based algorithm using chronic intracranial EEG data via subdural grids from 19 children with medication-resistant neocortical epilepsy to: (i) replicate human expert annotation of artefacts and high-frequency oscillations with or without spikes, and (ii) discover epileptogenic high-frequency oscillations by designing a novel weakly supervised model. The 'purification power' of deep learning is then used to automatically relabel the high-frequency oscillations to distill epileptogenic high-frequency oscillations. Using 12 958 annotated high-frequency oscillation events from 19 patients, the model achieved 96.3% accuracy on artefact detection (F1 score = 96.8%) and 86.5% accuracy on classifying high-frequency oscillations with or without spikes (F1 score = 80.8%) using patient-wise cross-validation. Based on the algorithm trained from 84 602 high-frequency oscillation events from nine patients who achieved seizure-freedom after resection, the majority of such discovered epileptogenic high-frequency oscillations were found to be ones with spikes (78.6%, P < 0.001). While the resection ratio of detected high-frequency oscillations (number of resected events/number of detected events) did not correlate significantly with post-operative seizure freedom (the area under the curve = 0.76, P = 0.06), the resection ratio of epileptogenic high-frequency oscillations positively correlated with post-operative seizure freedom (the area under the curve = 0.87, P = 0.01). We discovered that epileptogenic high-frequency oscillations had a higher signal intensity associated with ripple (80-250 Hz) and fast ripple (250-500 Hz) bands at the high-frequency oscillation onset and with a lower frequency band throughout the event time window (the inverted T-shaped), compared to non-epileptogenic high-frequency oscillations. We then designed perturbations on the input of the trained model for non-epileptogenic high-frequency oscillations to determine the model's decision-making logic. The model confidence significantly increased towards epileptogenic high-frequency oscillations by the artificial introduction of the inverted T-shaped signal template (mean probability increase: 0.285, P < 0.001), and by the artificial insertion of spike-like signals into the time domain (mean probability increase: 0.452, P < 0.001). With this deep learning-based framework, we reliably replicated high-frequency oscillation classification tasks by human experts. Using a reverse engineering technique, we distinguished epileptogenic high-frequency oscillations from others and identified its salient features that aligned with current knowledge

Characterizing physiological high-frequency oscillations using deep learning

Objective.Intracranially-recorded interictal high-frequency oscillations (HFOs) have been proposed as a promising spatial biomarker of the epileptogenic zone. However, HFOs can also be recorded in the healthy brain regions, which complicates the interpretation of HFOs. The present study aimed to characterize salient features of physiological HFOs using deep learning (DL).Approach.We studied children with neocortical epilepsy who underwent intracranial strip/grid evaluation. Time-series EEG data were transformed into DL training inputs. The eloquent cortex (EC) was defined by functional cortical mapping and used as a DL label. Morphological characteristics of HFOs obtained from EC (ecHFOs) were distilled and interpreted through a novel weakly supervised DL model.Main results.A total of 63 379 interictal intracranially-recorded HFOs from 18 children were analyzed. The ecHFOs had lower amplitude throughout the 80-500 Hz frequency band around the HFO onset and also had a lower signal amplitude in the low frequency band throughout a one-second time window than non-ecHFOs, resembling a bell-shaped template in the time-frequency map. A minority of ecHFOs were HFOs with spikes (22.9%). Such morphological characteristics were confirmed to influence DL model prediction via perturbation analyses. Using the resection ratio (removed HFOs/detected HFOs) of non-ecHFOs, the prediction of postoperative seizure outcomes improved compared to using uncorrected HFOs (area under the ROC curve of 0.82, increased from 0.76).Significance.We characterized salient features of physiological HFOs using a DL algorithm. Our results suggested that this DL-based HFO classification, once trained, might help separate physiological from pathological HFOs, and efficiently guide surgical resection using HFOs

PyHFO: lightweight deep learning-powered end-to-end high-frequency oscillations analysis application.

Objective. This study aims to develop and validate an end-to-end software platform, PyHFO, that streamlines the application of deep learning (DL) methodologies in detecting neurophysiological biomarkers for epileptogenic zones from EEG recordings.Approach. We introduced PyHFO, which enables time-efficient high-frequency oscillation (HFO) detection algorithms like short-term energy and Montreal Neurological Institute and Hospital detectors. It incorporates DL models for artifact and HFO with spike classification, designed to operate efficiently on standard computer hardware.Main results. The validation of PyHFO was conducted on three separate datasets: the first comprised solely of grid/strip electrodes, the second a combination of grid/strip and depth electrodes, and the third derived from rodent studies, which sampled the neocortex and hippocampus using depth electrodes. PyHFO demonstrated an ability to handle datasets efficiently, with optimization techniques enabling it to achieve speeds up to 50 times faster than traditional HFO detection applications. Users have the flexibility to employ our pre-trained DL model or use their EEG data for custom model training.Significance. PyHFO successfully bridges the computational challenge faced in applying DL techniques to EEG data analysis in epilepsy studies, presenting a feasible solution for both clinical and research settings. By offering a user-friendly and computationally efficient platform, PyHFO paves the way for broader adoption of advanced EEG data analysis tools in clinical practice and fosters potential for large-scale research collaborations