Transfer: Cross Modality Knowledge Transfer using Adversarial Networks -- A Study on Gesture Recognition

Knowledge transfer across sensing technology is a novel concept that has been recently explored in many application domains, including gesture-based human computer interaction. The main aim is to gather semantic or data driven information from a source technology to classify / recognize instances of unseen classes in the target technology. The primary challenge is the significant difference in dimensionality and distribution of feature sets between the source and the target technologies. In this paper, we propose TRANSFER, a generic framework for knowledge transfer between a source and a target technology. TRANSFER uses a language-based representation of a hand gesture, which captures a temporal combination of concepts such as handshape, location, and movement that are semantically related to the meaning of a word. By utilizing a pre-specified syntactic structure and tokenizer, TRANSFER segments a hand gesture into tokens and identifies individual components using a token recognizer. The tokenizer in this language-based recognition system abstracts the low-level technology-specific characteristics to the machine interface, enabling the design of a discriminator that learns technology-invariant features essential for recognition of gestures in both source and target technologies. We demonstrate the usage of TRANSFER for three different scenarios: a) transferring knowledge across technology by learning gesture models from video and recognizing gestures using WiFi, b) transferring knowledge from video to accelerometer, and d) transferring knowledge from accelerometer to WiFi signals

Merging Deep Learning with Expert Knowledge for Seizure Onset Zone localization from rs-fMRI in Pediatric Pharmaco Resistant Epilepsy

Surgical disconnection of Seizure Onset Zones (SOZs) at an early age is an effective treatment for Pharmaco-Resistant Epilepsy (PRE). Pre-surgical localization of SOZs with intra-cranial EEG (iEEG) requires safe and effective depth electrode placement. Resting-state functional Magnetic Resonance Imaging (rs-fMRI) combined with signal decoupling using independent component (IC) analysis has shown promising SOZ localization capability that guides iEEG lead placement. However, SOZ ICs identification requires manual expert sorting of 100s of ICs per patient by the surgical team which limits the reproducibility and availability of this pre-surgical screening. Automated approaches for SOZ IC identification using rs-fMRI may use deep learning (DL) that encodes intricacies of brain networks from scarcely available pediatric data but has low precision, or shallow learning (SL) expert rule-based inference approaches that are incapable of encoding the full spectrum of spatial features. This paper proposes DeepXSOZ that exploits the synergy between DL based spatial feature and SL based expert knowledge encoding to overcome performance drawbacks of these strategies applied in isolation. DeepXSOZ is an expert-in-the-loop IC sorting technique that a) can be configured to either significantly reduce expert sorting workload or operate with high sensitivity based on expertise of the surgical team and b) can potentially enable the usage of rs-fMRI as a low cost outpatient pre-surgical screening tool. Comparison with state-of-art on 52 children with PRE shows that DeepXSOZ achieves sensitivity of 89.79%, precision of 93.6% and accuracy of 84.6%, and reduces sorting effort by 6.7-fold. Knowledge level ablation studies show a pathway towards maximizing patient outcomes while optimizing the machine-expert collaboration for various scenarios.Comment: This paper is currently under review in IEEE Journa

High Fidelity Fast Simulation of Human in the Loop Human in the Plant (HIL-HIP) Systems

Non-linearities in simulation arise from the time variance in wireless mobile networks when integrated with human in the loop, human in the plant (HIL-HIP) physical systems under dynamic contexts, leading to simulation slowdown. Time variance is handled by deriving a series of piece wise linear time invariant simulations (PLIS) in intervals, which are then concatenated in time domain. In this paper, we conduct a formal analysis of the impact of discretizing time-varying components in wireless network-controlled HIL-HIP systems on simulation accuracy and speedup, and evaluate trade-offs with reliable guarantees. We develop an accurate simulation framework for an artificial pancreas wireless network system that controls blood glucose in Type 1 Diabetes patients with time varying properties such as physiological changes associated with psychological stress and meal patterns. PLIS approach achieves accurate simulation with greater than 2.1 times speedup than a non-linear system simulation for the given dataset.Comment: To appear in ACM MSWIM 202

EdGCon: Auto-assigner of Iconicity Ratings Grounded by Lexical Properties to Aid in Generation of Technical Gestures

Gestures that share similarities in their forms and are related in their meanings, should be easier for learners to recognize and incorporate into their existing lexicon. In that regard, to be more readily accepted as standard by the Deaf and Hard of Hearing community, technical gestures in American Sign Language (ASL) will optimally share similar in forms with their lexical neighbors. We utilize a lexical database of ASL, ASL-LEX, to identify lexical relations within a set of technical gestures. We use automated identification for 3 unique sub-lexical properties in ASL- location, handshape and movement. EdGCon assigned an iconicity rating based on the lexical property similarities of the new gesture with an existing set of technical gestures and the relatedness of the meaning of the new technical word to that of the existing set of technical words. We collected 30 ad hoc crowdsourced technical gestures from different internet websites and tested them against 31 gestures from the DeafTEC technical corpus. We found that EdGCon was able to correctly auto-assign the iconicity ratings 80.76% of the time.Comment: Accepted for publication in ACM SAC 202

Data_Sheet_1_The expert's knowledge combined with AI outperforms AI alone in seizure onset zone localization using resting state fMRI.docx

We evaluated whether integration of expert guidance on seizure onset zone (SOZ) identification from resting state functional MRI (rs-fMRI) connectomics combined with deep learning (DL) techniques enhances the SOZ delineation in patients with refractory epilepsy (RE), compared to utilizing DL alone. Rs-fMRI was collected from 52 children with RE who had subsequently undergone ic-EEG and then, if indicated, surgery for seizure control (n = 25). The resting state functional connectomics data were previously independently classified by two expert epileptologists, as indicative of measurement noise, typical resting state network connectivity, or SOZ. An expert knowledge integrated deep network was trained on functional connectomics data to identify SOZ. Expert knowledge integrated with DL showed a SOZ localization accuracy of 84.8 ± 4.5% and F1 score, harmonic mean of positive predictive value and sensitivity, of 91.7 ± 2.6%. Conversely, a DL only model yielded an accuracy of <50% (F1 score 63%). Activations that initiate in gray matter, extend through white matter, and end in vascular regions are seen as the most discriminative expert-identified SOZ characteristics. Integration of expert knowledge of functional connectomics can not only enhance the performance of DL in localizing SOZ in RE but also lead toward potentially useful explanations of prevalent co-activation patterns in SOZ. RE with surgical outcomes and preoperative rs-fMRI studies can yield expert knowledge most salient for SOZ identification.</p

Table_4_Automated seizure onset zone locator from resting-state functional MRI in drug-resistant epilepsy.docx

ObjectiveAccurate localization of a seizure onset zone (SOZ) from independent components (IC) of resting-state functional magnetic resonance imaging (rs-fMRI) improves surgical outcomes in children with drug-resistant epilepsy (DRE). Automated IC sorting has limited success in identifying SOZ localizing ICs in adult normal rs-fMRI or uncategorized epilepsy. Children face unique challenges due to the developing brain and its associated surgical risks. This study proposes a novel SOZ localization algorithm (EPIK) for children with DRE.MethodsEPIK is developed in a phased approach, where fMRI noise-related biomarkers are used through high-fidelity image processing techniques to eliminate noise ICs. Then, the SOZ markers are used through a maximum likelihood-based classifier to determine SOZ localizing ICs. The performance of EPIK was evaluated on a unique pediatric DRE dataset (n = 52). A total of 24 children underwent surgical resection or ablation of an rs-fMRI identified SOZ, concurrently evaluated with an EEG and anatomical MRI. Two state-of-art techniques were used for comparison: (a) least squares support-vector machine and (b) convolutional neural networks. The performance was benchmarked against expert IC sorting and Engel outcomes for surgical SOZ resection or ablation. The analysis was stratified across age and sex.ResultsEPIK outperformed state-of-art techniques for SOZ localizing IC identification with a mean accuracy of 84.7% (4% higher), a precision of 74.1% (22% higher), a specificity of 81.9% (3.2% higher), and a sensitivity of 88.6% (16.5% higher). EPIK showed consistent performance across age and sex with the best performance in those SignificanceAutomated SOZ localization from rs-fMRI, validated against surgical outcomes, indicates the potential for clinical feasibility. It eliminates the need for expert sorting, outperforms prior automated methods, and is consistent across age and sex.</p

Table_3_Automated seizure onset zone locator from resting-state functional MRI in drug-resistant epilepsy.docx

ObjectiveAccurate localization of a seizure onset zone (SOZ) from independent components (IC) of resting-state functional magnetic resonance imaging (rs-fMRI) improves surgical outcomes in children with drug-resistant epilepsy (DRE). Automated IC sorting has limited success in identifying SOZ localizing ICs in adult normal rs-fMRI or uncategorized epilepsy. Children face unique challenges due to the developing brain and its associated surgical risks. This study proposes a novel SOZ localization algorithm (EPIK) for children with DRE.MethodsEPIK is developed in a phased approach, where fMRI noise-related biomarkers are used through high-fidelity image processing techniques to eliminate noise ICs. Then, the SOZ markers are used through a maximum likelihood-based classifier to determine SOZ localizing ICs. The performance of EPIK was evaluated on a unique pediatric DRE dataset (n = 52). A total of 24 children underwent surgical resection or ablation of an rs-fMRI identified SOZ, concurrently evaluated with an EEG and anatomical MRI. Two state-of-art techniques were used for comparison: (a) least squares support-vector machine and (b) convolutional neural networks. The performance was benchmarked against expert IC sorting and Engel outcomes for surgical SOZ resection or ablation. The analysis was stratified across age and sex.ResultsEPIK outperformed state-of-art techniques for SOZ localizing IC identification with a mean accuracy of 84.7% (4% higher), a precision of 74.1% (22% higher), a specificity of 81.9% (3.2% higher), and a sensitivity of 88.6% (16.5% higher). EPIK showed consistent performance across age and sex with the best performance in those SignificanceAutomated SOZ localization from rs-fMRI, validated against surgical outcomes, indicates the potential for clinical feasibility. It eliminates the need for expert sorting, outperforms prior automated methods, and is consistent across age and sex.</p

Table_1_Automated seizure onset zone locator from resting-state functional MRI in drug-resistant epilepsy.docx

ObjectiveAccurate localization of a seizure onset zone (SOZ) from independent components (IC) of resting-state functional magnetic resonance imaging (rs-fMRI) improves surgical outcomes in children with drug-resistant epilepsy (DRE). Automated IC sorting has limited success in identifying SOZ localizing ICs in adult normal rs-fMRI or uncategorized epilepsy. Children face unique challenges due to the developing brain and its associated surgical risks. This study proposes a novel SOZ localization algorithm (EPIK) for children with DRE.MethodsEPIK is developed in a phased approach, where fMRI noise-related biomarkers are used through high-fidelity image processing techniques to eliminate noise ICs. Then, the SOZ markers are used through a maximum likelihood-based classifier to determine SOZ localizing ICs. The performance of EPIK was evaluated on a unique pediatric DRE dataset (n = 52). A total of 24 children underwent surgical resection or ablation of an rs-fMRI identified SOZ, concurrently evaluated with an EEG and anatomical MRI. Two state-of-art techniques were used for comparison: (a) least squares support-vector machine and (b) convolutional neural networks. The performance was benchmarked against expert IC sorting and Engel outcomes for surgical SOZ resection or ablation. The analysis was stratified across age and sex.ResultsEPIK outperformed state-of-art techniques for SOZ localizing IC identification with a mean accuracy of 84.7% (4% higher), a precision of 74.1% (22% higher), a specificity of 81.9% (3.2% higher), and a sensitivity of 88.6% (16.5% higher). EPIK showed consistent performance across age and sex with the best performance in those SignificanceAutomated SOZ localization from rs-fMRI, validated against surgical outcomes, indicates the potential for clinical feasibility. It eliminates the need for expert sorting, outperforms prior automated methods, and is consistent across age and sex.</p

Table_2_Automated seizure onset zone locator from resting-state functional MRI in drug-resistant epilepsy.docx

ObjectiveAccurate localization of a seizure onset zone (SOZ) from independent components (IC) of resting-state functional magnetic resonance imaging (rs-fMRI) improves surgical outcomes in children with drug-resistant epilepsy (DRE). Automated IC sorting has limited success in identifying SOZ localizing ICs in adult normal rs-fMRI or uncategorized epilepsy. Children face unique challenges due to the developing brain and its associated surgical risks. This study proposes a novel SOZ localization algorithm (EPIK) for children with DRE.MethodsEPIK is developed in a phased approach, where fMRI noise-related biomarkers are used through high-fidelity image processing techniques to eliminate noise ICs. Then, the SOZ markers are used through a maximum likelihood-based classifier to determine SOZ localizing ICs. The performance of EPIK was evaluated on a unique pediatric DRE dataset (n = 52). A total of 24 children underwent surgical resection or ablation of an rs-fMRI identified SOZ, concurrently evaluated with an EEG and anatomical MRI. Two state-of-art techniques were used for comparison: (a) least squares support-vector machine and (b) convolutional neural networks. The performance was benchmarked against expert IC sorting and Engel outcomes for surgical SOZ resection or ablation. The analysis was stratified across age and sex.ResultsEPIK outperformed state-of-art techniques for SOZ localizing IC identification with a mean accuracy of 84.7% (4% higher), a precision of 74.1% (22% higher), a specificity of 81.9% (3.2% higher), and a sensitivity of 88.6% (16.5% higher). EPIK showed consistent performance across age and sex with the best performance in those SignificanceAutomated SOZ localization from rs-fMRI, validated against surgical outcomes, indicates the potential for clinical feasibility. It eliminates the need for expert sorting, outperforms prior automated methods, and is consistent across age and sex.</p