Embodied Executable Policy Learning with Language-based Scene Summarization

Large Language models (LLMs) have shown remarkable success in assisting robot learning tasks, i.e., complex household planning. However, the performance of pretrained LLMs heavily relies on domain-specific templated text data, which may be infeasible in real-world robot learning tasks with image-based observations. Moreover, existing LLMs with text inputs lack the capability to evolve with non-expert interactions with environments. In this work, we introduce a novel learning paradigm that generates robots' executable actions in the form of text, derived solely from visual observations, using language-based summarization of these observations as the connecting bridge between both domains. Our proposed paradigm stands apart from previous works, which utilized either language instructions or a combination of language and visual data as inputs. Moreover, our method does not require oracle text summarization of the scene, eliminating the need for human involvement in the learning loop, which makes it more practical for real-world robot learning tasks. Our proposed paradigm consists of two modules: the SUM module, which interprets the environment using visual observations and produces a text summary of the scene, and the APM module, which generates executable action policies based on the natural language descriptions provided by the SUM module. We demonstrate that our proposed method can employ two fine-tuning strategies, including imitation learning and reinforcement learning approaches, to adapt to the target test tasks effectively. We conduct extensive experiments involving various SUM/APM model selections, environments, and tasks across 7 house layouts in the VirtualHome environment. Our experimental results demonstrate that our method surpasses existing baselines, confirming the effectiveness of this novel learning paradigm.Comment: 15 pages. arXiv admin note: text overlap with arXiv:2107.06912 by other author

Can Brain Signals Reveal Inner Alignment with Human Languages?

Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced \textbf{MTAM}, a \textbf{M}ultimodal \textbf{T}ransformer \textbf{A}lignment \textbf{M}odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at \url{https://github.com/Jason-Qiu/EEG_Language_Alignment}.Comment: EMNLP 2023 Finding

Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?

Recent advancements in Large Language Models (LLMs) have drawn increasing attention since the learned embeddings pretrained on large-scale datasets have shown powerful ability in various downstream applications. However, whether the learned knowledge by LLMs can be transferred to clinical cardiology remains unknown. In this work, we aim to bridge this gap by transferring the knowledge of LLMs to clinical Electrocardiography (ECG). We propose an approach for cardiovascular disease diagnosis and automatic ECG diagnosis report generation. We also introduce an additional loss function by Optimal Transport (OT) to align the distribution between ECG and language embedding. The learned embeddings are evaluated on two downstream tasks: (1) automatic ECG diagnosis report generation, and (2) zero-shot cardiovascular disease detection. Our approach is able to generate high-quality cardiac diagnosis reports and also achieves competitive zero-shot classification performance even compared with supervised baselines, which proves the feasibility of transferring knowledge from LLMs to the cardiac domain.Comment: EACL 202

Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment

Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. In specific, our method first decomposes both video and article into segments in order to capture the structural semantics, respectively. Then SCCS follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries

Artificial Intelligence Framework Identifies Candidate Targets for Drug Repurposing in Alzheimer’s Disease

Background: Genome-wide association studies (GWAS) have identified numerous susceptibility loci for Alzheimer’s disease (AD). However, utilizing GWAS and multi-omics data to identify high-confidence AD risk genes (ARGs) and druggable targets that can guide development of new therapeutics for patients suffering from AD has heretofore not been successful. Methods: To address this critical problem in the field, we have developed a network-based artificial intelligence framework that is capable of integrating multi-omics data along with human protein–protein interactome networks to accurately infer accurate drug targets impacted by GWAS-identified variants to identify new therapeutics. When applied to AD, this approach integrates GWAS findings, multi-omics data from brain samples of AD patients and AD transgenic animal models, drug-target networks, and the human protein–protein interactome, along with large-scale patient database validation and in vitro mechanistic observations in human microglia cells. Results: Through this approach, we identified 103 ARGs validated by various levels of pathobiological evidence in AD. Via network-based prediction and population-based validation, we then showed that three drugs (pioglitazone, febuxostat, and atenolol) are significantly associated with decreased risk of AD compared with matched control populations. Pioglitazone usage is significantly associated with decreased risk of AD (hazard ratio (HR) = 0.916, 95% confidence interval [CI] 0.861–0.974, P = 0.005) in a retrospective case-control validation. Pioglitazone is a peroxisome proliferator-activated receptor (PPAR) agonist used to treat type 2 diabetes, and propensity score matching cohort studies confirmed its association with reduced risk of AD in comparison to glipizide (HR = 0.921, 95% CI 0.862–0.984, P = 0.0159), an insulin secretagogue that is also used to treat type 2 diabetes. In vitro experiments showed that pioglitazone downregulated glycogen synthase kinase 3 beta (GSK3β) and cyclin-dependent kinase (CDK5) in human microglia cells, supporting a possible mechanism-of-action for its beneficial effect in AD. Conclusions: In summary, we present an integrated, network-based artificial intelligence methodology to rapidly translate GWAS findings and multi-omics data to genotype-informed therapeutic discovery in AD

Network medicine links SARS-CoV-2/COVID-19 infection to brain microvascular injury and neuroinflammation in dementia-like cognitive impairment

Background Dementia-like cognitive impairment is an increasingly reported complication of SARS-CoV-2 infection. However, the underlying mechanisms responsible for this complication remain unclear. A better understanding of causative processes by which COVID-19 may lead to cognitive impairment is essential for developing preventive and therapeutic interventions. Methods In this study, we conducted a network-based, multimodal omics comparison of COVID-19 and neurologic complications. We constructed the SARS-CoV-2 virus-host interactome from protein-protein interaction assay and CRISPR-Cas9-based genetic assay results and compared network-based relationships therein with those of known neurological manifestations using network proximity measures. We also investigated the transcriptomic profiles (including single-cell/nuclei RNA-sequencing) of Alzheimer’s disease (AD) marker genes from patients infected with COVID-19, as well as the prevalence of SARS-CoV-2 entry factors in the brains of AD patients not infected with SARS-CoV-2. Results We found significant network-based relationships between COVID-19 and neuroinflammation and brain microvascular injury pathways and processes which are implicated in AD. We also detected aberrant expression of AD biomarkers in the cerebrospinal fluid and blood of patients with COVID-19. While transcriptomic analyses showed relatively low expression of SARS-CoV-2 entry factors in human brain, neuroinflammatory changes were pronounced. In addition, single-nucleus transcriptomic analyses showed that expression of SARS-CoV-2 host factors (BSG and FURIN) and antiviral defense genes (LY6E, IFITM2, IFITM3, and IFNAR1) was elevated in brain endothelial cells of AD patients and healthy controls relative to neurons and other cell types, suggesting a possible role for brain microvascular injury in COVID-19-mediated cognitive impairment. Overall, individuals with the AD risk allele APOE E4/E4 displayed reduced expression of antiviral defense genes compared to APOE E3/E3 individuals. Conclusion Our results suggest significant mechanistic overlap between AD and COVID-19, centered on neuroinflammation and microvascular injury. These results help improve our understanding of COVID-19-associated neurological manifestations and provide guidance for future development of preventive or treatment interventions, although causal relationship and mechanistic pathways between COVID-19 and AD need future investigations

Family-Based Samples Can Play an Important Role in Genetic Association Studies

Over the past two decades, DNA samples from thousands of families have been collected and genotyped for linkage studies of common complex diseases such as type 2 diabetes, asthma and prostate cancer. Unfortunately, little success has been achieved in identifying genetic susceptibility risk factors through these considerable efforts. However, significant success in identifying common disease risk-associated variants has been recently achieved from genome-wide association (GWA) studies using unrelated case-control samples. These GWA studies are typically performed using population-based cases and controls that are ascertained irrespective of their family history for the disease of interest. Few genetic association studies have taken full advantage of the considerable resources that are available from the linkage-based family collections despite evidence showing cases that have a positive family history of disease are more likely to carry common genetic variants associated with disease susceptibility. Herein, we argue that population stratification is still a concern in case-control genetic association studies, despite the development of analytic methods designed to account for this source of confounding, for a subset of SNPs in the genome, most notably those SNPs in regions involved with natural selection. We note that current analytic approaches designed to address the issue of population stratification in case-control studies cannot definitively distinguish between true and false associations and we argue that family-based samples can still serve an invaluable role in following-up findings from case-control studies

Histone demethylase KDM5D drives sex-specific differences in colorectal cancer

View full abstracthttps://openworks.mdanderson.org/leading-edge/1011/thumbnail.jp

Multimodal Single-Cell/Nucleus RNA Sequencing Data Analysis Uncovers Molecular Networks Between Disease-Associated Microglia and Astrocytes With Implications for Drug Repurposing in Alzheimer’s Disease

Because disease-associated microglia (DAM) and disease-associated astrocytes (DAA) are involved in the pathophysiology of Alzheimer\u27s disease (AD), we systematically identified molecular networks between DAM and DAA to uncover novel therapeutic targets for AD. Specifically, we develop a network-based methodology that leverages single-cell/nucleus RNA sequencing data from both transgenic mouse models and AD patient brains, as well as drug-target network, metaboliteenzyme associations, the human protein-protein interactome, and large-scale longitudinal patient data. Through this approach, we find both common and unique gene network regulators between DAM (i.e., PAK1, MAPK14, and CSF1R) and DAA (i.e., NFKB1, FOS, and JUN) that are significantly enriched by neuro-inflammatory pathways and well-known genetic variants (i.e., BIN1). We identify shared immune pathways between DAM and DAA, including Th17 cell differentiation and chemokine signaling. Last, integrative metabolite-enzyme network analyses suggest that fatty acids and amino acids may trigger molecular alterations in DAM and DAA. Combining network-based prediction and retrospective case-control observations with 7.2 million individuals, we identify that usage of fluticasone (an approved glucocorticoid receptor agonist) is significantly associated with a reduced incidence of AD (hazard ratio [HR] = 0.86, 95% confidence interval [CI] 0.83-0.89, P \u3c 1.0 × 10 - 8). Propensity score-stratified cohort studies reveal that usage of mometasone (a stronger glucocorticoid receptor agonist) is significantly associated with a decreased risk of AD (HR = 0.74, 95% CI 0.68-0.81, P \u3c 1.0 × 10 - 8) compared to fluticasone after adjusting age, gender, and disease comorbidities. In summary, we present a network-based, multimodal methodology for single-cell/nucleus genomics-informed drug discovery and have identified fluticasone and mometasone as potential treatments in AD