Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Multimodal language analysis often considers relationships between features based on text and those based on acoustical and visual properties. Text features typically outperform non-text features in sentiment analysis or emotion recognition tasks in part because the text features are derived from advanced language models or word embeddings trained on massive data sources while audio and video features are human-engineered and comparatively underdeveloped. Given that the text, audio, and video are describing the same utterance in different ways, we hypothesize that the multimodal sentiment analysis and emotion recognition can be improved by learning (hidden) correlations between features extracted from the outer product of text and audio (we call this text-based audio) and analogous text-based video. This paper proposes a novel model, the Interaction Canonical Correlation Network (ICCN), to learn such multimodal embeddings. ICCN learns correlations between all three modes via deep canonical correlation analysis (DCCA) and the proposed embeddings are then tested on several benchmark datasets and against other state-of-the-art multimodal embedding algorithms. Empirical results and ablation studies confirm the effectiveness of ICCN in capturing useful information from all three views

Development of a growth-coupled selection platform for directed evolution of heme biosynthetic enzymes in Corynebacterium glutamicum

Heme is an important tetrapyrrole compound, and has been widely applied in food and medicine industries. Although microbial production of heme has been developed with metabolic engineering strategies during the past 20 years, the production levels are relatively low due to the multistep enzymatic processes and complicated regulatory mechanisms of microbes. Previous studies mainly adopted the strategies of strengthening precursor supply and product transportation to engineer microbes for improving heme biosynthesis. Few studies focused on the engineering and screening of efficient enzymes involved in heme biosynthesis. Herein, a growth-coupled, high-throughput selection platform based on the detoxification of Zinc-protoporphyrin IX (an analogue of heme) was developed and applied to directed evolution of coproporphyrin ferrochelatase, catalyzing the insertion of metal ions into porphyrin ring to generate heme or other tetrapyrrole compounds. A mutant with 3.03-fold increase in kcat/KM was selected. Finally, growth-coupled directed evolution of another three key enzymes involved in heme biosynthesis was tested by using this selection platform. The growth-coupled selection platform developed here can be a simple and effective strategy for directed evolution of the enzymes involved in the biosynthesis of heme or other tetrapyrrole compounds

MCL1 gene co-expression module stratifies multiple myeloma and predicts response to proteasome inhibitor-based therapy

Multiple myeloma (MM) is the second most common hematologic cancer, characterized by abnormal accumulation of plasma cells in the bone marrow. The extensive biological and clinical heterogeneity of MM hinders effective treatment and etiology research. Several molecular classification systems of prognostic impact have been proposed, but they do not predict the response to treatment nor do they correlate to plasma cell development pathways. Here we describe the classification of MM into two distinct subtypes based on the expression levels of a gene module coexpressed with MCL1 (MCL1-M), a regulator of plasma cell survival. The classification system enabled prediction of the prognosis and the response to bortezomib-based therapy. Moreover, the two MM subtypes were associated with two different plasma cell differentiation pathways (enrichment of a preplasmablast signature versus aberrant expression of B cell genes). 1q gain, harboring 63 of the 87 MCL1-M members including MCL1, was found in about 80% of the MM with upregulated MCL1-M expression. Clonal analysis showed that 1q gain tended to occur as an early clonal event. Members of MCL1-M captured both MM cell-intrinsically acting signals and the signals regulating the interaction between MM cells with bone marrow microenvironment. MCL1-M members were co-expressed in mouse germinal center B cells. Together, these findings indicate that MCL1-M may play previously inadequately recognized, initiating role in the pathogenesis of MM. Our findings suggest that MCL1-M signature-based molecular clustering of MM constitutes a solid framework toward understanding the etiology of this disease and establishing personalized care. Article Summary: A pathogenic mechanism-guided molecular classification would facilitate treatment decision and etiology research of multiple myeloma. On the basis of the expression levels of a gene module coexpressed with MCL1, w

Age effect of deafening on stereotyped song maintenance in adult male bengalese finches Lonchura striata domestica

Birdsong is a complex learned vocal behavior that relies on auditory experience for development. However, it appears that among different species of close-ended songbirds, there are some variations in the necessity of auditory feedback for maintaining stereotyped adult song. In zebra finches, the deterioration of adult songs following deafness depends on the birds’ age. It is unknown whether this age effect is a general rule in other avian species as well. Therefore, we chose Bengalese finches, whose songs show more complexity and have much heavier dependency on auditory feedback than that of zebra finches, to compare the degree of song degradation after hearing loss in old (over 18 months old) and young adult birds (5–6 months old). We found that both syllable sequence and syllable phonology were much less severely affected by deafening in old adults than that in young ones. Moreover, young adults almost lost their capability to sing trills over 6 months following deafening, while old birds continued to sing plenty of trills and trilled syllables after the same period of deafening. Our results suggest that age plays an important role in affecting the dependency of adult song maintenance on auditory feedback in Bengalese finches. Furthermore, the age dependency may be a general phenomenon in different species of close-ended songbirds [Current Zoology 55(3):212–218, 2009]

A Mathematical Morphological Network Fault Diagnosis Method for Rolling Bearings Based on Acoustic Array Signal

To extract valuable characteristic information from the acoustic radiation signal of rolling bearings, a novel mathematical morphological network (MMNet) is proposed. First, a mathematical morphological network layer is constructed by leveraging the advantages of a multi-scale enhanced top-hat morphological operator (MEAVGH) that can extract positive and negative pulses, which are then integrated into the deep learning network. Second, the input signal undergoes processing with different scale structural elements (SEs) to obtain multi-branch data. This is followed by channel attention and spatial attention mechanism-based weighting of the generated multi-branch data. Finally, the fused information is fed to the neural network to yield the final result. The experimental results demonstrate the efficacy of the proposed method in extracting fault feature information, achieving a fault classification accuracy of 98.56%. Furthermore, the algorithm exhibits robustness and high training efficiency. Comparative analysis reveals that the proposed method outperforms other approaches regarding cluster analysis, accuracy, recall rate, and computational efficiency. These findings further highlight the advantages of MMNet in acoustic signal-based fault diagnosis for rolling bearings

A combination of ultrasound-targeted microbubble destruction with transplantation of bone marrow mesenchymal stem cells promotes recovery of acute liver injury

Abstract Background Bone marrow mesenchymal stem cells (BMSCs) can provide an additional source of therapeutic stem cells for regeneration of liver cells during acute liver injury (ALI). However, the insufficient hepatic homing by the transplanted BMSCs limits their applications. Ultrasound-targeted microbubble destruction (UTMD) has been reported to promote the homing of transplanted stem cells into the ischemic myocardium. In this study, we investigated whether UTMD promotes the hepatic homing of BMSCs in ALI rats and evaluated the therapeutic effect. Methods BMSCs were isolated from the femurs and tibias of Sprague-Dawley (SD) rats. The isolated BMSCs were stably transfected with a lentivirus expressing enhanced green fluorescent protein (EGFP) that can be visualized and quantified in vivo after transplantation. Both tumor necrosis factor α (TNF-α) and stromal cell-derived factor 1 (SDF-1) were used to verify the appropriate ultrasound parameters. The ALI rats were divided into four groups: control, BMSCs, UTMD, and UTMD + BMSCs. The protein and mRNA expression levels of SDF-1, intercellular cell adhesion molecule (ICAM-1), vascular cell adhesion molecule 1 (VCAM-1), hepatocyte growth factor (HGF), and monocyte chemotactic protein 1 (MCP-1) in the exposed livers were analyzed at 48 h after treatment. ALI recovery was determined by serum biochemical parameters and histology. Results The isolated rat BMSCs demonstrated a good proliferation potential that was both osteogenic and adipogenic in differentiation and expressed cluster of differentiation (CD) 29 and CD90, but not CD45 or CD11b/c. After BMSC and/or UTMD treatment, the number of GFP-labeled BMSCs in the UTMD + BMSCs group was significantly higher than that of the BMSCs group (9.8 ± 2.3 vs. 5.2 ± 1.1/per high-power field). Furthermore, the expression of GFP mRNA was performed for evaluation of the homing rate of BMSCs in injury sites as well. In addition, the expression levels of SDF-1, ICAM-1, VCAM-1, HGF, and MCP-1 were higher (p < 0.01) in UTMD+BMSCs group. The serum levels of biomarkers were significantly lower in the UTMD + BMSCs group, and the apoptotic rate of hepatocytes in the UTMD + BMSCs group was markedly lower than that of the BMSCs group (all p < 0.05). The hepatic pathology was significantly alleviated in the UTMD + BMSCs group. Conclusions UTMD treatment efficiently induced a favorable microenvironment for cell engraftment, resulting in improvement of hepatic homing of BMSCs, which was probably mediated through upregulation of the expression of adhesion molecules and cytokines. UTMD treatment appeared to be an effective and noninvasive approach to achieve better efficacy of BMSC-based therapy for repairing a severely injured liver