Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization

Most current multi-modal summarization methods follow a cascaded manner, where an off-the-shelf object detector is first used to extract visual features, then these features are fused with language representations to generate the summary with an encoder-decoder model. The cascaded way cannot capture the semantic alignments between images and paragraphs, which are crucial to a precise summary. In this paper, we propose ViL-Sum to jointly model paragraph-level \textbf{Vi}sion-\textbf{L}anguage Semantic Alignment and Multi-Modal \textbf{Sum}marization. The core of ViL-Sum is a joint multi-modal encoder with two well-designed tasks, image reordering and image selection. The joint multi-modal encoder captures the interactions between modalities, where the reordering task guides the model to learn paragraph-level semantic alignment and the selection task guides the model to selected summary-related images in the final summary. Experimental results show that our proposed ViL-Sum significantly outperforms current state-of-the-art methods. In further analysis, we find that two well-designed tasks and joint multi-modal encoder can effectively guide the model to learn reasonable paragraphs-images and summary-images relations

Retrieval-Augmented Classification with Decoupled Representation

Retrieval augmented methods have shown promising results in various classification tasks. However, existing methods focus on retrieving extra context to enrich the input, which is noise sensitive and non-expandable. In this paper, following this line, we propose a $k$-nearest-neighbor (KNN) -based method for retrieval augmented classifications, which interpolates the predicted label distribution with retrieved instances' label distributions. Different from the standard KNN process, we propose a decoupling mechanism as we find that shared representation for classification and retrieval hurts performance and leads to training instability. We evaluate our method on a wide range of classification datasets. Experimental results demonstrate the effectiveness and robustness of our proposed method. We also conduct extra experiments to analyze the contributions of different components in our model.\footnote{\url{https://github.com/xnliang98/knn-cls-w-decoupling}}Comment: preprin

The Odorant-Binding Protein Gene obp11 Shows Different Spatiotemporal Roles in the Olfactory System of Apis mellifera ligustica and Apis cerana cerana

Odorant-binding proteins participate in the olfactory system of the honeybee. Apis mellifera ligustica and Apis cerana cerana are species of honeybee that have different biologic functions. The two species have diversified olfactory systems, with A. cerana displaying sensitive olfactory involvement in collecting nectar and pollen from small plants; and A. mellifera collecting from large nectariferous plants. We hypothesized that, given this difference in biologic activity, the gene obp11 of A. mellifera and A. cerana may show different olfactory expression patterns. We cloned and sequenced the obp11 genes from A. mellifera (Amobp11) and A. cerana (Acobp11). Using quantitative real-time PCR, we demonstrated that nurse workers, which have the highest olfactory sensitivity in the A. mellifera hive, have the highest expression of Amobp11; whereas 1-day-emerged  workers, which have lowest olfactory sensitivity, have correspondingly low expression. However, the highest expression of Acobp11 is observed for foragers, which display the highest olfactory sensitivity in the A. cerana population. The OBP11 protein from the two species is highly conserved, with an apparent molecular weight and predicted extracellular localization that is similar to other OBP proteins. The expression of the obp11 gene in A. mellifera and A. cerana correlates with the different roles of the olfactory system for the two different species. These findings support the critical role of odorant-binding proteins in the Apis olfactory syste

GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking

Retrieval-enhanced text generation, which aims to leverage passages retrieved from a large passage corpus for delivering a proper answer given the input query, has shown remarkable progress on knowledge-intensive language tasks such as open-domain question answering and knowledge-enhanced dialogue generation. However, the retrieved passages are not ideal for guiding answer generation because of the discrepancy between retrieval and generation, i.e., the candidate passages are all treated equally during the retrieval procedure without considering their potential to generate the proper answers. This discrepancy makes a passage retriever deliver a sub-optimal collection of candidate passages to generate answers. In this paper, we propose the GeneRative Knowledge Improved Passage Ranking (GripRank) approach, addressing the above challenge by distilling knowledge from a generative passage estimator (GPE) to a passage ranker, where the GPE is a generative language model used to measure how likely the candidate passages can generate the proper answer. We realize the distillation procedure by teaching the passage ranker learning to rank the passages ordered by the GPE. Furthermore, we improve the distillation quality by devising a curriculum knowledge distillation mechanism, which allows the knowledge provided by the GPE can be progressively distilled to the ranker through an easy-to-hard curriculum, enabling the passage ranker to correctly recognize the provenance of the answer from many plausible candidates. We conduct extensive experiments on four datasets across three knowledge-intensive language tasks. Experimental results show advantages over the state-of-the-art methods for both passage ranking and answer generation on the KILT benchmark.Comment: 11 pages, 4 figure

Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System

Large-scale Language Models (LLMs) are constrained by their inability to process lengthy inputs. To address this limitation, we propose the Self-Controlled Memory (SCM) system to unleash infinite-length input capacity for large-scale language models. Our SCM system is composed of three key modules: the language model agent, the memory stream, and the memory controller. The language model agent iteratively processes ultra-long inputs and stores all historical information in the memory stream. The memory controller provides the agent with both long-term memory (archived memory) and short-term memory (flash memory) to generate precise and coherent responses. The controller determines which memories from archived memory should be activated and how to incorporate them into the model input. Our SCM system can be integrated with any LLMs to enable them to process ultra-long texts without any modification or fine-tuning. Experimental results show that our SCM system enables LLMs, which are not optimized for multi-turn dialogue, to achieve multi-turn dialogue capabilities that are comparable to ChatGPT, and to outperform ChatGPT in scenarios involving ultra-long document summarization or long-term conversations. Additionally, we will supply a test set, which covers common long-text input scenarios, for evaluating the abilities of LLMs in processing long documents.~\footnote{Working in progress.}\footnote{\url{https://github.com/wbbeyourself/SCM4LLMs}}Comment: Working in progres

Learning to Copy Coherent Knowledge for Response Generation

Knowledge-driven dialog has shown remarkable performance to alleviate the problem of generating uninformative responses in the dialog system. However, incorporating knowledge coherently and accurately into response generation is still far from being solved. Previous works dropped into the paradigm of non-goal-oriented knowledge-driven dialog, they are prone to ignore the effect of dialog goal, which has potential impacts on knowledge exploitation and response generation. To address this problem, this paper proposes a Goal-Oriented Knowledge Copy network, GOKC. Specifically, a goal-oriented knowledge discernment mechanism is designed to help the model discern the knowledge facts that are highly correlated to the dialog goal and the dialog context. Besides, a context manager is devised to copy facts not only from the discerned knowledge but also from the dialog goal and the dialog context, which allows the model to accurately restate the facts in the generated response. The empirical studies are conducted on two benchmarks of goal-oriented knowledge-driven dialog generation. The results show that our model can significantly outperform several state-of-the-art models in terms of both automatic evaluation and human judgments

KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation

Existing knowledge-grounded conversation systems generate responses typically in a retrieve-then-generate manner. They require a large knowledge base and a strong knowledge retrieval component, which is time- and resource-consuming. In this paper, we address the challenge by leveraging the inherent knowledge encoded in the pre-trained language models (PLMs). We propose Knowledgeable Prefix Tuning (KnowPrefix-Tuning), a two-stage tuning framework, bypassing the retrieval process in a knowledge-grounded conversation system by injecting prior knowledge into the lightweight knowledge prefix. The knowledge prefix is a sequence of continuous knowledge-specific vectors that can be learned during training. In addition, we propose a novel interactive re-parameterization mechanism that allows the prefix to interact fully with the PLM during the optimization of response generation. Experimental results demonstrate that KnowPrefix-Tuning outperforms fine-tuning and other lightweight tuning approaches, and performs comparably with strong retrieval-based baselines while being $3\times$ faster during inference.Comment: Accepted by ECML-PKDD 2023 (Research Track

Nuclear Pore Permeabilization Is a Convergent Signaling Event in Effector-Triggered Immunity

Nuclear transport of immune receptors, signal transducers, and transcription factors is an essential regulatory mechanism for immune activation. Whether and how this process is regulated at the level of the nuclear pore complex (NPC) remains unclear. Here, we report that CPR5, which plays a key inhibitory role in effector-triggered immunity (ETI) and programmed cell death (PCD) in plants, is a novel transmembrane nucleoporin. CPR5 associates with anchors of the NPC selective barrier to constrain nuclear access of signaling cargos and sequesters cyclin-dependent kinase inhibitors (CKIs) involved in ETI signal transduction. Upon activation by immunoreceptors, CPR5 undergoes an oligomer to monomer conformational switch, which coordinates CKI release for ETI signaling and reconfigures the selective barrier to allow significant influx of nuclear signaling cargos through the NPC. Consequently, these coordinated NPC actions result in simultaneous activation of diverse stress-related signaling pathways and constitute an essential regulatory mechanism specific for ETI/PCD induction

Tobacco exposure primes the secretion of CCL21 positively associated with tertiary lymphoid structure and response to immunotherapy

Background It has been reported that smoking history as a predictor of immunotherapy efficacy in patients with advanced lung cancer, however, the underlying mechanisms of this phenomenon remain largely unknown.Methods The patients with lung adenocarcinoma’s (LUAD) cohort and the orthotopical transplanted mouse model were used to explore the correlation between smoking status and tertiary lymphoid structure (TLS) and chemokine CCL21, respectively. Cell adhesion and co-immunoprecipitation assays were performed to explore the interaction between CD4+T cells and CD20+B cells under tobacco exposure. Chromatin immunoprecipitation-PCR was used to dissect the mechanism of upregulated CCL21 secretion in tobacco treatment. Serum CCL21 level was recorded in patients with LUAD treated with immunotherapy.Results Here we observed that individuals with a smoking history exhibit an increased quantity and maturation level of TLS compared with non-smokers, along with higher levels of CCL21 secretion. Tobacco exposure promoted CCL21 expression in an epithelial cell-intrinsic manner, of which BaP, the main component of tobacco, facilitated the nuclear retention of the aryl hydrocarbon receptor that occupied the promoter of CCL21. Additionally, the activated CCL21/CCR7 axis increased the CD11a expression of CD4+T cells, boosting the interaction with CD20+B cells dependent on ICAM1, which potentially induced the TLSs formation. Patients with elevated serum levels of CCL21 benefited more from immunotherapy.Conclusions Patients with a smoking history exhibited higher levels of TLS via the CCL21-dependent mechanism, serum CCL21 was identified as a reliable biomarker for predicting the efficacy of immunotherapy