Efficient Domain Adaptation of Sentence Embeddings using Adapters

Sentence embeddings enable us to capture the semantic similarity of short texts. Most sentence embedding models are trained for general semantic textual similarity (STS) tasks. Therefore, to use sentence embeddings in a particular domain, the model must be adapted to it in order to achieve good results. Usually, this is done by fine-tuning the entire sentence embedding model for the domain of interest. While this approach yields state-of-the-art results, all of the model's weights are updated during fine-tuning, making this method resource-intensive. Therefore, instead of fine-tuning entire sentence embedding models for each target domain individually, we propose to train lightweight adapters. These domain-specific adapters do not require fine-tuning all underlying sentence embedding model parameters. Instead, we only train a small number of additional parameters while keeping the weights of the underlying sentence embedding model fixed. Training domain-specific adapters allows always using the same base model and only exchanging the domain-specific adapters to adapt sentence embeddings to a specific domain. We show that using adapters for parameter-efficient domain adaptation of sentence embeddings yields competitive performance within 1% of a domain-adapted, entirely fine-tuned sentence embedding model while only training approximately 3.6% of the parameters.Comment: Accepted to the International Conference on Recent Advances in Natural Language Processing (RANLP 2023

Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches

Text classification of unseen classes is a challenging Natural Language Processing task and is mainly attempted using two different types of approaches. Similarity-based approaches attempt to classify instances based on similarities between text document representations and class description representations. Zero-shot text classification approaches aim to generalize knowledge gained from a training task by assigning appropriate labels of unknown classes to text documents. Although existing studies have already investigated individual approaches to these categories, the experiments in literature do not provide a consistent comparison. This paper addresses this gap by conducting a systematic evaluation of different similarity-based and zero-shot approaches for text classification of unseen classes. Different state-of-the-art approaches are benchmarked on four text classification datasets, including a new dataset from the medical domain. Additionally, novel SimCSE and SBERT-based baselines are proposed, as other baselines used in existing work yield weak classification results and are easily outperformed. Finally, the novel similarity-based Lbl2TransformerVec approach is presented, which outperforms previous state-of-the-art approaches in unsupervised text classification. Our experiments show that similarity-based approaches significantly outperform zero-shot approaches in most cases. Additionally, using SimCSE or SBERT embeddings instead of simpler text representations increases similarity-based classification results even further.Comment: Accepted to 6th International Conference on Natural Language Processing and Information Retrieval (NLPIR '22

Exploring the Landscape of Natural Language Processing Research

As an efficient approach to understand, generate, and process natural language texts, research in natural language processing (NLP) has exhibited a rapid spread and wide adoption in recent years. Given the increasing amount of research work in this area, several NLP-related approaches have been surveyed in the research community. However, a comprehensive study that categorizes established topics, identifies trends, and outlines areas for future research remains absent to this day. Contributing to closing this gap, we have systematically classified and analyzed research papers included in the ACL Anthology. As a result, we present a structured overview of the research landscape, provide a taxonomy of fields-of-study in NLP, analyze recent developments in NLP, summarize our findings, and highlight directions for future work.Comment: Accepted to the 14th International Conference on Recent Advances in Natural Language Processing (RANLP 2023

A Knowledge Graph Approach for Exploratory Search in Research Institutions

Over the past decades, research institutions have grown increasingly and consequently also their research output. This poses a significant challenge for researchers seeking to understand the research landscape of an institution. The process of exploring the research landscape of institutions has a vague information need, no precise goal, and is open-ended. Current applications are not designed to fulfill the requirements for exploratory search in research institutions. In this paper, we analyze exploratory search in research institutions and propose a knowledge graph-based approach to enhance this process.Comment: Accepted to 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KMI

AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity using Contrastive Learning and Structured Knowledge

Generic sentence embeddings provide a coarse-grained approximation of semantic textual similarity but ignore specific aspects that make texts similar. Conversely, aspect-based sentence embeddings provide similarities between texts based on certain predefined aspects. Thus, similarity predictions of texts are more targeted to specific requirements and more easily explainable. In this paper, we present AspectCSE, an approach for aspect-based contrastive learning of sentence embeddings. Results indicate that AspectCSE achieves an average improvement of 3.97% on information retrieval tasks across multiple aspects compared to the previous best results. We also propose using Wikidata knowledge graph properties to train models of multi-aspect sentence embeddings in which multiple specific aspects are simultaneously considered during similarity predictions. We demonstrate that multi-aspect embeddings outperform single-aspect embeddings on aspect-specific information retrieval tasks. Finally, we examine the aspect-based sentence embedding space and demonstrate that embeddings of semantically similar aspect labels are often close, even without explicit similarity training between different aspect labels.Comment: Accepted to the 14th International Conference on Recent Advances in Natural Language Processing (RANLP 2023

Microstructure-Based Lifetime Assessment of Austenitic Steel AISI 347 in View of Fatigue, Environmental Conditions and NDT

The assessment of metallic materials used in power plants’ piping represents a big challenge due to the thermal transients and the environmental conditions to which they are exposed. At present, a lack of information related to degradation mechanisms in structures and materials is covered by safety factors in its design, and in some cases, the replacement of components is prescribed after a determined period of time without knowledge of the true degree of degradation. In the collaborative project “Microstructure-based assessment of maximum service life of nuclear materials and components exposed to corrosion and fatigue (MibaLeb)”, a methodology for the assessment of materials’ degradation is being developed, which combines the use of NDT techniques for materials characterization, an optimized fatigue lifetime analysis using short time evaluation procedures (STEPs) and numerical simulations. In this investigation, the AISI 347 (X6CrNiNb18-10) is being analyzed at different conditions in order to validate the methodology. Besides microstructural analysis, tensile and fatigue tests, all to characterize the material, a pressurized hot water pipe exposed to a series of flow conditions will be evaluated in terms of full-scale testing as well as prognostic evaluation, where the latter will be based on the materials’ data generated, which should prognose changes in the material’s condition, specifically in a pre-cracked stage. This paper provides an overview of the program, while the more material’s related aspects are presented in the subsequent paper

A Short-Time Approach for Fatigue Life Evaluation of AISI 347 Steel for Nuclear Power Energy Applications

AISI 347 austenitic steel is, as an example, used in nuclear energy piping systems. Piping filled with superheated steam or cooled water is particularly exposed to high stresses, whereupon local material properties in the pipes can change significantly, especially in the case of additional corrosive influences, leading to aging of the material. In the absence of appropriate information, such local material property variations are currently covered rather blanketly by safety factors set during the design of those components. An increase in qualified information could improve the assessment of the condition of such aged components. As part of the collaborative project “Microstructure-based assessment of the maximum service life of core materials and components subjected to corrosion and fatigue (MiBaLeB)”, the short-time procedure, StrainLife, was developed and validated by several fatigue tests. With this procedure, a complete S–N curve of a material can be determined on the basis of three fatigue tests only, which reduces the effort compared to a conventional approach significantly and is thus ideal for assessing the condition of aged material, where the material is often rare, and a cost-effective answer is often very needed. The procedure described is not just limited to traditional parameters, such as stress and strain, considered in destructive testing but rather extends into parameters derived from non-destructive testing, which may allow further insight into what may be happening within a material’s microstructure. To evaluate the non-destructive quantities measured within the StrainLife procedure and to correlate them with the aging process in a material, several fatigue tests were performed on unnotched and notched specimens under cyclic loading at room and elevated temperatures, as well as under various media conditions, such as distilled water and reactor pressure vessel boiling water (BWR) conditions

The astrobiology primer: An outline of general knowledge - Version 1, 2006

Peer reviewe

Search for dark matter produced in association with bottom or top quarks in √s = 13 TeV pp collisions with the ATLAS detector

A search for weakly interacting massive particle dark matter produced in association with bottom or top quarks is presented. Final states containing third-generation quarks and miss- ing transverse momentum are considered. The analysis uses 36.1 fb−1 of proton–proton collision data recorded by the ATLAS experiment at √s = 13 TeV in 2015 and 2016. No significant excess of events above the estimated backgrounds is observed. The results are in- terpreted in the framework of simplified models of spin-0 dark-matter mediators. For colour- neutral spin-0 mediators produced in association with top quarks and decaying into a pair of dark-matter particles, mediator masses below 50 GeV are excluded assuming a dark-matter candidate mass of 1 GeV and unitary couplings. For scalar and pseudoscalar mediators produced in association with bottom quarks, the search sets limits on the production cross- section of 300 times the predicted rate for mediators with masses between 10 and 50 GeV and assuming a dark-matter mass of 1 GeV and unitary coupling. Constraints on colour- charged scalar simplified models are also presented. Assuming a dark-matter particle mass of 35 GeV, mediator particles with mass below 1.1 TeV are excluded for couplings yielding a dark-matter relic density consistent with measurements