MED-SE: Medical Entity Definition-based Sentence Embedding

We propose Medical Entity Definition-based Sentence Embedding (MED-SE), a novel unsupervised contrastive learning framework designed for clinical texts, which exploits the definitions of medical entities. To this end, we conduct an extensive analysis of multiple sentence embedding techniques in clinical semantic textual similarity (STS) settings. In the entity-centric setting that we have designed, MED-SE achieves significantly better performance, while the existing unsupervised methods including SimCSE show degraded performance. Our experiments elucidate the inherent discrepancies between the general- and clinical-domain texts, and suggest that entity-centric contrastive approaches may help bridge this gap and lead to a better representation of clinical sentences.Comment: 8 pages, 2 figures, 9 table

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

Evaluation of Large Language Models (LLMs) is challenging because aligning to human values requires the composition of multiple skills and the required set of skills varies depending on the instruction. Recent studies have evaluated the performance of LLMs in two ways, (1) automatic evaluation on several independent benchmarks and (2) human or machined-based evaluation giving an overall score to the response. However, both settings are coarse-grained evaluations, not considering the nature of user instructions that require instance-wise skill composition, which limits the interpretation of the true capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation protocol that can be used for both model-based and human-based evaluation which decomposes coarse-level scoring to an instance-wise skill set-level. Specifically, we define 12 fine-grained skills needed for LLMs to follow open-ended user instructions and construct an evaluation set by allocating a set of skills for each instance. Additionally, by annotating the target domains and difficulty level for each instance, FLASK provides a holistic view with a comprehensive analysis of a model's performance depending on skill, domain, and difficulty. Through using FLASK, we compare multiple open-sourced and proprietary LLMs and observe highly-correlated findings between model-based and human-based evaluations. FLASK enables developers to more accurately measure the model performance and how it can be improved by analyzing factors that make LLMs proficient in particular skills. For practitioners, FLASK can be used to recommend suitable models for particular situations through comprehensive comparison among various LLMs. We release the evaluation data and code implementation at https://github.com/kaistAI/FLASK

Atomic Layer Deposition of Ru Thin Film Using a Newly Synthesized Precursor with Open‐Coordinated Ligands

Abstract Ru films are grown on Pt, TiN, and SiO2 substrates via atomic layer deposition (ALD) using Ru(II)(η5‐C7H7O)(η5‐C7H9) as the novel Ru metalorganic precursor and O2 as the reactant. The ALD self‐limiting film growth is confirmed at the low temperature of 200 °C by manipulating the injection time and purge time of the Ru precursor and O2, and the saturated growth per cycle is 0.22 Å cy−1. It is confirmed that the combustion reaction occurs during the deposition process from the formation of the H2O and CO2 by‐products. Thin films with a low resistivity of 17–19 µΩ cm are grown at a thickness of ≈15 nm. The Ru incubation times are remarkably short at 200 °C (negligible on Pt, ≈30 cycles on TiN and SiO2), but increase with increasing temperature. The energy dispersive X‐ray mapping image of the Ru film on the pattern in which TiN is formed on SiO2 shows that the Ru film with a novel precursor has the intrinsic substrate selectivity at the deposition temperature of 300 °C. Furthermore, the substrate affects the properties of the Ru film. Particularly because Ti serves as an oxygen reservoir, a relatively large amount of RuOx is produced on the TiN substrate