364 research outputs found
Graph Neural Networks with Generated Parameters for Relation Extraction
Recently, progress has been made towards improving relational reasoning in
machine learning field. Among existing models, graph neural networks (GNNs) is
one of the most effective approaches for multi-hop relational reasoning. In
fact, multi-hop relational reasoning is indispensable in many natural language
processing tasks such as relation extraction. In this paper, we propose to
generate the parameters of graph neural networks (GP-GNNs) according to natural
language sentences, which enables GNNs to process relational reasoning on
unstructured text inputs. We verify GP-GNNs in relation extraction from text.
Experimental results on a human-annotated dataset and two distantly supervised
datasets show that our model achieves significant improvements compared to
baselines. We also perform a qualitative analysis to demonstrate that our model
could discover more accurate relations by multi-hop relational reasoning
Bayesian optimization with active learning of Ta-Nb-Hf-Zr-Ti system for spin transport properties
Designing materials with enhanced spin charge conversion, i.e., with high
spin Hall conductivity (SHC) and low longitudinal electric conductivity (hence
large spin Hall angle (SHA)), is a challenging task, especially in the presence
of a vast chemical space for compositionally complex alloys (CCAs). In this
work, focusing on the Ta-Nb-Hf-Zr-Ti system, we confirm that CCAs exhibit
significant spin Hall conductivities and propose a multi-objective Bayesian
optimization approach (MOBO) incorporated with active learning (AL) in order to
screen for the optimal compositions with significant SHC and SHA. As a result,
within less than 5 iterations we are able to target the TaZr-dominated systems
displaying both high magnitudes of SHC (~-2.0 (10 cm))
and SHA (~0.03). The SHC is mainly ascribed to the extrinsic skew scattering
mechanism. Our work provides an efficient route for identifying new materials
with significant SHE, which can be straightforwardly generalized to optimize
other properties in a vast chemical space
Interdisciplinary Fairness in Imbalanced Research Proposal Topic Inference: A Hierarchical Transformer-based Method with Selective Interpolation
The objective of topic inference in research proposals aims to obtain the
most suitable disciplinary division from the discipline system defined by a
funding agency. The agency will subsequently find appropriate peer review
experts from their database based on this division. Automated topic inference
can reduce human errors caused by manual topic filling, bridge the knowledge
gap between funding agencies and project applicants, and improve system
efficiency. Existing methods focus on modeling this as a hierarchical
multi-label classification problem, using generative models to iteratively
infer the most appropriate topic information. However, these methods overlook
the gap in scale between interdisciplinary research proposals and
non-interdisciplinary ones, leading to an unjust phenomenon where the automated
inference system categorizes interdisciplinary proposals as
non-interdisciplinary, causing unfairness during the expert assignment. How can
we address this data imbalance issue under a complex discipline system and
hence resolve this unfairness? In this paper, we implement a topic label
inference system based on a Transformer encoder-decoder architecture.
Furthermore, we utilize interpolation techniques to create a series of
pseudo-interdisciplinary proposals from non-interdisciplinary ones during
training based on non-parametric indicators such as cross-topic probabilities
and topic occurrence probabilities. This approach aims to reduce the bias of
the system during model training. Finally, we conduct extensive experiments on
a real-world dataset to verify the effectiveness of the proposed method. The
experimental results demonstrate that our training strategy can significantly
mitigate the unfairness generated in the topic inference task.Comment: 19 pages, Under review. arXiv admin note: text overlap with
arXiv:2209.1391
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
The task of empowering large language models (LLMs) to accurately express
their confidence, referred to as confidence elicitation, is essential in
ensuring reliable and trustworthy decision-making processes. Previous methods,
which primarily rely on model logits, have become less suitable for LLMs and
even infeasible with the rise of closed-source LLMs (e.g., commercialized LLM
APIs). This leads to a growing need to explore the untapped area of
\emph{non-logit-based} approaches to estimate the uncertainty of LLMs. Hence,
in this study, we investigate approaches for confidence elicitation that do not
require model fine-tuning or access to proprietary information. We introduce
three categories of methods: verbalize-based, consistency-based, and their
hybrid methods for benchmarking, and evaluate their performance across five
types of datasets and four widely-used LLMs. Our analysis of these methods
uncovers several key insights: 1) LLMs often exhibit a high degree of
overconfidence when verbalizing their confidence; 2) Prompting strategies such
as CoT, Top-K and Multi-step confidences improve calibration of verbalized
confidence; 3) Consistency-based methods outperform the verbalized confidences
in most cases, with particularly notable improvements on the arithmetic
reasoning task; 4) Hybrid methods consistently deliver the best performance
over their baselines, thereby emerging as a promising state-of-the-art
approach; 5) Despite these advancements, all investigated methods continue to
struggle with challenging tasks, such as those requiring professional
knowledge, leaving significant scope for improvement of confidence elicitation.Comment: 11 Page
- …