34 research outputs found
Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance
Adopting a two-stage paradigm of pretraining followed by fine-tuning,
Pretrained Language Models (PLMs) have achieved substantial advancements in the
field of natural language processing. However, in real-world scenarios, data
labels are often noisy due to the complex annotation process, making it
essential to develop strategies for fine-tuning PLMs with such noisy labels. To
this end, we introduce an innovative approach for fine-tuning PLMs using noisy
labels, which incorporates the guidance of Large Language Models (LLMs) like
ChatGPT. This guidance assists in accurately distinguishing between clean and
noisy samples and provides supplementary information beyond the noisy labels,
thereby boosting the learning process during fine-tuning PLMs. Extensive
experiments on synthetic and real-world noisy datasets further demonstrate the
superior advantages of our framework over the state-of-the-art baselines.Comment: EMNLP Findings 202
Hypothesis Search: Inductive Reasoning with Language Models
Inductive reasoning is a core problem-solving capacity: humans can identify
underlying principles from a few examples, which can then be robustly
generalized to novel scenarios. Recent work has evaluated large language models
(LLMs) on inductive reasoning tasks by directly prompting them yielding "in
context learning." This can work well for straightforward inductive tasks, but
performs very poorly on more complex tasks such as the Abstraction and
Reasoning Corpus (ARC). In this work, we propose to improve the inductive
reasoning ability of LLMs by generating explicit hypotheses at multiple levels
of abstraction: we prompt the LLM to propose multiple abstract hypotheses about
the problem, in natural language, then implement the natural language
hypotheses as concrete Python programs. These programs can be directly verified
by running on the observed examples and generalized to novel inputs. Because of
the prohibitive cost of generation with state-of-the-art LLMs, we consider a
middle step to filter the set of hypotheses that will be implemented into
programs: we either ask the LLM to summarize into a smaller set of hypotheses,
or ask human annotators to select a subset of the hypotheses. We verify our
pipeline's effectiveness on the ARC visual inductive reasoning benchmark, its
variant 1D-ARC, and string transformation dataset SyGuS. On a random 40-problem
subset of ARC, our automated pipeline using LLM summaries achieves 27.5%
accuracy, significantly outperforming the direct prompting baseline (accuracy
of 12.5%). With the minimal human input of selecting from LLM-generated
candidates, the performance is boosted to 37.5%. (And we argue this is a lower
bound on the performance of our approach without filtering.) Our ablation
studies show that abstract hypothesis generation and concrete program
representations are both beneficial for LLMs to perform inductive reasoning
tasks
Federated Knowledge Graph Completion via Latent Embedding Sharing and Tensor Factorization
Knowledge graphs (KGs), which consist of triples, are inherently incomplete
and always require completion procedure to predict missing triples. In
real-world scenarios, KGs are distributed across clients, complicating
completion tasks due to privacy restrictions. Many frameworks have been
proposed to address the issue of federated knowledge graph completion. However,
the existing frameworks, including FedE, FedR, and FEKG, have certain
limitations. = FedE poses a risk of information leakage, FedR's optimization
efficacy diminishes when there is minimal overlap among relations, and FKGE
suffers from computational costs and mode collapse issues. To address these
issues, we propose a novel method, i.e., Federated Latent Embedding Sharing
Tensor factorization (FLEST), which is a novel approach using federated tensor
factorization for KG completion. FLEST decompose the embedding matrix and
enables sharing of latent dictionary embeddings to lower privacy risks.
Empirical results demonstrate FLEST's effectiveness and efficiency, offering a
balanced solution between performance and privacy. FLEST expands the
application of federated tensor factorization in KG completion tasks.Comment: Accepted by ICDM 202
Mitigating Popularity Bias in Recommendation with Unbalanced Interactions: A Gradient Perspective
Recommender systems learn from historical user-item interactions to identify
preferred items for target users. These observed interactions are usually
unbalanced following a long-tailed distribution. Such long-tailed data lead to
popularity bias to recommend popular but not personalized items to users. We
present a gradient perspective to understand two negative impacts of popularity
bias in recommendation model optimization: (i) the gradient direction of
popular item embeddings is closer to that of positive interactions, and (ii)
the magnitude of positive gradient for popular items are much greater than that
of unpopular items. To address these issues, we propose a simple yet efficient
framework to mitigate popularity bias from a gradient perspective.
Specifically, we first normalize each user embedding and record accumulated
gradients of users and items via popularity bias measures in model training. To
address the popularity bias issues, we develop a gradient-based embedding
adjustment approach used in model testing. This strategy is generic,
model-agnostic, and can be seamlessly integrated into most existing recommender
systems. Our extensive experiments on two classic recommendation models and
four real-world datasets demonstrate the effectiveness of our method over
state-of-the-art debiasing baselines.Comment: Recommendation System, Popularity Bia
AutoMLP: Automated MLP for Sequential Recommendations
Sequential recommender systems aim to predict users' next interested item
given their historical interactions. However, a long-standing issue is how to
distinguish between users' long/short-term interests, which may be
heterogeneous and contribute differently to the next recommendation. Existing
approaches usually set pre-defined short-term interest length by exhaustive
search or empirical experience, which is either highly inefficient or yields
subpar results. The recent advanced transformer-based models can achieve
state-of-the-art performances despite the aforementioned issue, but they have a
quadratic computational complexity to the length of the input sequence. To this
end, this paper proposes a novel sequential recommender system, AutoMLP, aiming
for better modeling users' long/short-term interests from their historical
interactions. In addition, we design an automated and adaptive search algorithm
for preferable short-term interest length via end-to-end optimization. Through
extensive experiments, we show that AutoMLP has competitive performance against
state-of-the-art methods, while maintaining linear computational complexity.Comment: Accepted by WWW'2
Preserving the woody plant tree of life in China under future climate and land-cover changes
The tree of life (TOL) is severely threatened by climate and land-cover changes. Preserving the TOL is urgent, but has not been included in the post-2020 global biodiversity framework. Protected areas (PAs) are fundamental for biological conservation. However, we know little about the effectiveness of existing PAs in preserving the TOL of plants and how to prioritize PA expansion for better TOL preservation under future climate and land-cover changes. Here, using high-resolution distribution maps of 8732 woody species in China and phylogeny-based Zonation, we find that current PAs perform poorly in preserving the TOL both at present and in 2070s. The geographical coverage of TOL branches by current PAs is approx. 9%, and less than 3% of the identified priority areas for preserving the TOL are currently protected. Interestingly, the geographical coverage of TOL branches by PAs will be improved from 9% to 52-79% by the identified priority areas for PA expansion. Human pressures in the identified priority areas are high, leading to high cost for future PA expansion. We thus suggest that besides nature reserves and national parks, other effective area-based conservation measures should be considered. Our study argues for the inclusion of preserving the TOL in the post-2020 conservation framework, and provides references for decision-makers to preserve the Earth's evolutionary history.Fil: Peng, Shijia. Peking University; ChinaFil: Hu, Ruocheng. Peking University; ChinaFil: Velazco, Santiago JosĂ© ElĂas. Consejo Nacional de Investigaciones CientĂficas y TĂ©cnicas. Centro CientĂfico TecnolĂłgico Conicet - Nordeste. Instituto de BiologĂa Subtropical. Instituto de BiologĂa Subtropical - Nodo Puerto IguazĂş | Universidad Nacional de Misiones. Instituto de BiologĂa Subtropical. Instituto de BiologĂa Subtropical - Nodo Puerto IguazĂş; Argentina. Universidade Federal da Integração Latino-Americana; Brasil. University of California; Estados UnidosFil: Luo, Yuan. Peking University; ChinaFil: Lyu, Tong. Peking University; ChinaFil: Zhang, Xiaoling. Peking University; ChinaFil: Zhang, Jian. East China Normal University; ChinaFil: Wang, Zhiheng. Peking University; Chin
Internal Feature Selection Method of CSP Based on L1-Norm and Dempster–Shafer Theory
The common spatial pattern (CSP) algorithm is a well-recognized spatial filtering method for feature extraction in motor imagery (MI)-based brain–computer interfaces (BCIs). However, due to the influence of nonstationary in electroencephalography (EEG) and inherent defects of the CSP objective function, the spatial filters, and their corresponding features are not necessarily optimal in the feature space used within CSP. In this work, we design a new feature selection method to address this issue by selecting features based on an improved objective function. Especially, improvements are made in suppressing outliers and discovering features with larger interclass distances. Moreover, a fusion algorithm based on the Dempster–Shafer theory is proposed, which takes into consideration the distribution of features. With two competition data sets, we first evaluate the performance of the improved objective functions in terms of classification accuracy, feature distribution, and embeddability. Then, a comparison with other feature selection methods is carried out in both accuracy and computational time. Experimental results show that the proposed methods consume less additional computational cost and result in a significant increase in the performance of MI-based BCI systems