366 research outputs found
What changed your mind : the roles of dynamic topics and discourse in argumentation process
In our world with full of uncertainty, debates and argumentation contribute to the progress of science and society. Despite of the in- creasing attention to characterize human arguments, most progress made so far focus on the debate outcome, largely ignoring the dynamic patterns in argumentation processes. This paper presents a study that automatically analyzes the key factors in argument persuasiveness, beyond simply predicting who will persuade whom. Specifically, we propose a novel neural model that is able to dynamically track the changes of latent topics and discourse in argumentative conversations, allowing the investigation of their roles in influencing the outcomes of persuasion. Extensive experiments have been conducted on argumentative conversations on both social media and supreme court. The results show that our model outperforms state-of-the-art models in identifying persuasive arguments via explicitly exploring dynamic factors of topic and discourse. We further analyze the effects of topics and discourse on persuasiveness, and find that they are both useful -- topics provide concrete evidence while superior discourse styles may bias participants, especially in social media arguments. In addition, we draw some findings from our empirical results, which will help people better engage in future persuasive conversations
Multi-language transfer learning for low-resource legal case summarization
Analyzing and evaluating legal case reports are labor-intensive tasks for judges and lawyers, who usually base their decisions on report abstracts, legal principles, and commonsense reasoning. Thus, summarizing legal documents is time-consuming and requires excellent human expertise. Moreover, public legal corpora of specific languages are almost unavailable. This paper proposes a transfer learning approach with extractive and abstractive techniques to cope with the lack of labeled legal summarization datasets, namely a low-resource scenario. In particular, we conducted extensive multi- and cross-language experiments. The proposed work outperforms the state-of-the-art results of extractive summarization on the Australian Legal Case Reports dataset and sets a new baseline for abstractive summarization. Finally, syntactic and semantic metrics assessments have been carried out to evaluate the accuracy and the factual consistency of the machine-generated legal summaries
Emotion Embeddings \unicode{x2014} Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets
Human emotion is expressed in many communication modalities and media formats
and so their computational study is equally diversified into natural language
processing, audio signal analysis, computer vision, etc. Similarly, the large
variety of representation formats used in previous research to describe
emotions (polarity scales, basic emotion categories, dimensional approaches,
appraisal theory, etc.) have led to an ever proliferating diversity of
datasets, predictive models, and software tools for emotion analysis. Because
of these two distinct types of heterogeneity, at the expressional and
representational level, there is a dire need to unify previous work on
increasingly diverging data and label types. This article presents such a
unifying computational model. We propose a training procedure that learns a
shared latent representation for emotions, so-called emotion embeddings,
independent of different natural languages, communication modalities, media or
representation label formats, and even disparate model architectures.
Experiments on a wide range of heterogeneous affective datasets indicate that
this approach yields the desired interoperability for the sake of reusability,
interpretability and flexibility, without penalizing prediction quality. Code
and data are archived under https://doi.org/10.5281/zenodo.7405327 .Comment: 18 pages, 6 figure
Evaluating Large Language Models: A Comprehensive Survey
Large language models (LLMs) have demonstrated remarkable capabilities across
a broad spectrum of tasks. They have attracted significant attention and been
deployed in numerous downstream applications. Nevertheless, akin to a
double-edged sword, LLMs also present potential risks. They could suffer from
private data leaks or yield inappropriate, harmful, or misleading content.
Additionally, the rapid progress of LLMs raises concerns about the potential
emergence of superintelligent systems without adequate safeguards. To
effectively capitalize on LLM capacities as well as ensure their safe and
beneficial development, it is critical to conduct a rigorous and comprehensive
evaluation of LLMs.
This survey endeavors to offer a panoramic perspective on the evaluation of
LLMs. We categorize the evaluation of LLMs into three major groups: knowledge
and capability evaluation, alignment evaluation and safety evaluation. In
addition to the comprehensive review on the evaluation methodologies and
benchmarks on these three aspects, we collate a compendium of evaluations
pertaining to LLMs' performance in specialized domains, and discuss the
construction of comprehensive evaluation platforms that cover LLM evaluations
on capabilities, alignment, safety, and applicability.
We hope that this comprehensive overview will stimulate further research
interests in the evaluation of LLMs, with the ultimate goal of making
evaluation serve as a cornerstone in guiding the responsible development of
LLMs. We envision that this will channel their evolution into a direction that
maximizes societal benefit while minimizing potential risks. A curated list of
related papers has been publicly available at
https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers.Comment: 111 page
- …