18 research outputs found
ChatGPT as a Factual Inconsistency Evaluator for Abstractive Text Summarization
The performance of abstractive text summarization has been greatly boosted by
pre-trained language models recently. The main concern of existing abstractive
summarization methods is the factual inconsistency problem of their generated
summary. To alleviate the problem, many efforts have focused on developing
effective factuality evaluation metrics based on natural language inference and
question answering et al. However, they have limitations of high computational
complexity and relying on annotated data. Most recently, large language models
such as ChatGPT have shown strong ability in not only natural language
understanding but also natural language inference. In this paper, we study the
factual inconsistency evaluation ability of ChatGPT under the zero-shot setting
by evaluating it on the coarse-grained and fine-grained factuality evaluation
tasks including binary natural language inference (NLI), summary ranking, and
consistency rating. Experimental results show that ChatGPT outperforms previous
SOTA evaluation metrics on 6/9 datasets across three tasks, demonstrating its
great potential for assessing factual inconsistency in the zero-shot setting.
The results also highlight the importance of prompt design and the need for
future efforts to address ChatGPT's limitations on evaluation bias, wrong
reasoning, and hallucination.Comment: ongoing work, 12 pages, 4 figure
Approximate Inference for Determinantal Point Processes
In this thesis we explore a probabilistic model that is well-suited to a variety of subset selection tasks: the determinantal point process (DPP). DPPs were originally developed in the physics community to describe the repulsive interactions of fermions. More recently, they have been applied to machine learning problems such as search diversification and document summarization, which can be cast as subset selection tasks. A challenge, however, is scaling such DPP-based methods to the size of the datasets of interest to this community, and developing approximations for DPP inference tasks whose exact computation is prohibitively expensive.
A DPP defines a probability distribution over all subsets of a ground set of items. Consider the inference tasks common to probabilistic models, which include normalizing, marginalizing, conditioning, sampling, estimating the mode, and maximizing likelihood. For DPPs, exactly computing the quantities necessary for the first four of these tasks requires time cubic in the number of items or features of the items. In this thesis, we propose a means of making these four tasks tractable even in the realm where the number of items and the number of features is large. Specifically, we analyze the impact of randomly projecting the features down to a lower-dimensional space and show that the variational distance between the resulting DPP and the original is bounded. In addition to expanding the circumstances in which these first four tasks are tractable, we also tackle the other two tasks, the first of which is known to be NP-hard (with no PTAS) and the second of which is conjectured to be NP-hard. For mode estimation, we build on submodular maximization techniques to develop an algorithm with a multiplicative approximation guarantee. For likelihood maximization, we exploit the generative process associated with DPP sampling to derive an expectation-maximization (EM) algorithm. We experimentally verify the practicality of all the techniques that we develop, testing them on applications such as news and research summarization, political candidate comparison, and product recommendation
Translating Collocations for Bilingual Lexicons: A Statistical Approach
Collocations are notoriously difficult for non-native speakers to translate, primarily because they are opaque and cannot be translated on a word-by-word basis. We describe a program named Champollion which, given a pair of parallel corpora in two different languages and a list of collocations in one of them, automatically produces their translations. Our goal is to provide a tool for compiling bilingual lexical information above the word level in multiple languages, for different domains. The algorithm we use is based on statistical methods and produces p-word translations of n-word collocations in which n and p need not be the same. For example, Champollion translates make...decision, employment equity, and stock market into prendre...décision, équité en matière d'emploi, and bourse respectively. Testing Champollion on three years' worth of the Hansards corpus yielded the French translations of 300 collocations for each year, evaluated at 73% accuracy on average. In this paper, we describe the statistical measures used, the algorithm, and the implementation of Champollion, presenting our results and evaluation
Text Alignment Is An Efficient Unified Model for Massive NLP Tasks
Large language models (LLMs), typically designed as a function of next-word
prediction, have excelled across extensive NLP tasks. Despite the generality,
next-word prediction is often not an efficient formulation for many of the
tasks, demanding an extreme scale of model parameters (10s or 100s of billions)
and sometimes yielding suboptimal performance. In practice, it is often
desirable to build more efficient models -- despite being less versatile, they
still apply to a substantial subset of problems, delivering on par or even
superior performance with much smaller model sizes. In this paper, we propose
text alignment as an efficient unified model for a wide range of crucial tasks
involving text entailment, similarity, question answering (and answerability),
factual consistency, and so forth. Given a pair of texts, the model measures
the degree of alignment between their information. We instantiate an alignment
model (Align) through lightweight finetuning of RoBERTa (355M parameters) using
5.9M examples from 28 datasets. Despite its compact size, extensive experiments
show the model's efficiency and strong performance: (1) On over 20 datasets of
aforementioned diverse tasks, the model matches or surpasses FLAN-T5 models
that have around 2x or 10x more parameters; the single unified model also
outperforms task-specific models finetuned on individual datasets; (2) When
applied to evaluate factual consistency of language generation on 23 datasets,
our model improves over various baselines, including the much larger GPT-3.5
(ChatGPT) and sometimes even GPT-4; (3) The lightweight model can also serve as
an add-on component for LLMs such as GPT-3.5 in question answering tasks,
improving the average exact match (EM) score by 17.94 and F1 score by 15.05
through identifying unanswerable questions.Comment: NeurIPS 2023 Camera Ready, Code available at
https://github.com/yuh-zha/Alig
Towards Personalized and Human-in-the-Loop Document Summarization
The ubiquitous availability of computing devices and the widespread use of
the internet have generated a large amount of data continuously. Therefore, the
amount of available information on any given topic is far beyond humans'
processing capacity to properly process, causing what is known as information
overload. To efficiently cope with large amounts of information and generate
content with significant value to users, we require identifying, merging and
summarising information. Data summaries can help gather related information and
collect it into a shorter format that enables answering complicated questions,
gaining new insight and discovering conceptual boundaries.
This thesis focuses on three main challenges to alleviate information
overload using novel summarisation techniques. It further intends to facilitate
the analysis of documents to support personalised information extraction. This
thesis separates the research issues into four areas, covering (i) feature
engineering in document summarisation, (ii) traditional static and inflexible
summaries, (iii) traditional generic summarisation approaches, and (iv) the
need for reference summaries. We propose novel approaches to tackle these
challenges, by: i)enabling automatic intelligent feature engineering, ii)
enabling flexible and interactive summarisation, iii) utilising intelligent and
personalised summarisation approaches. The experimental results prove the
efficiency of the proposed approaches compared to other state-of-the-art
models. We further propose solutions to the information overload problem in
different domains through summarisation, covering network traffic data, health
data and business process data.Comment: PhD thesi
Topic Modeling for Analysing Similarity between Users in Twitter
La minerÃa de datos en redes sociales está ganando importancia debido a que permite realizar campañas de marketing más precisas. Por ejemplo, Google realiza un análisis de todos nuestros datos: vÃdeos que vemos, términos que buscamos, páginas webs a las que accedemos, aplicaciones que descargamos, etc. para conocernos mejor y mostrarnos publicidad personalizada.
LDA es un modelo estadÃstico generativo para modelar documentos. Existen diversos algoritmos que dado un conjunto de documentos permiten obtener un modelo LDA que podrÃa haber generado esos documentos. Con ese modelo es posible observar los temas usados en esos documentos y las palabras más relevantes para cada tema.
En el presente trabajo se pretende realizar una primera aproximación a la minerÃa de datos en Twitter. Para ello, usando la API de Twitter se han descargado tweets de diversos usuarios y de sus seguidores. Posteriormente se han procesado esos Tweets generando documentos y se ha aplicado la implementación de Gensim del algoritmo Online LDA para obtener los temas de los documentos. Posteriormente, se han comparado los temas de los usuarios con los de sus seguidores.
También se proporciona un análisis del estado del arte de la minerÃa de datos en Twitter
Politische Maschinen: Maschinelles Lernen für das Verständnis von sozialen Maschinen
This thesis investigates human-algorithm interactions in sociotechnological ecosystems. Specifically, it applies machine learning and statistical methods to uncover political dimensions of algorithmic influence in social media platforms and automated decision making systems. Based on the results, the study discusses the legal, political and ethical consequences of algorithmic implementations.Diese Arbeit untersucht Mensch-Algorithmen-Interaktionen in sozio-technologischen Ökosystemen. Sie wendet maschinelles Lernen und statistische Methoden an, um politische Dimensionen des algorithmischen Einflusses auf Socialen Medien und automatisierten Entscheidungssystemen aufzudecken. Aufgrund der Ergebnisse diskutiert die Studie die rechtlichen, politischen und ethischen Konsequenzen von algorithmischen Anwendungen
WiFi-Based Human Activity Recognition Using Attention-Based BiLSTM
Recently, significant efforts have been made to explore human activity recognition (HAR) techniques that use information gathered by existing indoor wireless infrastructures through WiFi signals without demanding the monitored subject to carry a dedicated device. The key intuition is that different activities introduce different multi-paths in WiFi signals and generate different patterns in the time series of channel state information (CSI). In this paper, we propose and evaluate a full pipeline for a CSI-based human activity recognition framework for 12 activities in three different spatial environments using two deep learning models: ABiLSTM and CNN-ABiLSTM. Evaluation experiments have demonstrated that the proposed models outperform state-of-the-art models. Also, the experiments show that the proposed models can be applied to other environments with different configurations, albeit with some caveats. The proposed ABiLSTM model achieves an overall accuracy of 94.03%, 91.96%, and 92.59% across the 3 target environments. While the proposed CNN-ABiLSTM model reaches an accuracy of 98.54%, 94.25% and 95.09% across those same environments