970 research outputs found
G2T: A simple but versatile framework for topic modeling based on pretrained language model and community detection
It has been reported that clustering-based topic models, which cluster
high-quality sentence embeddings with an appropriate word selection method, can
generate better topics than generative probabilistic topic models. However,
these approaches suffer from the inability to select appropriate parameters and
incomplete models that overlook the quantitative relation between words with
topics and topics with text. To solve these issues, we propose graph to topic
(G2T), a simple but effective framework for topic modelling. The framework is
composed of four modules. First, document representation is acquired using
pretrained language models. Second, a semantic graph is constructed according
to the similarity between document representations. Third, communities in
document semantic graphs are identified, and the relationship between topics
and documents is quantified accordingly. Fourth, the word--topic distribution
is computed based on a variant of TFIDF. Automatic evaluation suggests that G2T
achieved state-of-the-art performance on both English and Chinese documents
with different lengths. Human judgements demonstrate that G2T can produce
topics with better interpretability and coverage than baselines. In addition,
G2T can not only determine the topic number automatically but also give the
probabilistic distribution of words in topics and topics in documents. Finally,
G2T is publicly available, and the distillation experiments provide instruction
on how it works
Tokenization with Factorized Subword Encoding
In recent years, language models have become increasingly larger and more
complex. However, the input representations for these models continue to rely
on simple and greedy subword tokenization methods. In this paper, we propose a
novel tokenization method that factorizes subwords onto discrete triplets using
a VQ-VAE model. The effectiveness of the proposed tokenization method, referred
to as the Factorizer, is evaluated on language modeling and morpho-syntactic
tasks for 7 diverse languages. Results indicate that this method is more
appropriate and robust for morphological tasks than the commonly used byte-pair
encoding (BPE) tokenization algorithm.Comment: Findings of ACL 202
- …