105,008 research outputs found
Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio
Despite rapid advancement in the field of Constrained Natural Language
Generation, little time has been spent on exploring the potential of language
models which have had their vocabularies lexically, semantically, and/or
phonetically constrained. We find that most language models generate compelling
text even under significant constraints. We present a simple and universally
applicable technique for modifying the output of a language model by
compositionally applying filter functions to the language models vocabulary
before a unit of text is generated. This approach is plug-and-play and requires
no modification to the model. To showcase the value of this technique, we
present an easy to use AI writing assistant called Constrained Text Generation
Studio (CTGS). CTGS allows users to generate or choose from text with any
combination of a wide variety of constraints, such as banning a particular
letter, forcing the generated words to have a certain number of syllables,
and/or forcing the words to be partial anagrams of another word. We introduce a
novel dataset of prose that omits the letter e. We show that our method results
in strictly superior performance compared to fine-tuning alone on this dataset.
We also present a Huggingface space web-app presenting this technique called
Gadsby. The code is available to the public here:
https://github.com/Hellisotherpeople/Constrained-Text-Generation-StudioComment: Published in the proceedings of the 2nd Workshop on When Creative AI
Meets Conversational AI (CAI2), COLING 2022, 6 pages, System Demonstration
Pape
Computational and Robotic Models of Early Language Development: A Review
We review computational and robotics models of early language learning and
development. We first explain why and how these models are used to understand
better how children learn language. We argue that they provide concrete
theories of language learning as a complex dynamic system, complementing
traditional methods in psychology and linguistics. We review different modeling
formalisms, grounded in techniques from machine learning and artificial
intelligence such as Bayesian and neural network approaches. We then discuss
their role in understanding several key mechanisms of language development:
cross-situational statistical learning, embodiment, situated social
interaction, intrinsically motivated learning, and cultural evolution. We
conclude by discussing future challenges for research, including modeling of
large-scale empirical data about language acquisition in real-world
environments.
Keywords: Early language learning, Computational and robotic models, machine
learning, development, embodiment, social interaction, intrinsic motivation,
self-organization, dynamical systems, complexity.Comment: to appear in International Handbook on Language Development, ed. J.
Horst and J. von Koss Torkildsen, Routledg
Topic-based mixture language modelling
This paper describes an approach for constructing a mixture of language models based on simple statistical notions of semantics using probabilistic models developed for information retrieval. The approach encapsulates corpus-derived semantic information and is able to model varying styles of text. Using such information, the corpus texts are clustered in an unsupervised manner and a mixture of topic-specific language models is automatically created. The principal contribution of this work is to characterise the document space resulting from information retrieval techniques and to demonstrate the approach for mixture language modelling.
A comparison is made between manual and automatic clustering in order to elucidate how the global content information is expressed in the space. We also compare (in terms of association with manual clustering and language modelling accuracy) alternative term-weighting schemes and the effect of singular value decomposition dimension reduction (latent semantic analysis). Test set perplexity results using the British National Corpus indicate that the approach can improve the potential of statistical language modelling. Using an adaptive procedure, the conventional model may be tuned to track text data with a slight increase in computational cost
- …