A Primer on Seq2Seq Models for Generative Chatbots

The recent spread of Deep Learning-based solutions for Artificial Intelligence and the development of Large Language Models has pushed forwards significantly the Natural Language Processing area. The approach has quickly evolved in the last ten years, deeply affecting NLP, from low-level text pre-processing tasks –such as tokenisation or POS tagging– to high-level, complex NLP applications like machine translation and chatbots. This paper examines recent trends in the development of open-domain data-driven generative chatbots, focusing on the Seq2Seq architectures. Such architectures are compatible with multiple learning approaches, ranging from supervised to reinforcement and, in the last years, allowed to realise very engaging open-domain chatbots. Not only do these architectures allow to directly output the next turn in a conversation but, to some extent, they also allow to control the style or content of the response. To offer a complete view on the subject, we examine possible architecture implementations as well as training and evaluation approaches. Additionally, we provide information about the openly available corpora to train and evaluate such models and about the current and past chatbot competitions. Finally, we present some insights on possible future directions, given the current research status

Visually Grounded Language Learning: a review of language games, datasets, tasks, and models

In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by ``listening to the radio" only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein's idea of `language games' to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.Comment: Preprint for JAIR before copyeditin

Speaking The Subject: A Discourse Analysis Of Undergraduate Seminar Practice

This dissertation explores talk in an undergraduate seminar context. Research design was informed by an interpretive, ethnomethodological approach to understanding talk as a situated activity. A series of student-led seminars were audio recorded; students and staff were interviewed and post-seminar group debriefing sessions were held. The data was subsequently transcribed and analysed using a functional systemic linguistics and discourse analysis approach. Analysis identified structural and linguistic elements of seminar talk and links between language, identity, power and status was explored through an analysis of the discursive processes at work in the seminar events.An heuristic model of the seminar as a socio-pedagogic space, a site of hegemonic struggle, was used to aid concept development. A number of issues emerged within an interpretative framework of the cognitive, interpersonal and textual elements of seminar talk. In the analysis of the textual meta-function of seminars, how complexity is achieved and how conversational moves are patterned, seminars appear to constitute a hybrid talk variety, a highly unusual textual form in which participants need to learn how to participate. Tensions were found between the social and the cognitive elements of seminars. Student participants tend to use the seminar to achieve social effects, identifying and maintaining interpersonal relationships. The collaborative discourse strategies they employ constrain other opportunities for achieving educational outcomes. The learning which does take place is more likely to be related to personal and skills development than to learning about the academic subject. Students deployed a range of heteroglossic discursive strategies to practice their skills in forming ideas, marshalling evidence and constructing argument. The discursive practices of seminar events foreground tensions between socially situated identities. The research identifies a number of areas for improving practice including: enhanced specification of seminar processes and outcomes; embedding opportunities for preparation and critical reflection; teaching the subject of communication and foregrounding understandings of the discursive practices at work in seminars so as to empower individual learners