Natural Language based Context Modeling and Reasoning with LLMs: A Tutorial

Large language models (LLMs) have become phenomenally surging, since 2018--two decades after introducing context-awareness into computing systems. Through taking into account the situations of ubiquitous devices, users and the societies, context-aware computing has enabled a wide spectrum of innovative applications, such as assisted living, location-based social network services and so on. To recognize contexts and make decisions for actions accordingly, various artificial intelligence technologies, such as Ontology and OWL, have been adopted as representations for context modeling and reasoning. Recently, with the rise of LLMs and their improved natural language understanding and reasoning capabilities, it has become feasible to model contexts using natural language and perform context reasoning by interacting with LLMs such as ChatGPT and GPT-4. In this tutorial, we demonstrate the use of texts, prompts, and autonomous agents (AutoAgents) that enable LLMs to perform context modeling and reasoning without requiring fine-tuning of the model. We organize and introduce works in the related field, and name this computing paradigm as the LLM-driven Context-aware Computing (LCaC). In the LCaC paradigm, users' requests, sensors reading data, and the command to actuators are supposed to be represented as texts. Given the text of users' request and sensor data, the AutoAgent models the context by prompting and sends to the LLM for context reasoning. LLM generates a plan of actions and responds to the AutoAgent, which later follows the action plan to foster context-awareness. To prove the concepts, we use two showcases--(1) operating a mobile z-arm in an apartment for assisted living, and (2) planning a trip and scheduling the itinerary in a context-aware and personalized manner.Comment: Under revie

Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies

Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. However, their efficacy is undermined by undesired and inconsistent behaviors, including hallucination, unfaithful reasoning, and toxic content. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. Techniques leveraging automated feedback -- either produced by the LLM itself or some external system -- are of particular interest as they are a promising way to make LLM-based solutions more practical and deployable with minimal human feedback. This paper presents a comprehensive review of this emerging class of techniques. We analyze and taxonomize a wide array of recent work utilizing these strategies, including training-time, generation-time, and post-hoc correction. We also summarize the major applications of this strategy and conclude by discussing future directions and challenges.Comment: Work in Progress. Version

자기회귀모델 기반 텍스트 생성을 위한 효과적인 학습 방법에 관한 연구

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2021.8. 김효석.The rise of deep neural networks has promoted tremendous advances in natural language processing research. Natural language generation is a subfield of natural language processing, which is inevitable in building a human-like artificial intelligence since they take responsibility for delivering the decision-making of machines in natural language. For neural network-based text generation techniques, which have achieved most state-of-the-art performance, autoregressive methods are generally adapted because of their correspondence to the word-by-word nature of human language production. In this dissertation, we investigate two different ways to train autoregressive text generation models, which are based on deep neural networks. We first focus on a token-level training of question generation, which aims to generate a question related to a given input passage. The proposed Answer-Separated Seq2Seq effectively mitigates a problem from the previous question generation models that a significant proportion of the generated questions include words in the target answer. While autoregressive methods are primarily trained with maximum likelihood estimation, they suffer from several problems, such as exposure bias. As a remedy, we propose a sequence-level GAN-based approach for text generation that promotes collaborative training in both continuous and discrete representations of text. To aggregate the achievement of the research mentioned above, we finally propose a novel way of training a sequence-level question generation model, adopting a pre-trained language model, one of the most significant breakthroughs in natural language processing, along with Proximal Policy Optimization.자연어 처리 연구는 딥 뉴럴넷의 도입으로 인해 대대적인 발전을 거쳤다. 자연어 처리 연구의 일종인 자연어 생성은 기계가 내린 결정을 사람이 이해할 수 있도록 전달하는 기능이 있다, 그렇기에 사람을 모방하는 인공지능 시스템을 구축하는 데에 있어 필수 불가결한 요소이다. 일반적으로 뉴럴넷 기반의 텍스트 생성 태스크에서는 자동회귀 방법론들이 주로 사용되는데, 이는 사람의 언어 생성 과정과 유사한 양상을 띠기 때문이다. 본 학위 논문에서는 두 가지 뉴럴넷 기반의 자동회귀 텍스트 생성 모델 학습 기법에 대해 제안한다. 첫 번째 방법론에서는 토큰 레벨에서의 질문 생성 모델 학습 방법에 대해 소개한다. 논문에서 제안하는 답변 분리 시퀀스-투-시퀀스 모델은 기존에 존재하는 질문 생성 모델로 생성된 질문이 답변에 해당하는 내용을 포함하는 문제점을 효과적으로 해결한다. 주로 최대 우도 추정법을 통해 학습되는 자동회귀 방법론에는 노출 편향 등과 같은 문제점이 존재한다. 이러한 문제점을 해결하기 위해 논문에서는 텍스트의 연속 공간 표현과 이산 공간 표현 모두에 대해 상호보완적으로 학습하는 시퀀스 레벨의 적대 신경망 기반의 텍스트 생성 기법을 제안한다. 마지막으로 앞선 방법론들을 종합하여 시퀀스 레벨의 질문 생성기법을 제안하며, 이러한 과정에서 최신 자연어 처리 방법 중 하나인 사전 학습 언어 모델과 근위 정책 최적화 방법을 이용한다.1 INTRODUCTION 1 1.1 Contributions 4 2 BACKGROUND 8 2.1 Sequence-to-Sequence model 8 2.1.1 Sequence-to-Sequence model with Attention Mechanism 8 2.2 Autoregressive text generation 11 2.2.1 Maximum Likelihood Training 11 2.2.2 Pros and cons of autoregressive methods 11 2.3 Non-autoregressive text generation 13 2.4 Transformers 13 2.5 Reinforcement Learning 16 2.5.1 Policy Gradient 17 3 TOKEN-LEVEL TRAINING OF CONDITIONAL TEXT GENERATION MODEL 19 3.1 Related Work 22 3.2 Task Definition 23 3.3 Base Model: Encoder-Decoder with Attention 23 3.4 Answer-Separated Seq2Seq 25 3.4.1 Encoder 27 3.4.2 Answer-Separated Decoder 28 3.5 Experimental Settings 30 3.5.1 Dataset 30 3.5.2 Implementation Details 30 3.5.3 Evaluation Methods 32 3.6 Results 32 3.6.1 Performance Comparison 32 3.6.2 Impact of Answer Separation 34 3.6.3 Question Generation for Machine Comprehension 36 3.7 Conclusion 38 4 SEQUENCE-LEVEL TRAINING OF UNCONDITIONAL TEXT GENERATION 40 4.1 Background 42 4.1.1 Generative Adversarial Networks 42 4.1.2 Continuous-space Methods 44 4.1.3 Discrete-space Methods 44 4.2 ConcreteGAN 45 4.2.1 Autoencoder Reconstruction 45 4.2.2 Adversarial Training in the Latent Code Space 47 4.2.3 Adversarial Training with Textual Outputs 48 4.3 Experiments 49 4.3.1 Dataset 50 4.3.2 Experimental Settings 50 4.3.3 Evaluation Metrics 51 4.3.4 Experimental Results for Quality & Diversity 52 4.3.5 Experimental Results for FD score 56 4.3.6 Human Evaluation 56 4.3.7 Analyses of Code Space 57 4.4 Conclusion 60 5 SEQUENCE-LEVEL TRAINING OF CONDITIONAL TEXT GENERATION 61 5.1 Introduction 61 5.2 Background 63 5.2.1 Pre-trained Language Model 63 5.2.2 Proximal Policy Optimization 70 5.3 Methods 72 5.3.1 Step One: Token-level Fine-tuning 72 5.3.2 Step Two: Sequence-level Fine-tuning with Question-specific Reward 72 5.4 Experiments 74 5.4.1 Implementation Details 75 5.4.2 Quantitative Analysis 76 5.4.3 Qualitative Analysis 76 5.5 Conclusion 78 6 CONCLUSION 80 7 APPENDIX* 82 7.1 Generated Samples 82 7.2 Comparison of ARAE and ARAE* 84 7.3 Human Evaluation Criteria 85박

Leveraging Feedback in Conversational Question Answering Systems

172 p.Tesi honen helburua martxan jarri eta geroko sistemek gizakiekin duten elkarregina erabiltzeada, gizakien feedbacka sistementzako ikasketa eta egokitzapen seinale bezala erabiliz.Elkarrizketa sistemek martxan jartzerakoan jasaten duten domeinu aldaketan jartzen dugufokua. Helburu honetarako, feedback bitar esplizituaren kasua aztertzen dugu, hau baitagizakientzat feedbacka emateko seinale errazena.Sistemak martxan jarri eta gero hobetzeko, lehenik eta behin DoQA izeneko galdera-erantzunmotako elkarriketez osatutako datu multzo bat eraiki dugu. Datu multzo honekcrowdsourcing bidez jasotako 2.437 dialogo ditu. Aurreko lanekin konparatuz gero, DoQAkbenetazko informazio beharrak islatzen ditu, datu multzo barneko elkarrizketak naturalagoaketa koherenteagoak izanik. Datu multzo sortu eta gero, feedback-weighted learning (FWL)izeneko algoritmo bat diseinatu dugu, feedback bitarra bakarrik erabiliz aurretikentrenatutako sistema gainbegiratu bat hobetzeko gai dena. Azkenik, algoritmo honen mugakaztertzen ditugu jasotako feedbacka zaratatsua den kasuetarako eta FWL moldatzen dugueszenatoki zaratsuari aurre egiteko. Kasu honetan lortzen ditugun emaitza negatiboakerakusten dute erabiltzaileetatik jasotako feedback zaratsua modelatzearen erronka, hauebaztea oraindik ikerkuntza galdera ireki bat delarik

An In-depth Investigation of User Response Simulation for Conversational Search

Conversational search has seen increased recent attention in both the IR and NLP communities. It seeks to clarify and solve a user's search need through multi-turn natural language interactions. However, most existing systems are trained and demonstrated with recorded or artificial conversation logs. Eventually, conversational search systems should be trained, evaluated, and deployed in an open-ended setting with unseen conversation trajectories. A key challenge is that training and evaluating such systems both require a human-in-the-loop, which is expensive and does not scale. One strategy for this is to simulate users, thereby reducing the scaling costs. However, current user simulators are either limited to only respond to yes-no questions from the conversational search system, or unable to produce high quality responses in general. In this paper, we show that current state-of-the-art user simulation system could be significantly improved by replacing it with a smaller but advanced natural language generation model. But rather than merely reporting this new state-of-the-art, we present an in-depth investigation of the task of simulating user response for conversational search. Our goal is to supplement existing works with an insightful hand-analysis of what challenges are still unsolved by the advanced model, as well as to propose our solutions for them. The challenges we identified include (1) dataset noise, (2) a blind spot that is difficult for existing models to learn, and (3) a specific type of misevaluation in the standard empirical setup. Except for the dataset noise issue, we propose solutions to cover the training blind spot and to avoid the misevaluation. Our proposed solutions lead to further improvements. Our best system improves the previous state-of-the-art significantly.Comment: 9 page