Gemmell, Carlos

Mackie, Iain

Owoicho, Paul Ogbonoko

Rossetto, Federico

We present GRILLBot, an open-source multi-modal task-oriented voice assistant to help users perform complex tasks, focusing on the domains of cooking and home improvement. GRILLBot curates and leverages web information extraction to build coverage over a broad range of tasks for which a user can receive guidance. To represent each task, we propose TaskGraphs as a dynamic graph unifying steps, requirements, and curated domain knowledge enabling contextual question answering, and detailed explanations. Multi-modal elements play a key role in GRILLBot both helping the user navigate through the task and enriching the experience with helpful videos and images that are automatically linked throughout the task. We leverage a contextual neural semantic parser to enable flexible navigation when interacting with the system by jointly encoding stateful information with the conversation history. GRILLBot enables dynamic and adaptable task planning and assistance for complex tasks by combining elements of task representations that incorporate text and structure, combined with neural models for search, question answering, and dialogue state management. GRILLBot competed in the Alexa prize TaskBot Challenge as one of the finalists

Fischer, Sophie

Dalton, Jeffrey

Enlighten

Proceedings of the SIGdial 2022 Conference, pages 654–658Heriot-Watt University, Edinburgh, UK. 07-09, September, 2022 ©2022 Association for Computational Linguistics654GRILLBot: An Assistant for Real-World Tasks with Neural SemanticParsing and Graph-Based RepresentationsCarlos Gemmell, Iain Mackie, Paul Owoicho, Federico Rossetto{c.gemmell.1, i.mackie.1, p.owoicho.1, f.rossetto.1}@research.gla.ac.ukSophie Fischer, Jeffrey Dalton{sophie.fischer, jeff.dalton}@glasgow.ac.ukUniversity of GlasgowGlasgow, Scotland, UKAbstractGRILLBot is the winning system in the 2022Alexa Prize TaskBot Challenge, moving to-wards the next generation of multimodal taskassistants. It is a voice assistant to guideusers through complex real-world tasks in thedomains of cooking and home improvement.These are long-running and complex tasks thatrequire flexible adjustment and adaptation. Thedemo highlights the core aspects, including anovel Neural Decision Parser for contextual-ized semantic parsing, a new “TaskGraph” staterepresentation that supports conditional execu-tion, knowledge-grounded chit-chat, and au-tomatic enrichment of tasks with images andvideos.1 IntroductionWe present GRILLBot, a task-oriented multi-modal conversational assistant developed duringthe 2021/2022 Alexa Prize TaskBot Challenge(Gemmell et al., 2022). GRILLBot aims to bean open research platform for complex tasks andsupports flexible graph-based task representations,contextual semantic parsing, and incorporates im-age and video content for clarity and instruction.We release the core components of the system asOAT1 (Open Assistant Toolkit).GRILLBot is still deployed throughout theUnited States with users able to invoke the botby issuing the command “Hey Alexa, Assist me”to their voice-only or screened Alexa device. Oursystem provides open-ended assistance focusing inthe domains of cooking and home improvement.It guides the user through all phases of the task,from performing preference elicitation to guiding auser to a relevant task from large task corpora, (i.e.“making a New York-style pizza" or "how to paint awall”) and then proceeds to assist in executing thetask in an engaging way. Its capabilities include1https://github.com/grill-lab/OATquestion answering, task-oriented chit-chat, andinstructional video content.GRILLBot is part of the first generation of assis-tants (Ipek et al.) that leverage screen-enabled con-versational devices for complex real-world tasks.These tasks are extensive, with some taking overan hour. As a result, a performant system requireslong-term state tracking with capabilities to adaptto a changing environment. It achieves this by in-troducing a new novel task structure, a TaskGraph,that captures the actions and information depen-dencies to guide the user through a complex task.TaskGraphs are enriched offline with content frominformation extraction, knowledge-based content,and multimedia images and videos.Traditional task-oriented dialogue systems(Young et al., 2013) take a slot-filling approach toderiving system actions. Academic datasets suchas MultiWOZ (Budzianowski et al., 2018) captureslot-value pairs from the user utterances within aconstrained set of domains enabling data-drivenneural models. Andreas et al. (2020) extend thistraditional representation towards semantic parsingwith dataflow graphs while constrained to the do-main of events booking in the SMCalFlow dataset.The neural decision parser in GRILLBot similarlygenerates code but focused on all aspects of a con-versation from navigation to task search and ques-tion answering. Other challenges such as DSTC11(Kottur et al., 2021) attempt this fully featured task-oriented experience, yet only do so in a virtual set-ting. GRILLBot stands apart as a system requiredto engage with real-world users in their environ-ment and assist in complex tasks for cooking andhome improvement.2 System overviewThe system uses a micro-service architecture witha centralized Orchestrator that defines the systembehavior. We use a phase-based policy to transition655Figure 1: Example TaskGraph (right) and conversation (left) connect each utterance with the information in thegraph nodes. The figure shows how we use conditional nodes (in light blue) to manage yes/no questions to unlockdifferent branches of instructions. Conditional nodes can also unlock autonomous actions like setting a timer. Thefigure also highlights how the requirement nodes (in light green) are used by the system to enrich the experienceby adding specific information that the user will need to perform the step. Finally, the purple box highlights extrainformation contained inside the step nodes to ground the QA system in domain knowledge.from searching for a suitable task (i.e. planningphase) to guiding users in performing a task (i.e.execution phase).The Orchestrator is the central process that di-rects and receives all the information from othermicroservices. Specifically, these child compo-nents are called functionalities and provide the nec-essary tools required by policies during conversa-tions. The main components inside functionalitiesare: Neural Decision Parser, Task Searcher andQuestion Answering. We discuss the Neural Deci-sion Parser in Section 4.The task searcher leverages our collection ofTaskGraphs to find candidate tasks. Our approachis based on a combination of traditional sparse &dense retrieval and neural re-ranking (Gemmellet al., 2022). The question answering system pro-vides extra task information, handles user ques-tions, and provides chit-chat elements. It uses acollection of QA systems across six categories.3 TaskGraphsA TaskGraph is a new graph-based representationbased on a directed acyclic graph that encodes theactions and information dependencies that the sys-tem needs to enable complex dialogue flows. In-formation is represented with heterogeneous nodes,each with a specific role:• Steps: Represent a task instruction for theuser, including visual information and textualdescriptions.• Requirements: Represent tools and ingredi-ents that are needed to perform the task andcan be grounded to specific steps.• Conditions: Represents yes/no gates that re-quire external information to resolve duringexecution dynamically.• Logic: Represents logical operations. Thesecan be used in conjunction with other nodes to656enable compact dependencies and smootherexecution flows. We currently support ∧, ∨and ¬ operations.• Actions: Represents actions taken by the sys-tem. This could be operations like setting atimer or adding items to a list.• Extra Information: Represents domain/task-specific knowledge like tips or fun facts thatcan enrich the user experience during the exe-cution of the task.Combining all these nodes, we can obtain adap-tive interactions where system-initiative allows thesystem to adapt to the user’s needs. Figure 3shows how TaskGraphs can be leveraged using atask-oriented conversation helping a user cooking“creamy zucchini pasta.”3.1 Offline TaskGraph CurationA key part of the system is providing relevant andhigh-quality TaskGraphs that satisfy the user’s taskgoal. For example, if a user asks, “I want to cook agluten-free meal based around lamb shoulder“, thesystem must find a suitable TaskGraph.To enable this, the system has to process rich andexecutable TaskGraphs offline with enough scaleto cover most user needs. Offline processing alsodecouples the heavy processing stages and dataenrichment from online processing.Web content We leverage domain expertsto identify high-quality seed websites for eachdomain, e.g. wholefoodsmarket.comand seriouseat.com for cooking andwikihow.com for home improvement. We useCommon Crawl to download the raw HTML fortarget domains and develop website-specific wrap-pers to extract semi-structured information abouteach task, i.e. title, author, description, ingredients,images, steps, ratings, videos, infoboxes, FAQs.Synthesize TaskGraphs The next stage of the of-fline process takes the semi-structured informationextracted and synthesizes executable TaskGraphs.This creates multi-modal task nodes and connec-tions from previously linear task steps. For ex-ample, we can create expressive graphs that con-tain a summary and a detailed description for eachtask step, which can be accessed by users whorequire additional context. We also leverage infor-mation extraction methods, such as noun phrasedetection (Honnibal and Montani, 2017), to cre-ate graph connections that link required ingredi-ents and tools to each step. Additionally, complexgraph structure and manual augmentation can beadded using a custom-developed excalidraw.comgraph interface. This allows loading automaticallyprocessed TaskGraphs, adding additional graphnodes and connections, and exporting the updatedTaskGraphs.Multimodal augmentation Visual informationplays a crucial role in improving the success and en-joyment of users being guided through real-worldtasks. For example, showing “How-to” videos,images and lists of tools and ingredients offers amore compelling and useful user experience. Fig-ure 2 depicts the multi-modal experience where thescreen text outlines the instruction, a list shows theingredients required, an image enriches the userexperience, and a video offers a technical demon-stration.Figure 2: Multimodal UI containing text, buttons, im-ages, and videos.We also develop a means of enriching task nodesif the task steps do not have aligned images andvideos. First, we extract actions (i.e "cut the beef")from a step based on a dependency parse of the steptext using the spaCy toolkit. For images, we useCLIP (Radford et al., 2021) to search over an im-age corpus of all other task steps images. This usesthe cosine similarity between image and step ac-tion embeddings to identify the relevant images foreach step. For videos, we develop a video corpusof domain-focused techniques, which is an indexbased on the video title using S-BERT (Reimersand Gurevych, 2019). Similar to image retrieval,we embed the step action as a query and rank thetitles of each “How-To” video through a cosinesimilarity of the step action embedding.6574 Neural Decision ParserFigure 3: Example of several possible in-context parsesduring task execution. The Neural Decision Parser au-toregressively generates the function call and argumentsas code in our DSL.TaskGraphs allows complex representations ofreal-world tasks. Due to the complex conversa-tional dialogue required, traditional non-contextualintent classifiers struggle to manage stateful tran-sitions. For this reason, we develop a Neural De-cision Parser that leverages both TaskGraphs anduser history for contextualized semantic parsing.Specifically, the model takes in natural languagerepresentations of a TaskGraph and prior conver-sational context to generate actions in the form ofthe custom GRILLBot Domain Specific Language(DSL). These generated arguments supply the tasksub-components with parsed knowledge relating tothe conversation.For example, if a user asks “Can you goback to the first step?”, the Neural DecisionParser would generate a parameterized parsestep_select(1).Contextual Semantic Parsing as a DSL Figure4 shows our state transition domain specific lan-guage (DSL) that captures all system actions. ThisDSL outlines a parameterized global command setthat is understood throughout the system to derivewhat actions or external APIs should be called, andwhat the response utterance should be. This flexi-ble navigation allows for a complex conversationdesign that leverages TaskGraphs.Model We use a single T5 large model (Raffelet al., 2020) to generate an agent action based onthe TaskGraph and conversational content. Usinga pre-trained language model allows advanced lan-guage capabilities to be leveraged across all systemparts, including coreference resolution, search pa-rameterization, setting timers, and state prediction.# User s p e c i f i e s which t a s k t o e x e c u t e> s e l e c t ( o p t i o n = I n t )# Catch a l l f o r u s e r q u e s t i o n s> a n s w e r _ q u e s t i o n ( )# Catch a l l f o r t a s k s e a r c h# Vague and Theme query c a t e g o r i e s> s e a r c h ( vague=Bool , theme= S t r i n g )# Go t o p r i o r node> p r e v i o u s ( )# Go t o n e x t s c h e d u l e d node> next ( )# N a v i g a t e t o s p e c i f i c t a s k s t e p s> s t e p _ s e l e c t ( s t e p = I n t )# S e t t i m e r wi th p a r s e d t ime span> t i m e r ( span = S t r i n g )# P r o v i d e d e t a i l s a b o u t a s t e p> c h i t _ c h a t ( )Figure 4: A sample of the Neural Decision Parser out-put DSL with intent-based functions and parameterizedarguments that a T5 model generates at inference time.We train the Neural Decision Parser by annotat-ing simulated conversations with the appropriatefunction calls and associated arguments. Our anno-tated training data comprises 1,200 turns across var-ious conversation stages and includes TaskGraphand conversational context. Through user studiesand system comparisons, we find that this approachachieves strong performance, and allows flexibletask navigation.5 ConclusionThis demo presents GRILLBot, a newly devel-oped Alexa Prize Taskbot system for complex real-world tasks with rich multimodal capability. Itdemonstrates multiple novel components includ-ing TaskGraphs to manage long complex tasks thatare automatically enriched with offline process toadd multimodal image and instructional video con-tent. It also shows key elements of the systemthat make it engaging, including its flexible NeuralDecision Parser that performs contextual semanticparsing as parametrized code generation. The re-sult is a demonstration of a new research platformdesigned from the ground-up around around flex-ible cloud micro-services and large-scale neurallanguage models.658ReferencesJacob Andreas, John Bufe, David Burkett, Charles Chen,Josh Clausman, Jean Crawford, Kate Crim, JordanDeLoach, Leah Dorner, Jason Eisner, et al. 2020.Task-oriented dialogue as dataflow synthesis. Trans-actions of the Association for Computational Linguis-tics, 8:556–571.Paweł Budzianowski, Tsung-Hsien Wen, Bo-HsiangTseng, Iñigo Casanueva, Stefan Ultes, Osman Ra-madan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the2018 Conference on EMNLP, pages 5016–5026.Carlos Gemmell, Sophie Fisher, Iain Mackie, PaulOwoicho, Federico Rossetto, and Jeffrey Dalton.2022. Grillbot: A flexible conversational agent forsolving complex real-world tasks. In 2022 AlexaPrize Proceedings.Matthew Honnibal and Ines Montani. 2017. spaCy 2:Natural language understanding with Bloom embed-dings, convolutional neural networks and incrementalparsing. To appear.Anna Gottardi Osman Ipek, Giuseppe Castellucci ShuiHu, Lavina Vaz Yao Lu, Anju Khatri, Anjali Chadha,Desheng Zhang, Sattvik Sahai, Prerna Dwivedi,Hangjie Shi, Lucy Hu, et al. Alexa, let’s work to-gether: Introducing the first alexa prize taskbot chal-lenge on conversational task assistance.Satwik Kottur, Seungwhan Moon, Alborz Geramifard,and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodalconversations. In Proceedings of the 2021 Confer-ence on Empirical Methods in Natural Language Pro-cessing, pages 4903–4912, Online and Punta Cana,Dominican Republic. Association for ComputationalLinguistics.Alec Radford, Jong Wook Kim, Chris Hallacy, AdityaRamesh, Gabriel Goh, Sandhini Agarwal, Girish Sas-try, Amanda Askell, Pamela Mishkin, Jack Clark,et al. 2021. Learning transferable visual models fromnatural language supervision. In ICML, pages 8748–8763. PMLR.Colin Raffel, Noam Shazeer, Adam Roberts, KatherineLee, Sharan Narang, Michael Matena, Yanqi Zhou,Wei Li, and Peter J Liu. 2020. Exploring the limitsof transfer learning with a unified text-to-text trans-former. Journal of ML Research, 21:1–67.Nils Reimers and Iryna Gurevych. 2019. Sentence-bert:Sentence embeddings using siamese bert-networks.In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing. Associa-tion for Computational Linguistics.Steve Young, Milica Gašić, Blaise Thomson, and Ja-son D Williams. 2013. Pomdp-based statistical spo-ken dialog systems: A review. Proceedings of theIEEE, 101(5):1160–1179.

GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representations

Abstract

Similar works

Full text

Available Versions

Enlighten