Guidelines for annotating the LUNA corpus with frame information

This document defines the annotation workflow aimed at adding frame information to the LUNA corpus of conversational speech. In particular, it details both the corpus pre-processing steps and the proper annotation process, giving hints about how to choose the frame and the frame element labels. Besides, the description of 20 new domain-specific and language-specific frames is reported. To our knowledge, this is the first attempt to adapt the frame paradigm to dialogs and at the same time to define new frames and frame elements for the specific domain of software/hardware assistance. The technical report is structured as follows: in Section 2 an overview of the FrameNet project is given, while Section 3 introduces the LUNA project and the annotation framework involving the Italian dialogs. Section 4 details the annotation workflow, including the format preparation of the dialog files and the annotation strategy. In Section 5 we discuss the main issues of the annotation of frame information in dialogs and we describe how the standard annotation procedure was changed in order to face such issues. Then, the 20 newly introduced frames are reported in Section 6

Artificial Cognition for Social Human-Robot Interaction: An Implementation

© 2017 The Authors Human–Robot Interaction challenges Artificial Intelligence in many regards: dynamic, partially unknown environments that were not originally designed for robots; a broad variety of situations with rich semantics to understand and interpret; physical interactions with humans that requires fine, low-latency yet socially acceptable control strategies; natural and multi-modal communication which mandates common-sense knowledge and the representation of possibly divergent mental models. This article is an attempt to characterise these challenges and to exhibit a set of key decisional issues that need to be addressed for a cognitive robot to successfully share space and tasks with a human. We identify first the needed individual and collaborative cognitive skills: geometric reasoning and situation assessment based on perspective-taking and affordance analysis; acquisition and representation of knowledge models for multiple agents (humans and robots, with their specificities); situated, natural and multi-modal dialogue; human-aware task planning; human–robot joint task achievement. The article discusses each of these abilities, presents working implementations, and shows how they combine in a coherent and original deliberative architecture for human–robot interaction. Supported by experimental results, we eventually show how explicit knowledge management, both symbolic and geometric, proves to be instrumental to richer and more natural human–robot interactions by pushing for pervasive, human-level semantics within the robot's deliberative system

Continually improving grounded natural language understanding through human-robot dialog

As robots become ubiquitous in homes and workplaces such as hospitals and factories, they must be able to communicate with humans. Several kinds of knowledge are required to understand and respond to a human's natural language commands and questions. If a person requests an assistant robot to take me to Alice's office, the robot must know that Alice is a person who owns some unique office, and that take me means it should navigate there. Similarly, if a person requests bring me the heavy, green mug, the robot must have accurate mental models of the physical concepts heavy, green, and mug. To avoid forcing humans to use key phrases or words robots already know, this thesis focuses on helping robots understanding new language constructs through interactions with humans and with the world around them. To understand a command in natural language, a robot must first convert that command to an internal representation that it can reason with. Semantic parsing is a method for performing this conversion, and the target representation is often semantic forms represented as predicate logic with lambda calculus. Traditional semantic parsing relies on hand-crafted resources from a human expert: an ontology of concepts, a lexicon connecting language to those concepts, and training examples of language with abstract meanings. One thrust of this thesis is to perform semantic parsing with sparse initial data. We use the conversations between a robot and human users to induce pairs of natural language utterances with the target semantic forms a robot discovers through its questions, reducing the annotation effort of creating training examples for parsing. We use this data to build more dialog-capable robots in new domains with much less expert human effort (Thomason et al., 2015; Padmakumar et al., 2017). Meanings of many language concepts are bound to the physical world. Understanding object properties and categories, such as heavy, green, and mug requires interacting with and perceiving the physical world. Embodied robots can use manipulation capabilities, such as pushing, picking up, and dropping objects to gather sensory data about them. This data can be used to understand non-visual concepts like heavy and empty (e.g. get the empty carton of milk from the fridge), and assist with concepts that have both visual and non-visual expression (e.g. tall things look big and also exert force sooner than short things when pressed down on). A second thrust of this thesis focuses on strategies for learning these concepts using multi-modal sensory information. We use human-in-the-loop learning to get labels between concept words and actual objects in the environment (Thomason et al., 2016, 2017). We also explore ways to tease out polysemy and synonymy in concept words (Thomason and Mooney, 2017) such as light, which can refer to a weight or a color, the latter sense being synonymous with pale. Additionally, pushing, picking up, and dropping objects to gather sensory information is prohibitively time-consuming, so we investigate strategies for using linguistic information and human input to expedite exploration when learning a new concept (Thomason et al., 2018). Finally, we build an integrated agent with both parsing and perception capabilities that learns from conversations with users to improve both components over time. We demonstrate that parser learning from conversations (Thomason et al., 2015) can be combined with multi-modal perception (Thomason et al., 2016) using predicate-object labels gathered through opportunistic active learning (Thomason et al., 2017) during those conversations to improve performance for understanding natural language commands from humans. Human users also qualitatively rate this integrated learning agent as more usable after it has improved from conversation-based learning.Computer Science

Improving Grounded Natural Language Understanding through Human-Robot Dialog

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world

Identifying power relationships in dialogues

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Cataloged from student submitted PDF version of thesis.Includes bibliographical references (p. 175-179).Understanding power relationships is an important step towards building computers that can understand human social relationships. Power relationships can arise due to dierences in the roles of the speakers, as between bosses and employees. Power can also affect the manner of communication between social equals, as between friends or acquaintances. There are numerous potential uses for an automatic system that can understand power relationships. These include: the analysis of the organizational structure of formal and ad-hoc groups, the profiling of in influential individuals within a group, or identifying aggressive or power-inappropriate language in email or other Internet media. In this thesis, we explore the problem of engineering eective power identication systems. We show methods for constructing an eective ground truth corpus for analyzing power. We focus on three areas of modeling that help in improving the prediction of power relationships. 1) Utterance Level Language Cues - patterns of language use can help distinguish the speech of leaders or followers. We show a set of eective syntactic/semantic features that best capture these linguistic manifestations of power. 2) Dialog Level Interactions - the manner of interaction between speakers can inform us about the underlying power dynamics. We use Hidden Markov Models to organize and model the information from these interaction-based cues. 3) Social conventions - speaker behavior is in influenced by their background knowledge, in particular, conventional rules of communication. We use a generative hierarchical Bayesian framework to model dialogs as mental processes; then we extend these models to include components that encode basic social conventions such as politeness. We apply our integrated system, PRISM, on the Nixon Watergate Transcripts, to demonstrate that our system can perform robustly on real world data.by Yuan Kui Shen.Ph.D

On power in the workplace

Diese Arbeit beschäftigt sich mit dem Thema „Macht(ausübung) am Arbeitsplatz“. Das Forschungsgebiet ist hierfür in zwei Abschnitte aufgeteilt. Der Erste konzentriert sich auf Sprechakte der Kontrolle, wobei das Hauptaugenmerk auf Direktiven, Aufforderungen und Ratschlägen gelegt ist. Der zweite Teil ist ausschließlich der Gender-Studie gewidmet. Die Analyse basiert auf Dialogen, die aus den beiden amerikanischen Filmen „Selbst ist die Braut“ und „Erin Brokovich“ stammen. Das Ergebnis zeigt, dass 181 Kontrollsprechakte erkannt wurden, welche in 120 Direktiven, 38 Aufforderungen, 7 Ratschläge und 16 Grenzfälle aufgegliedert werden konnten. Unter anderem beinhaltet die Untersuchung auch die verschiedenen Subtypen der Kontrollsprechakte, wie auch deren Kategorisierung anhand ihrer Form. Diese Form ist die offensichtlichste Art um Direktiven, Aufforderungen und Ratschläge im Alltag auszudrücken (vgl. Vine 2004: 40), was auch das Ergebnis der Forschung bestätigt. 60% der identifizierten Kontrollsprechakte konnten als Imperative ausgemacht werden. In der Untersuchung wurde hauptsächlich eine formale Analyse als die Basis für die Kategorisierung benutzt. Es ist jedoch nicht immer möglich, sich nur auf die Kategorisierung durch die Form zu verlassen, da dies nicht ausreichend für Daten wäre. Deshalb wurde die Unterscheidung zwischen den Kontrollakte zusätzlich auch anhand der Eigenschaften des Inhalts gemacht. Folglich, beinhaltet die Analyse auch die Klassifizierung verschiedener Subtypen von Kontrollakte, wobei die interessantesten Instanzen herausgegriffen und detailliert beschrieben werden. Die Gender-Studie ist in zwei Bereiche gliedert. Erstens wird versucht Höflichkeitsstrategien zwischen beiden Geschlechtern herauszufiltern. Diese wurden in den Komplimenten des Datensatzes identifiziert. Interessanterweise, konnte eine Korrelation zwischen dem Akt des Komplimente machens und der Machtdemonstration am Arbeitsplatz festgestellt werden. Auf Basis der Analyse der Kontrollakte, welche mit Machtausübung assoziiert werden, konnte eine die Geschlechtsuntersuchung eingeleitet werden. Auf Grundlage der Theorie “powerful versus powerless language” (O’Barr and Atkins 1988) wird veranschaulicht, dass der Sprachstil nicht geschlechterabhängig ist.The focus of this research paper is on power in the workplace. The research area is divided into two sections. The first section deals with the speech acts of control: directives, request, and advice, by formally focusing on their head acts. The second section focuses exclusively on the gender-related study. The analysis is based on dialog stemming from two American films: The Proposal and Erin Brockovich. There are 181 control speech acts identified: 120 directives, 38 requests, 7 pieces of advice, and 16 borderline cases. The different sub-types of the control head acts, as well as their categorization according to form has been provided. The control speech acts identified in the dataset in most cases take the form of the imperative, which is the most obvious and recognizable way of issuing directives, requests or advice in a situation (cf. Vine 2004: 40). There are 109 instances (60%) of imperatives. In the analysis the form is frequently used as the basis for categorizing the control acts. However, it is not always possible to rely on a purely formal categorization system as it does not adequately account for the control act head acts data. Therefore, a distinction between the head acts has also been drawn according to the features of the content. Consequently, the classification of different sub-types of control acts is offered, and the most interesting instances are discussed in detail. The focus in the gender study is on two different aspects of language. The first traces out and discerns politeness strategies amongst the two genders. The politeness strategies are identified in the examples of compliments. There are 7 instances of complementing in the dataset. The speech acts of complimenting are selected due to an observation that there is a remarkable correlation between complimenting and power in the workplace. Due to the study on the control speech acts, which are closely associated with power exercise and power management, the second gender-related study is conducted. It is based on the “powerful versus powerless language” theory (O’Barr and Atkins 1998) and supports the view: saying that the way people talk is independent of their gender

A web-based graphical user interface to display spatial data

This dissertation presents the design and implementation of a graphical user interface (GUI) to display spatial data in a web-based environment. The work is a case study for a web-based framework for distributed applications, the Web Computing Skeleton, using a distributed open spatial query mechanism to display the geographic data. The design is based on investigation of geographic information systems (GISs), GUI design and properties of spatial query mechanisms. The purpose ofthe GUI is to integrate information about a geographic area; display, manipulate and query geographic-based spatial data; execute queries about spatial relationships and analyse the attribute data to calculate the shortest routes for emergency response. The GUI is implemented as a Java applet embedded in a web document that communicates with the application server via generic GIS classes that provide a common interface to various GIS data sources used in the spatial query mechanism to access a geographic database. Features that are supported by the distributed open spatial query mechanism include a basic set of spatial selection criteria, spatial selection based on pointing, specification of a query window, description of a map scale and identification of a map legend. The design is based on a formal design process that includes the selection of a conceptual model, identification of task flow, major windows and dialog flow, the definition of fields and detailed window layout and finally the definition of field constraints and defaults. The conceptual model characterises the application and provides a framework for users to learn the system model. This model is conceptualised as a map that the user manipulates directly. Unlike a typical map, which just shows spatial data such as roads, cities, and country borders, the GIS links attribute data like population statistics to the spatial data. This link between the map data and the attribute data makes the GIS a powerful tool to manipulate and display data. To measure the performance of displaying spatial data, two main factors are considered, namely processing speed and display quality. Factors that affect the processing speed include the rate of data transfer from the generic GIS classes, the rate data is downloaded over the network and the speed of execution of the drawing. Two factors that influence the spatial data display quality are pixel distance and bitmap quality. The pixel distance set in the geographic database is represented by two pixels on the display screen, which affects the display quality since the pixel distance is the upper limit for display granularity. This means that setting the pixel distance is a trade-off between the processing speed and the display quality. Bitmaps are raster images that are made up of pixels or cells. To improve the raster image quality, the bitmap resolution can be adjusted to display more pixels per centimetre.Dissertation (MSc (Computer Science))--University of Pretoria, 2007.Computer Scienceunrestricte

From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction.

Intelligent collaborative agents are becoming common in the human society. From virtual assistants such as Siri and Google Now to assistive robots, they contribute to human activities in a variety of ways. As they become more pervasive, the challenge of customizing them to a variety of environments and tasks becomes critical. It is infeasible for engineers to program them for each individual use. Our research aims at building interactive robots and agents that adapt to new environments autonomously by interacting with human users using natural modalities. This dissertation studies the problem of learning novel tasks from human-agent dialog. We propose a novel approach for interactive task learning, situated interactive instruction (SII), and investigate approaches to three computational challenges that arise in designing SII agents: situated comprehension, mixed-initiative interaction, and interactive task learning. We propose a novel mixed-modality grounded representation for task verbs which encompasses their lexical, semantic, and task-oriented aspects. This representation is useful in situated comprehension and can be learned through human-agent interactions. We introduce the Indexical Model of comprehension that can exploit extra-linguistic contexts for resolving semantic ambiguities in situated comprehension of task commands. The Indexical model is integrated with a mixed-initiative interaction model that facilitates a flexible task-oriented human-agent dialog. This dialog serves as the basis of interactive task learning. We propose an interactive variation of explanation-based learning that can acquire the proposed representation. We demonstrate that our learning paradigm is efficient, can transfer knowledge between structurally similar tasks, integrates agent-driven exploration with instructional learning, and can acquire several tasks. The methods proposed in this thesis are integrated in Rosie - a generally instructable agent developed in the Soar cognitive architecture and embodied on a table-top robot.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/111573/1/shiwali_1.pd