MULTI-MODAL TASK INSTRUCTIONS TO ROBOTS BY NAIVE USERS

This thesis presents a theoretical framework for the design of user-programmable robots. The objective of the work is to investigate multi-modal unconstrained natural instructions given to robots in order to design a learning robot. A corpus-centred approach is used to design an agent that can reason, learn and interact with a human in a natural unconstrained way. The corpus-centred design approach is formalised and developed in detail. It requires the developer to record a human during interaction and analyse the recordings to find instruction primitives. These are then implemented into a robot. The focus of this work has been on how to combine speech and gesture using rules extracted from the analysis of a corpus. A multi-modal integration algorithm is presented, that can use timing and semantics to group, match and unify gesture and language. The algorithm always achieves correct pairings on a corpus and initiates questions to the user in ambiguous cases or missing information. The domain of card games has been investigated, because of its variety of games which are rich in rules and contain sequences. A further focus of the work is on the translation of rule-based instructions. Most multi-modal interfaces to date have only considered sequential instructions. The combination of frame-based reasoning, a knowledge base organised as an ontology and a problem solver engine is used to store these rules. The understanding of rule instructions, which contain conditional and imaginary situations require an agent with complex reasoning capabilities. A test system of the agent implementation is also described. Tests to confirm the implementation by playing back the corpus are presented. Furthermore, deployment test results with the implemented agent and human subjects are presented and discussed. The tests showed that the rate of errors that are due to the sentences not being defined in the grammar does not decrease by an acceptable rate when new grammar is introduced. This was particularly the case for complex verbal rule instructions which have a large variety of being expressed

Temporality and modality in entailment graph induction

The ability to draw inferences is core to semantics and the field of Natural Language Processing. Answering a seemingly simple question like ‘Did Arsenal play Manchester yesterday’ from textual evidence that says ‘Arsenal won against Manchester yesterday’ requires modeling the inference that ‘winning’ entails ‘playing’. One way of modeling this type of lexical semantics is with Entailment Graphs, collections of meaning postulates that can be learned in an unsupervised way from large text corpora. In this work, we explore the role that temporality and linguistic modality can play in inducing Entailment Graphs. We identify inferences that were previously not supported by Entailment Graphs (such as that ‘visiting’ entails an ‘arrival’ before the visit) and inferences that were likely to be learned incorrectly (such as that ‘winning’ entails ‘losing’). Temporality is shown to be useful in alleviating these challenges, in the Entailment Graph representation as well as the learning algorithm. An exploration of linguistic modality in the training data shows, counterintuitively, that there is valuable signal in modalized predications. We develop three datasets for evaluating a system’s capability of modeling these inferences, which were previously underrepresented in entailment rule evaluations. Finally, in support of the work on modality, we release a relation extraction system that is capable of annotating linguistic modality, together with a comprehensive modality lexicon

Proceedings of the 1st Doctoral Consortium at the European Conference on Artificial Intelligence (DC-ECAI 2020)

1st Doctoral Consortium at the European Conference on Artificial Intelligence (DC-ECAI 2020), 29-30 August, 2020 Santiago de Compostela, SpainThe DC-ECAI 2020 provides a unique opportunity for PhD students, who are close to finishing their doctorate research, to interact with experienced researchers in the field. Senior members of the community are assigned as mentors for each group of students based on the student’s research or similarity of research interests. The DC-ECAI 2020, which is held virtually this year, allows students from all over the world to present their research and discuss their ongoing research and career plans with their mentor, to do networking with other participants, and to receive training and mentoring about career planning and career option

Proceedings of the 2010 Annual Conference of the Gesellschaft für Semantik

Sinn & Bedeutung - the annual conference of the Gesellschaft für Semantik - aims to bring together both established researchers and new blood working on current issues in natural language semantics, pragmatics, the syntax-semantics interface, the philosophy of language or carrying out psycholinguistic studies related to meaning. Every year, the conference moves to a different location in Europe. The 2010 conference - Sinn & Bedeutung 15 - took place on September 9 - 11 at Saarland University, Saarbrücken, organized by the Department for German Studies

Meaning and Explanation.

My dissertation investigates the semantic contribution of the individual words ‘why’ and ‘because', attempting to get clear on whether and how some of our central explanatory terminology gets disambiguated, and thereby to make some progress on a theory of ‘why’-questions that can tell us something substantive about explanation. I argue that ‘why’ and ‘because’ have literal causal senses, as well as distinct senses that we use to communicate metaphysical explanations. I show that apparent further semantic variations in the meaning of ‘because’ in its so-called epistemic and metalinguistic uses are illusory, and give a full explanation of those variations in terms of syntactic ambiguities. Finally, I argue that a causal metaphor unifies the senses of 'why' and 'because' at issue in metaphysical explanations with their literal causal senses. What this semantic investigation turns out to offer us, I argue, is a new understanding of the centrality of causal explanation to explanation in general.PHDPhilosophyUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/108881/1/jshaheen_1.pd

Modal auxiliary verbs and contexts

Modal auxiliary verbs, such as `could', `might', `must', `would', and others, have different readings depending on the context in which they occur (Kratzer 1981). The sentence `Jess could fry the fish' can mean that, in a time previous to the utterance of the sentence, Jess had the ability to fry the fish, or it can mean that, at the time of the utterance, Jess frying the fish is a possible event. Modal auxiliary verbs often create intensional environments, leading the events described by the second verb to be understood to be non-actual events. When the readings are described as being determined by a context, it is often a broad notion of non-linguistic and extra-sentential linguistic context that is the focus of the interpretation. For example, descriptive pragmatic constraints are used in Lewis 1973 and Kratzer 1981 to characterize types of accessibility relations and types of orderings of worlds. A large part of the meaning of modal auxiliary verbs, however, centers around how the events described by the second verb are situated relative to the time at which the sentence containing the modal auxiliary is used. Information about the temporal situation of an event is conveyed through the linguistic context in which a modal auxiliary verb occurs, including, but not limited to, lexical properties of the linguistic expressions describing the event in the scope of the modal auxiliary, lexical properties of the modal auxiliary itself, and temporal and aspectual marking on linguistic expressions in the verbal projections. In order to provide a framework for representing the interactions of tense, aspect, and modality, a fragment of English is given in a Multi-Modal Combinatorial Categorial Grammar (Baldridge & Kruijff, Steedman 2012). Modal auxiliaries are given verb-like lexical entries in the grammar using lexical entries that combine features from Villavicenio 2002 and standard attribute value matrices of Head Driven Phrase Structure Grammar (Pollard & Sag 1999, Sag, Wasow, & Bender 2003). Modal auxiliaries have default lexical arguments with which they combine, and they combine with temporal and aspectual meaning that is sometimes morphologically manifested through grammatical tense and aspect. Portions of the combinatory methods are based on Bach 1983, who argued for less constrained combinatorial rules and unification of features in order to represent modal auxiliaries. The notion of event semantics (Davidson 1967) plays an important role in the formulation of the compositional semantics due to the way in which event times are related to aspectual meaning. The grammar uses a Neo-Davidsonian approach (Parsons 1990) to representing the arguments of the verb and builds on the work of Champollion 2015. The temporal component is very important in this work and uses portions of the temporal and event ontology proposed in Muskens 1995, 2003. Two paradigms of modal auxiliaries are proposed: Tense-bearing modal auxiliaries and non-tense-bearing modal auxiliaries. Within each paradigm, readings are shown to have differing semantics with respect to the semantic roles with which they combine and the temporal and aspectual readings that they can have. Differing results with respect to their behaviour in describing various states of affairs are addressed as is their behaviour in expressing past tense, sequence of tense contexts (Abusch 1997), and the distribution of perfect aspect. The formal grammar distinguishes parts of the meaning of sentences with modal auxiliary verbs that can be represented in terms of composition of temporal and aspectual expressions with modal auxiliary verbs or composition of a modal auxiliary verb with its arguments on one hand from parts of the meaning that are constrained by a broader notion of context, on the other hand. The notion of a broader context is not, however, neglected in the treatment. The English language fragment presented in the grammar is interpreted in a relativist semantic model, motivated by the assessment-sensitivity of epistemic modal auxiliaries (MacFarlane 2011, Lasersohn 2005, Lasersohn 2015). Readings that do not require assessment sensitivity are given truth conditions according to those given for monadic truth in Lasersohn 2015. The interaction of readings with their grammatical distribution provides additional theoretical insights into the linguistic contexts that are conducive to assessment sensitivity, actuality inferences, and counterfactual readings. Most notably, it is shown that assessment sensitivity is only present in modal auxiliaries that are in the non-tense-bearing paradigm. Parts of the theoretical treatment presented in this work have been applied in areas of automated classification of modal auxiliary verbs (Moon 2011, Moon 2012, Moon et al. 2016), showing that temporal, aspectual, and argument structure information can be used to determine the most likely reading of a modal auxiliary at the sentence level, increasing the ease of reading identification for automated tools