Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.Comment: 7 pages, 8 figure

The kindergarten-path effect revisited: children’s use of context in processing structural ambiguities

Research with adults has shown that ambiguous spoken sentences are resolved efficiently, exploiting multiple cues—including referential context—to select the intended meaning. Paradoxically, children appear to be insensitive to referential cues when resolving ambiguous sentences, relying instead on statistical properties intrinsic to the language such as verb biases. The possibility that children’s insensitivity to referential context may be an artifact of the experimental design used in previous work was explored with 60 4- to 11-year-olds. An act-out task was designed to discourage children from making incorrect pragmatic inferences and to prevent premature and ballistic responses by enforcing delayed actions. Performance on this task was compared directly with the standard act-out task used in previous studies. The results suggest that young children (5 years) do not use contextual information, even under conditions designed to maximize their use of such cues, but that adult-like processing is evident by approximately 8 years of age. These results support and extend previous findings by Trueswell and colleagues (Cognition (1999), Vol. 73, pp. 89–134) and are consistent with a constraint-based learning account of children’s linguistic development.</p

Information structure in linguistic theory and in speech production : validation of a cross-linguistic data set

The aim of this paper is to validate a dataset collected by means of production experiments which are part of the Questionnaire on Information Structure. The experiments generate a range of information structure contexts that have been observed in the literature to induce specific constructions. This paper compares the speech production results from a subset of these experiments with specific claims about the reflexes of information structure in four different languages. The results allow us to evaluate and in most cases validate the efficacy of our elicitation paradigms, to identify potentially fruitful avenues of future research, and to highlight issues involved in interpreting speech production data of this kind

Domain transfer for deep natural language generation from abstract meaning representations

Stochastic natural language generation systems that are trained from labelled datasets are often domainspecific in their annotation and in their mapping from semantic input representations to lexical-syntactic outputs. As a result, learnt models fail to generalize across domains, heavily restricting their usability beyond single applications. In this article, we focus on the problem of domain adaptation for natural language generation. We show how linguistic knowledge from a source domain, for which labelled data is available, can be adapted to a target domain by reusing training data across domains. As a key to this, we propose to employ abstract meaning representations as a common semantic representation across domains. We model natural language generation as a long short-term memory recurrent neural network encoderdecoder, in which one recurrent neural network learns a latent representation of a semantic input, and a second recurrent neural network learns to decode it to a sequence of words. We show that the learnt representations can be transferred across domains and can be leveraged effectively to improve training on new unseen domains. Experiments in three different domains and with six datasets demonstrate that the lexical-syntactic constructions learnt in one domain can be transferred to new domains and achieve up to 75-100% of the performance of in-domain training. This is based on objective metrics such as BLEU and semantic error rate and a subjective human rating study. Training a policy from prior knowledge from a different domain is consistently better than pure in-domain training by up to 10%

On the automaticity of language processing

People speak and listen to language all the time. Given this high frequency of use, it is often suggested that at least some aspects of language processing are highly overlearned and therefore occur “automatically”. Here we critically examine this suggestion. We first sketch a framework that views automaticity as a set of interrelated features of mental processes and a matter of degree rather than a single feature that is all-or-none. We then apply this framework to language processing. To do so, we carve up the processes involved in language use according to (a) whether language processing takes place in monologue or dialogue, (b) whether the individual is comprehending or producing language, (c) whether the spoken or written modality is used, and (d) the linguistic processing level at which they occur, that is, phonology, the lexicon, syntax, or conceptual processes. This exercise suggests that while conceptual processes are relatively non-automatic (as is usually assumed), there is also considerable evidence that syntactic and lexical lower-level processes are not fully automatic. We close by discussing entrenchment as a set of mechanisms underlying automatization

The Role of Thematic Roles in Sentence Processing: Evidence from Structural Priming in Young Children

The syntactic realization of a verb’s arguments is constrained by the role that the argument plays in the meaning of the verb. In most linguistic frameworks, these constraints are captured by mappings between syntactic functions and thematic roles. Such mappings clearly shape our interpretation of novel verbs. But there is controversy about when these mappings develop and whether they are employed in the processing of utterances containing known verbs. We explored these issues using the visual-world paradigm and structural priming during comprehension in 4-year-old children. In Experiment I, we found robust priming of dative constructions. This priming persisted when animacy cues were put in competition with argument structure, indicating that the locus of priming was either in syntax or in the mapping between thematic roles and syntactic functions. Experiment II demonstrated priming from locatives to datives indicating that this priming was not purely syntactic. Together these experiments provide evidence for the use of thematic mappings during sentence processing, independent of confounding syntactic or conceptual factors. We discuss the developmental implications, apparent discrepancies with the adult priming literature, and the compatibility of our findings with different theories of argument structure alternations.Psycholog

The development of conversational and communication skills

This thesis investigates the development of children's conversational and communication skills. This is done by investigating both communicative process and outcome in two communication media: face-to-face interaction and audio-only interaction. Communicative outcome is objectively measured by assessing accuracy of performance of communication tasks. A multi-level approach to the assessment of communicative process is taken. Non-verbal aspects of process which are investigated are gaze and gesture. Verbal aspects of process range from global linguistic assessments such as length of conversational turn, to a detailed coding of utterance function according to Conversational Games analysis. The results show that children of 6 years and less do not adapt to the loss of visual signals in audio-only communication, and their performance suffers. Both the structure of children's dialogues and their use of visual signals were found to differ from that of adults. It is concluded that both verbal and non-verbal communication strategies develop into adulthood. Successful integration of these different aspects of communication is central to being an effective communicator