Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Computational and Robotic Models of Early Language Development: A Review

We review computational and robotics models of early language learning and development. We first explain why and how these models are used to understand better how children learn language. We argue that they provide concrete theories of language learning as a complex dynamic system, complementing traditional methods in psychology and linguistics. We review different modeling formalisms, grounded in techniques from machine learning and artificial intelligence such as Bayesian and neural network approaches. We then discuss their role in understanding several key mechanisms of language development: cross-situational statistical learning, embodiment, situated social interaction, intrinsically motivated learning, and cultural evolution. We conclude by discussing future challenges for research, including modeling of large-scale empirical data about language acquisition in real-world environments. Keywords: Early language learning, Computational and robotic models, machine learning, development, embodiment, social interaction, intrinsic motivation, self-organization, dynamical systems, complexity.Comment: to appear in International Handbook on Language Development, ed. J. Horst and J. von Koss Torkildsen, Routledg

SEMA4A: An ontology for emergency notification systems accessibility

This is the post-print version of the final paper published in Expert Systems with Applications. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2009 Elsevier B.V.Providing alert communication in emergency situations is vital to reduce the number of victims. Reaching this goal is challenging due to users’ diversity: people with disabilities, elderly and children, and other vulnerable groups. Notifications are critical when an emergency scenario is going to happen (e.g. a typhoon approaching) so the ability to transmit notifications to different kind of users is a crucial feature for such systems. In this work an ontology was developed by investigating different sources: accessibility guidelines, emergency response systems, communication devices and technologies, taking into account the different abilities of people to react to different alarms (e.g. mobile phone vibration as an alarm for deafblind people). We think that the proposed ontology addresses the information needs for sharing and integrating emergency notification messages over distinct emergency response information systems providing accessibility under different conditions and for different kind of users.Ministerio de Educación y Cienci

Classifying Crises-Information Relevancy with Semantics

Social media platforms have become key portals for sharing and consuming information during crisis situations. However, humanitarian organisations and affected communities often struggle to sieve through the large volumes of data that are typically shared on such platforms during crises to determine which posts are truly relevant to the crisis, and which are not. Previous work on automatically classifying crisis information was mostly focused on using statistical features. However, such approaches tend to be inappropriate when processing data on a type of crisis that the model was not trained on, such as processing information about a train crash, whereas the classifier was trained on floods, earthquakes, and typhoons. In such cases, the model will need to be retrained, which is costly and time-consuming. In this paper, we explore the impact of semantics in classifying Twitter posts across same, and different, types of crises. We experiment with 26 crisis events, using a hybrid system that combines statistical features with various semantic features extracted from external knowledge bases. We show that adding semantic features has no noticeable benefit over statistical features when classifying same-type crises, whereas it enhances the classifier performance by up to 7.2% when classifying information about a new type of crisis

Developmental Stages of Perception and Language Acquisition in a Perceptually Grounded Robot

The objective of this research is to develop a system for language learning based on a minimum of pre-wired language-specific functionality, that is compatible with observations of perceptual and language capabilities in the human developmental trajectory. In the proposed system, meaning (in terms of descriptions of events and spatial relations) is extracted from video images based on detection of position, motion, physical contact and their parameters. Mapping of sentence form to meaning is performed by learning grammatical constructions that are retrieved from a construction inventory based on the constellation of closed class items uniquely identifying the target sentence structure. The resulting system displays robust acquisition behavior that reproduces certain observations from developmental studies, with very modest “innate” language specificity

Autonomy Operating System for UAVs: Pilot-in-a-Box

The Autonomy Operating System (AOS) is an open flight software platform with Artificial Intelligence for smart UAVs. It is built to be extendable with new apps, similar to smartphones, to enable an expanding set of missions and capabilities. AOS has as its foundations NASAs core flight executive and core flight software (cFEcFS). Pilot-in-a-Box (PIB) is an expanding collection of interacting AOS apps that provide the knowledge and intelligence onboard a UAV to safely and autonomously fly in the National Air Space, eventually without a remote human ground crew. Longer-term, the goal of PIB is to provide the capability for pilotless air vehicles such as air taxis that will be key for new transportation concepts such as mobility-on-demand. PIB provides the procedural knowledge, situational awareness, and anticipatory planning (thinking ahead of the plane) that comprises pilot competencies. These competencies together with a natural language interface will enable Pilot-in-a-Box to dialogue directly with Air Traffic Management from takeoff through landing. This paper describes the overall AOS architecture, Artificial Intelligence reasoning engines, Pilot-in-a-box competencies, and selected experimental flight tests to date

The use of colloquial words in advanced French interlanguage

This article addresses the issue of underrepresentation or avoidance of colloquial words in a cross-sectional corpus of advanced French interlanguage (IL) of 29 Dutch L1 speakers and in a longitudinal corpus of 6 Hiberno-Irish English L1 speakers compared with a control of 6 native speakers of French. The main independent variable analysed in the latter corpus is the effect of spending a year in a francophone environment. This analysis is supplemented by a separate study of sociobiographical and psychological factors that affect the use of colloquial vocabulary in the cross-sectional corpus. Colloquial words are not exceptionally complex morphologically and present no specific grammatical difficulties, yet they are very rare in our data. Multivariate regression analyses suggest that only active authentic communication in the target language (TL) predicts the use of colloquial lexemes in the cross-sectional corpus. This result was confirmed in the longitudinal corpus where a t-test showed that the proportion of colloquial lexemes increased significantly after a year abroad