Towards Multimodal Open-World Learning in Deep Neural Networks

Over the past decade, deep neural networks have enormously advanced machine perception, especially object classification, object detection, and multimodal scene understanding. But, a major limitation of these systems is that they assume a closed-world setting, i.e., the train and the test distribution match exactly. As a result, any input belonging to a category that the system has never seen during training will not be recognized as unknown. However, many real-world applications often need this capability. For example, self-driving cars operate in a dynamic world where the data can change over time due to changes in season, geographic location, sensor types, etc. Handling such changes requires building models with open-world learning capabilities. In open-world learning, the system needs to detect novel examples which are not seen during training and update the system with new knowledge, without retraining from scratch. In this dissertation, we address gaps in the open-world learning literature and develop methods that enable efficient multimodal open-world learning in deep neural networks

NOUS: Construction and Querying of Dynamic Knowledge Graphs

The ability to construct domain specific knowledge graphs (KG) and perform question-answering or hypothesis generation is a transformative capability. Despite their value, automated construction of knowledge graphs remains an expensive technical challenge that is beyond the reach for most enterprises and academic institutions. We propose an end-to-end framework for developing custom knowledge graph driven analytics for arbitrary application domains. The uniqueness of our system lies A) in its combination of curated KGs along with knowledge extracted from unstructured text, B) support for advanced trending and explanatory questions on a dynamic KG, and C) the ability to answer queries where the answer is embedded across multiple data sources.Comment: Codebase: https://github.com/streaming-graphs/NOU

Challenges in Bridging Social Semantics and Formal Semantics on the Web

This paper describes several results of Wimmics, a research lab which names stands for: web-instrumented man-machine interactions, communities, and semantics. The approaches introduced here rely on graph-oriented knowledge representation, reasoning and operationalization to model and support actors, actions and interactions in web-based epistemic communities. The re-search results are applied to support and foster interactions in online communities and manage their resources

Training an adaptive dialogue policy for interactive learning of visually grounded word meanings

We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation framework - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effects of different dialogue policies and capabilities on the accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns. Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.Comment: 11 pages, SIGDIAL 2016 Conferenc