Learning to Generate Unambiguous Spatial Referring Expressions for Real-World Environments

Referring to objects in a natural and unambiguous manner is crucial for effective human-robot interaction. Previous research on learning-based referring expressions has focused primarily on comprehension tasks, while generating referring expressions is still mostly limited to rule-based methods. In this work, we propose a two-stage approach that relies on deep learning for estimating spatial relations to describe an object naturally and unambiguously with a referring expression. We compare our method to the state of the art algorithm in ambiguous environments (e.g., environments that include very similar objects with similar relationships). We show that our method generates referring expressions that people find to be more accurate ($\sim$30% better) and would prefer to use ($\sim$32% more often).Comment: International Conference on Intelligent Robots and Systems (IROS 2019), Demo 1: Finding the described object (https://youtu.be/BE6-F6chW0w), Demo 2: Referring to the pointed object (https://youtu.be/nmmv6JUpy8M), Supplementary Video (https://youtu.be/sFjBa_MHS98

Continuous Spatial and Temporal Representations in Machine Vision

This thesis explores continuous spatial and temporal representations in machine vision. For spatial representations, we explore the Spatial Semantic Pointer as a biologically plausible representation of continuous space its use in performing spatial memory and reasoning tasks. We show that SSPs can be used to encode visual images into high dimensional memory vectors. These vectors can be used to store, retrieve, and manipulate spatial information, as well as perform search and scanning tasks within the vector algebra space. We also demonstrate the psychological plausibility of these representations by qualitatively reproducing Kosslyn's famous map scanning experiment. For temporal representations, we extend the original 1D Legendre Memory Unit to take multi-dimensional input signals and compare its ability to store temporal information against the Long Short-Term Memory Unit on the task of video action recognition. We show that the multi-dimensional LMU is able to match the LSTM in representing visual data over time. In particular, we demonstrate that the LMU is able to achieve much better performance when the total number of parameters is limited and that the LMU architecture allows it to continue operating at with fewer parameters than the LSTM