Few-Shot Visual Grounding for Natural Human-Robot Interaction

Natural Human-Robot Interaction (HRI) is one of the key components for service robots to be able to work in human-centric environments. In such dynamic environments, the robot needs to understand the intention of the user to accomplish a task successfully. Towards addressing this point, we propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. Unlike most grounding methods that tackle the challenge using pre-trained object detectors via a two-stepped process, we develop a single stage zero-shot model that is able to provide predictions in unseen data. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets. Experimental results showed that the proposed model performs well in terms of accuracy and speed, while showcasing robustness to variation in the natural language input.Comment: 6 pages, 4 figures, ICARSC2021 accepte

A Hybrid Compositional Reasoning Approach for Interactive Robot Manipulation

— In this paper we present a neuro-symbolic (hybrid)compositional reasoning model for coupling language-guidedvisual reasoning with robot manipulation. A non-expert humanuser can prompt the robot agent using natural language,providing a referring expression, a question or a grasp actioninstruction. The model tackles all cases in a task-agnosticfashion through the utilization of a shared library of primitiveskills. Each primitive handles an independent sub-task, suchas reasoning about visual attributes, spatial relation comprehension, logic and enumeration, as well as arm control. Alanguage parser maps the input query to an executable programcomposed of such primitives depending on the context. Whilesome primitives are purely symbolic operations (e.g. counting),others are trainable neural functions (e.g. image/word grounding), therefore marrying the interpretability and systematicgeneralization benefits of discrete symbolic approaches withthe scalability and representational power of deep networks.We generate a synthetic dataset of tabletop scenes to train ourapproach and perform several evaluation experiments for visualreasoning. Results show that the proposed method achieves veryhigh accuracy while being transferable to real-world scenes withfew-shot visual fine-tuning. Finally, we integrate our methodwith a robot framework and demonstrate how it can serveas an interpretable solution for an interactive object pickingtask, both in simulation and with a real robot. Supplementarymaterial is available in this https URL

A Strong Transfer Baseline for RGB-D Fusion in Vision Transformers

The Vision Transformer (ViT) architecture has recently established its place in the computer vision literature, with multiple architectures for recognition of image data or other visual modalities. However, training ViTs for RGB-D object recognition remains an understudied topic, viewed in recent literature only through the lens of multi-task pretraining in multiple modalities. Such approaches are often computationally intensive and have not yet been applied for challenging object-level classification tasks. In this work, we propose a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains for single-view 3D object recognition, focusing on fusing RGB and depth representations encoded jointly by the ViT. Compared to previous works in multimodal Transformers, the key challenge here is to use the atested flexibility of ViTs to capture cross-modal interactions at the downstream and not the pretraining stage. We explore which depth representation is better in terms of resulting accuracy and compare two methods for injecting RGB-D fusion within the ViT architecture (i.e., early vs. late fusion). Our results in the Washington RGB-D Objects dataset demonstrates that in such RGB → RGB-D scenarios, late fusion techniques work better than most popularly employed early fusion. With our transfer baseline, adapted ViTs score up to 95.1\% top-1 accuracy in Washington, achieving new state-of-the-art results in this benchmark. We additionally evaluate our approach with an open-ended lifelong learning protocol, where we show that our adapted RGB-D encoder leads to features that outperform unimodal encoders, even without explicit fine-tuning. We further integrate our method with a robot framework and demonstrate how it can serve as a perception utility in an interactive robot learning scenario, both in simulation and with a real robot

Fighting the COVID-19 Infodemic with a Holistic BERT Ensemble

This paper describes the TOKOFOU system, an ensemble model for misinformation detection tasks based on six different transformer-based pre-trained encoders, implemented in the context of the COVID-19 Infodemic Shared Task for English. We fine tune each model on each of the task's questions and aggregate their prediction scores using a majority voting approach. TOKOFOU obtains an overall F1 score of 89.7%, ranking first.Comment: 4 pages, NLP4IF 202