4 research outputs found
Vital information matching in vision-and-language navigation
With the rapid development of artificial intelligence technology, many researchers have begun to focus on visual language navigation, which is one of the most important tasks in multi-modal machine learning. The focus of this multi-modal field is how to fuse multiple inputs, which is crucial for the integrated feedback of intrinsic information. However, the existing models are only implemented through simple data augmentation or expansion, and are obviously far from being able to tap the intrinsic relationship between modalities. In this paper, to overcome these challenges, a novel multi-modal matching feedback self-tuning model is proposed, which is a novel neural network called Vital Information Matching Feedback Self-tuning Network (VIM-Net). Our VIM-Net network is mainly composed of two matching feedback modules, a visual matching feedback module (V-mat) and a trajectory matching feedback module (T-mat). Specifically, V-mat matches the target information of visual recognition with the entity information extracted by the command; T-mat matches the serialized trajectory feature with the direction of movement of the command. Ablation experiments and comparative experiments are conducted on the proposed model using the Matterport3D simulator and the Room-to-Room (R2R) benchmark datasets, and the final navigation effect is shown in detail. The results prove that the model proposed in this paper is indeed effective on the task
Accessible Instruction-Following Agent
Humans can collaborate and complete tasks based on visual signals and
instruction from the environment. Training such a robot is difficult especially
due to the understanding of the instruction and the complicated environment.
Previous instruction-following agents are biased to English-centric corpus,
making it unrealizable to be applied to users that use multiple languages or
even low-resource languages. Nevertheless, the instruction-following agents are
pre-trained in a mode that assumes the user can observe the environment, which
limits its accessibility. In this work, we're trying to generalize the success
of instruction-following agents to non-English languages with little corpus
resources, and improve its intractability and accessibility. We introduce UVLN
(Universal Vision-Language Navigation), a novel machine-translation
instructional augmented framework for cross-lingual vision-language navigation,
with a novel composition of state-of-the-art large language model (GPT3) with
the image caption model (BLIP). We first collect a multilanguage
vision-language navigation dataset via machine translation. Then we extend the
standard VLN training objectives to a multilingual setting via a cross-lingual
language encoder. The alignment between different languages is captured through
a shared vision and action context via a cross-modal transformer, which encodes
the inputs of language instruction, visual observation, and action decision
sequences. To improve the intractability, we connect our agent with the large
language model that informs the situation and current state to the user and
also explains the action decisions. Experiments over Room Across Room Dataset
prove the effectiveness of our approach. And the qualitative results show the
promising intractability and accessibility of our instruction-following agent