12,783 research outputs found
Language-based sensing descriptors for robot object grounding
In this work, we consider an autonomous robot that is required
to understand commands given by a human through natural language.
Specifically, we assume that this robot is provided with an internal
representation of the environment. However, such a representation is unknown
to the user. In this context, we address the problem of allowing a
human to understand the robot internal representation through dialog.
To this end, we introduce the concept of sensing descriptors. Such representations
are used by the robot to recognize unknown object properties
in the given commands and warn the user about them. Additionally, we
show how these properties can be learned over time by leveraging past
interactions in order to enhance the grounding capabilities of the robot
A Review of Verbal and Non-Verbal Human-Robot Interactive Communication
In this paper, an overview of human-robot interactive communication is
presented, covering verbal as well as non-verbal aspects of human-robot
interaction. Following a historical introduction, and motivation towards fluid
human-robot communication, ten desiderata are proposed, which provide an
organizational axis both of recent as well as of future research on human-robot
communication. Then, the ten desiderata are examined in detail, culminating to
a unifying discussion, and a forward-looking conclusion
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities
We introduce the Qwen-VL series, a set of large-scale vision-language models
designed to perceive and understand both text and images. Comprising Qwen-VL
and Qwen-VL-Chat, these models exhibit remarkable performance in tasks like
image captioning, question answering, visual localization, and flexible
interaction. The evaluation covers a wide range of tasks including zero-shot
captioning, visual or document visual question answering, and grounding. We
demonstrate the Qwen-VL outperforms existing Large Vision Language Models
(LVLMs). We present their architecture, training, capabilities, and
performance, highlighting their contributions to advancing multimodal
artificial intelligence. Code, demo and models are available at
https://github.com/QwenLM/Qwen-VL.Comment: Code, demo and models are available at
https://github.com/QwenLM/Qwen-V
Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning
Recent advancements enlarge the capabilities of large language models (LLMs)
in zero-shot image-to-text generation and understanding by integrating
multi-modal inputs. However, such success is typically limited to English
scenarios due to the lack of large-scale and high-quality non-English
multi-modal resources, making it extremely difficult to establish competitive
counterparts in other languages. In this paper, we introduce the Ziya-Visual
series, a set of bilingual large-scale vision-language models (LVLMs) designed
to incorporate visual semantics into LLM for multi-modal dialogue. Composed of
Ziya-Visual-Base and Ziya-Visual-Chat, our models adopt the Querying
Transformer from BLIP-2, further exploring the assistance of optimization
schemes such as instruction tuning, multi-stage training and low-rank
adaptation module for visual-language alignment. In addition, we stimulate the
understanding ability of GPT-4 in multi-modal scenarios, translating our
gathered English image-text datasets into Chinese and generating
instruction-response through the in-context learning method. The experiment
results demonstrate that compared to the existing LVLMs, Ziya-Visual achieves
competitive performance across a wide range of English-only tasks including
zero-shot image-text retrieval, image captioning, and visual question
answering. The evaluation leaderboard accessed by GPT-4 also indicates that our
models possess satisfactory image-text understanding and generation
capabilities in Chinese multi-modal scenario dialogues. Code, demo and models
are available at
~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}
Using dialogue to learn math in the LeActiveMath project
We describe a tutorial dialogue system under development that assists students in learning how to differentiate equations. The system uses deep natural language understanding and generation to both interpret students â utterances and automatically generate a response that is both mathematically correct and adapted pedagogically and linguistically to the local dialogue context. A domain reasoner provides the necessary knowledge about how students should approach math problems as well as their (in)correctness, while a dialogue manager directs pedagogical strategies and keeps track of what needs to be done to keep the dialogue moving along.
MULTI-MODAL TASK INSTRUCTIONS TO ROBOTS BY NAIVE USERS
This thesis presents a theoretical framework for the design of user-programmable
robots. The objective of the work is to investigate multi-modal unconstrained natural
instructions given to robots in order to design a learning robot. A corpus-centred
approach is used to design an agent that can reason, learn and interact with a human in a
natural unconstrained way. The corpus-centred design approach is formalised and
developed in detail. It requires the developer to record a human during interaction and
analyse the recordings to find instruction primitives. These are then implemented into a
robot. The focus of this work has been on how to combine speech and gesture using
rules extracted from the analysis of a corpus. A multi-modal integration algorithm is
presented, that can use timing and semantics to group, match and unify gesture and
language. The algorithm always achieves correct pairings on a corpus and initiates
questions to the user in ambiguous cases or missing information. The domain of card
games has been investigated, because of its variety of games which are rich in rules and
contain sequences. A further focus of the work is on the translation of rule-based
instructions. Most multi-modal interfaces to date have only considered sequential
instructions. The combination of frame-based reasoning, a knowledge base organised as
an ontology and a problem solver engine is used to store these rules. The understanding
of rule instructions, which contain conditional and imaginary situations require an agent
with complex reasoning capabilities. A test system of the agent implementation is also
described. Tests to confirm the implementation by playing back the corpus are
presented. Furthermore, deployment test results with the implemented agent and human
subjects are presented and discussed. The tests showed that the rate of errors that are
due to the sentences not being defined in the grammar does not decrease by an
acceptable rate when new grammar is introduced. This was particularly the case for
complex verbal rule instructions which have a large variety of being expressed
- âŠ