In the rapidly evolving landscape of human-computer interaction, the
integration of vision capabilities into conversational agents stands as a
crucial advancement. This paper presents an initial implementation of a
dialogue manager that leverages the latest progress in Large Language Models
(e.g., GPT-4, IDEFICS) to enhance the traditional text-based prompts with
real-time visual input. LLMs are used to interpret both textual prompts and
visual stimuli, creating a more contextually aware conversational agent. The
system's prompt engineering, incorporating dialogue with summarisation of the
images, ensures a balance between context preservation and computational
efficiency. Six interactions with a Furhat robot powered by this system are
reported, illustrating and discussing the results obtained. By implementing
this vision-enabled dialogue system, the paper envisions a future where
conversational agents seamlessly blend textual and visual modalities, enabling
richer, more context-aware dialogues.Comment: 8 pages, 3 figure