2 research outputs found
VOICE: Visual Oracle for Interaction, Conversation, and Explanation
We present VOICE, a novel approach for connecting large language models'
(LLM) conversational capabilities with interactive exploratory visualization.
VOICE introduces several innovative technical contributions that drive our
conversational visualization framework. Our foundation is a pack-of-bots that
can perform specific tasks, such as assigning tasks, extracting instructions,
and generating coherent content. We employ fine-tuning and prompt engineering
techniques to tailor bots' performance to their specific roles and accurately
respond to user queries, and a new prompt-based iterative scene-tree generation
establishes a coupling with a structural model. Our text-to-visualization
method generates a flythrough sequence matching the content explanation.
Finally, 3D natural language interaction provides capabilities to navigate and
manipulate the 3D models in real-time. The VOICE framework can receive
arbitrary voice commands from the user and responds verbally, tightly coupled
with corresponding visual representation with low latency and high accuracy. We
demonstrate the effectiveness and high generalizability potential of our
approach by applying it to two distinct domains: analyzing three 3D molecular
models with multi-scale and multi-instance attributes, and showcasing its
effectiveness on a cartographic map visualization. A free copy of this paper
and all supplemental materials are available at https://osf.io/g7fbr/