2 research outputs found
All-in-One Image-Grounded Conversational Agents
As single-task accuracy on individual language and image tasks has improved
substantially in the last few years, the long-term goal of a generally skilled
agent that can both see and talk becomes more feasible to explore. In this
work, we focus on leveraging individual language and image tasks, along with
resources that incorporate both vision and language towards that objective. We
design an architecture that combines state-of-the-art Transformer and ResNeXt
modules fed into a novel attentive multimodal module to produce a combined
model trained on many tasks. We provide a thorough analysis of the components
of the model, and transfer performance when training on one, some, or all of
the tasks. Our final models provide a single system that obtains good results
on all vision and language tasks considered, and improves the state-of-the-art
in image-grounded conversational applications
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
We present our view of what is necessary to build an engaging open-domain
conversational agent: covering the qualities of such an agent, the pieces of
the puzzle that have been built so far, and the gaping holes we have not filled
yet. We present a biased view, focusing on work done by our own group, while
citing related work in each area. In particular, we discuss in detail the
properties of continual learning, providing engaging content, and being
well-behaved -- and how to measure success in providing them. We end with a
discussion of our experience and learnings, and our recommendations to the
community