Chapter 12 Answering Questions about Moving Objects in Videos

Current question answering systems succeed in many respects regarding questions about textual documents. However, information exists in other media, which provides both opportunities and challenges for question answering. We describe our efforts in extending question answering capabilities to video data: our implemented prototype, Spot, can answer questions about moving objects in a surveillance setting. This novel application of vision and language technology is situated within a larger framework designed to integrate knowledge from multiple domains under a common representation. We believe that our framework will support the next generation of multimodal natural language information access systems. 1