74,448 research outputs found
Learning and generalization of compositional representations of visual scenes
Complex visual scenes that are composed of multiple objects, each with
attributes, such as object name, location, pose, color, etc., are challenging
to describe in order to train neural networks. Usually,deep learning networks
are trained supervised by categorical scene descriptions. The common
categorical description of a scene contains the names of individual objects but
lacks information about other attributes. Here, we use distributed
representations of object attributes and vector operations in a vector symbolic
architecture to create a full compositional description of a scene in a
high-dimensional vector. To control the scene composition, we use artificial
images composed of multiple, translated and colored MNIST digits. In contrast
to learning category labels, here we train deep neural networks to output the
full compositional vector description of an input image. The output of the deep
network can then be interpreted by a VSA resonator network, to extract object
identity or other properties of indiviual objects. We evaluate the performance
and generalization properties of the system on randomly generated scenes.
Specifically, we show that the network is able to learn the task and generalize
to unseen seen digit shapes and scene configurations. Further, the
generalisation ability of the trained model is limited. For example, with a gap
in the training data, like an object not shown in a particular image location
during training, the learning does not automatically fill this gap.Comment: 10 pages, 6 figure
Grounding semantics in robots for Visual Question Answering
In this thesis I describe an operational implementation of an object detection and description system that incorporates in an end-to-end Visual Question Answering system and evaluated it on two visual question answering datasets for compositional language and elementary visual reasoning
- …