A key task to understand an image and its
corresponding caption is not only to find
out what is shown on the picture and described
in the text, but also what is the
exact relationship between these two elements.
The long-term objective of our
work is to be able to distinguish different
types of relationship, including literal
vs. non-literal usages, as well as finegrained
non-literal usages (i.e., symbolic
vs. iconic). Here, we approach this challenging
problem by answering the question:
‘How can we quantify the degrees
of similarity between the literal meanings
expressed within images and their captions?’.
We formulate this problem as a
ranking task, where links between entities
and potential regions are created and
ranked for relevance. Using a Ranking
SVM allows us to leverage from the preference
ordering of the links, which help us
in the similarity calculation for the cases
of visual or textual ambiguity, as well as
misclassified data. Our experiments show
that aggregating different features using a
supervised ranker achieves better results
than a baseline knowledge-base method.
However, much work still lies ahead, and
we accordingly conclude the paper with a
detailed discussion of a short- and longterm
outlook on how to push our work on
relationship classification one step further
