5 research outputs found
Understanding the message of images with knowledge base traversals
The message of news articles is often supported by the pointed use
of iconic images. These images together with their captions encourage
emotional involvement of the reader. Current algorithms
for understanding the semantics of news articles focus on its text,
often ignoring the image. On the other side, works that target the
semantics of images, mostly focus on recognizing and enumerating
the objects that appear in the image. In this work, we explore
the problem from another perspective: Can we devise algorithms to
understand the message encoded by images and their captions? To
answer this question, we study how well algorithms can describe an
image-caption pair in terms of Wikipedia entities, thereby casting
the problem as an entity-ranking task with an image-caption pair
as query. Our proposed algorithm brings together aspects of entity
linking, subgraph selection, entity clustering, relatedness measures,
and learning-to-rank. In our experiments, we focus on media-iconic
image-caption pairs which often reflect complex subjects such as
sustainable energy and endangered species. Our test collection includes
a gold standard of over 300 image-caption pairs about topics
at different levels of abstraction. We show that with a MAP of 0.69,
the best results are obtained when aggregating content-based and
graph-based features in a Wikipedia-derived knowledge base