Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning

Given the massive market of advertising and the sharply increasing online multimedia content (such as videos), it is now fashionable to promote advertisements (ads) together with the multimedia content. It is exhausted to find relevant ads to match the provided content manually, and hence, some automatic advertising techniques are developed. Since ads are usually hard to understand only according to its visual appearance due to the contained visual metaphor, some other modalities, such as the contained texts, should be exploited for understanding. To further improve user experience, it is necessary to understand both the topic and sentiment of the ads. This motivates us to develop a novel deep multimodal multitask framework to integrate multiple modalities to achieve effective topic and sentiment prediction simultaneously for ads understanding. In particular, our model first extracts multimodal information from ads and learn high-level and comparable representations. The visual metaphor of the ad is decoded in an unsupervised manner. The obtained representations are then fed into the proposed hierarchical multimodal attention modules to learn task-specific representations for final prediction. A multitask loss function is also designed to train both the topic and sentiment prediction models jointly in an end-to-end manner. We conduct extensive experiments on the latest and large advertisement dataset and achieve state-of-the-art performance for both prediction tasks. The obtained results could be utilized as a benchmark for ads understanding.Comment: 8 pages, 5 figure

The Industry and Policy Context for Digital Games for Empowerment and Inclusion:Market Analysis, Future Prospects and Key Challenges in Videogames, Serious Games and Gamification

The effective use of digital games for empowerment and social inclusion (DGEI) of people and communities at risk of exclusion will be shaped by, and may influence the development of a range of sectors that supply products, services, technology and research. The principal industries that would appear to be implicated are the 'videogames' industry, and an emerging 'serious games' industry. The videogames industry is an ecosystem of developers, publishers and other service providers drawn from the interactive media, software and broader ICT industry that services the mainstream leisure market in games, The 'serious games' industry is a rather fragmented and growing network of firms, users, research and policy makers from a variety of sectors. This emerging industry is are trying to develop knowledge, products, services and a market for the use of digital games, and products inspired by digital games, for a range of non-leisure applications. This report provides a summary of the state of play of these industries, their trajectories and the challenges they face. It also analyses the contribution they could make to exploiting digital games for empowerment and social inclusion. Finally, it explores existing policy towards activities in these industries and markets, and draws conclusions as to the future policy relevance of engaging with them to support innovation and uptake of effective digital game-based approaches to empowerment and social inclusion.JRC.J.3-Information Societ

Direct kernel biased discriminant analysis: a new content-based image retrieval relevance feedback algorithm

In recent years, a variety of relevance feedback (RF) schemes have been developed to improve the performance of content-based image retrieval (CBIR). Given user feedback information, the key to a RF scheme is how to select a subset of image features to construct a suitable dissimilarity measure. Among various RF schemes, biased discriminant analysis (BDA) based RF is one of the most promising. It is based on the observation that all positive samples are alike, while in general each negative sample is negative in its own way. However, to use BDA, the small sample size (SSS) problem is a big challenge, as users tend to give a small number of feedback samples. To explore solutions to this issue, this paper proposes a direct kernel BDA (DKBDA), which is less sensitive to SSS. An incremental DKBDA (IDKBDA) is also developed to speed up the analysis. Experimental results are reported on a real-world image collection to demonstrate that the proposed methods outperform the traditional kernel BDA (KBDA) and the support vector machine (SVM) based RF algorithms