The CHORUS gap analysis on user-centered methodology for design and evaluation of multi-media information access systems

CHORUS is a Coordination Action, a specific type of project funded by the European commission under its research programmes, intended to bring together research projects with common goals, in the field of search technologies for digital audio-visual content, one of the strategic objectives of the current research frame program. CHORUS coordinates a number of research projects in the general area of audio-visual and multi-media information access and management. The most important single contribution of the CHORUS work plan will be to provide a survey of the field and a roadmap with a gap analysis for the realisation of viable audio-visual search engines by European partners. This is done by several means. CHORUS organises Think-Tanks with industrial participation, focussed workshops to treat specific questions, and more general conferences for academic discussions. CHORUS is now in its final phase, and is currently preparing its final report together with a final conference to mark its publication

This is what a pandemic looks like: Visual framing of COVID-19 on search engines

In today's high-choice media environment, search engines play an integral role in informing individuals and societies about the latest events. The importance of search algorithms is even higher at the time of crisis, when users search for information to understand the causes and the consequences of the current situation and decide on their course of action. In our paper, we conduct a comparative audit of how different search engines prioritize visual information related to COVID-19 and what consequences it has for the representation of the pandemic. Using a virtual agent-based audit approach, we examine image search results for the term "coronavirus" in English, Russian and Chinese on five major search engines: Google, Yandex, Bing, Yahoo, and DuckDuckGo. Specifically, we focus on how image search results relate to generic news frames (e.g., the attribution of responsibility, human interest, and economics) used in relation to COVID-19 and how their visual composition varies between the search engines.Comment: 18 pages, 1 figure, 3 table

Learning Social Image Embedding with Deep Multimodal Attention Networks

Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visual content), simply employing the embedding learnt from network structure or data content results in sub-optimal social image representation. In this paper, we propose a novel social image embedding approach called Deep Multimodal Attention Networks (DMAN), which employs a deep model to jointly embed multimodal contents and link information. Specifically, to effectively capture the correlations between multimodal contents, we propose a multimodal attention network to encode the fine-granularity relation between image regions and textual words. To leverage the network structure for embedding learning, a novel Siamese-Triplet neural network is proposed to model the links among images. With the joint deep model, the learnt embedding can capture both the multimodal contents and the nonlinear network information. Extensive experiments are conducted to investigate the effectiveness of our approach in the applications of multi-label classification and cross-modal search. Compared to state-of-the-art image embeddings, our proposed DMAN achieves significant improvement in the tasks of multi-label classification and cross-modal search

How Does Web Advertising Affect Users' Information Seeking, Website Evaluation, and Source

Web advertising (ads) is currently a big issue for information seeking behavior on the Internet. Since websites are an important source of information in our daily lives, the websites have become popular media for advertising. The advertisers think that Web ads are the best way to get people to know quickly many things about a product or service [7]. However, from the viewers??? perspective, Web ads can be considered an obstacle to users??? information seeking on a website. Actually, Web ads received complains as being annoying or intruding, and it is considered to have negative effects such as affecting people???s task performance by many researchers [2, 5, 6]. The objective of this research is to know how Web advertising affects users??? information seeking, website evaluation, and source evaluation. Most previous studies have focused on the effectiveness of Web ads toward viewers. Not many studies researched on how Web ads affect users??? information seeking performance, website evaluation, and content evaluation of a website. According to visual attention theory, since people???s visual attention is limited, the more attention Web ads received from a viewer, the less amount of attention was available for the visual search task and the more damaging it was to the search task [5]. Therefore, Web ads will affect people???s information seeking performance, and consequently it will influence website evaluation and source evaluation of the website. My research question is: how do different types of Web advertising which attract people???s attention at differing levels affect users??? information seeking performance, website evaluation, and source evaluation? I will employ an empirical research design with various levels of attention grabbing Web ads and users??? information seeking performance, website evaluation, and source evaluation. I will use a total of 200 college students as the sample population in this research, and I will randomly assign the participants into four different groups (50 participants in each group): three experimental groups for low, medium, and high attention grabbing Web ads and one control group (a group with no Web ads). There would be no significant difference between groups in terms of years of computer experience, years of Internet experience, and gender

No Longer in Vogue? The Exploration of Motivations Underlying Millennials\u27 Information Seeking Through Digital Fashion Media

Information seeking is a key component of the consumer decision making process in which consumers sift through information to enhance their knowledge and develop their attitudes towards a good or service before making a purchase decision. With increasing global digitalization, more millennial consumers are looking to digital fashion media (fashion blogs and relevant social media) for information seeking as opposed to traditional media (fashion magazines). This study explores the millennial consumers\u27 psychological motivations to engage in information seeking through digital fashion media grounded by Functional Theory. In order to understand these motivations, six of focus group interviews were conducted with twenty four female consumers between the ages of 18 and 33. Following thematic analysis, six key motivations for information seeking through digital fashion media emerged: search autonomy, instant gratification, visual inspiration, authenticity, virtual storage and gratuitous information. Theoretical and practical implications are discussed

Hashing for Multimedia Similarity Modeling and Large-Scale Retrieval

In recent years, the amount of multimedia data such as images, texts, and videos have been growing rapidly on the Internet. Motivated by such trends, this thesis is dedicated to exploiting hashing-based solutions to reveal multimedia data correlations and support intra-media and inter-media similarity search among huge volumes of multimedia data. We start by investigating a hashing-based solution for audio-visual similarity modeling and apply it to the audio-visual sound source localization problem. We show that synchronized signals in audio and visual modalities demonstrate similar temporal changing patterns in certain feature spaces. We propose to use a permutation-based random hashing technique to capture the temporal order dynamics of audio and visual features by hashing them along the temporal axis into a common Hamming space. In this way, the audio-visual correlation problem is transformed into a similarity search problem in the Hamming space. Our hashing-based audio-visual similarity modeling has shown superior performances in the localization and segmentation of sounding objects in videos. The success of the permutation-based hashing method motivates us to generalize and formally define the supervised ranking-based hashing problem, and study its application to large-scale image retrieval. Specifically, we propose an effective supervised learning procedure to learn optimized ranking-based hash functions that can be used for large-scale similarity search. Compared with the randomized version, the optimized ranking-based hash codes are much more compact and discriminative. Moreover, it can be easily extended to kernel space to discover more complex ranking structures that cannot be revealed in linear subspaces. Experiments on large image datasets demonstrate the effectiveness of the proposed method for image retrieval. We further studied the ranking-based hashing method for the cross-media similarity search problem. Specifically, we propose two optimization methods to jointly learn two groups of linear subspaces, one for each media type, so that features\u27 ranking orders in different linear subspaces maximally preserve the cross-media similarities. Additionally, we develop this ranking-based hashing method in the cross-media context into a flexible hashing framework with a more general solution. We have demonstrated through extensive experiments on several real-world datasets that the proposed cross-media hashing method can achieve superior cross-media retrieval performances against several state-of-the-art algorithms. Lastly, to make better use of the supervisory label information, as well as to further improve the efficiency and accuracy of supervised hashing, we propose a novel multimedia discrete hashing framework that optimizes an instance-wise loss objective, as compared to the pairwise losses, using an efficient discrete optimization method. In addition, the proposed method decouples the binary codes learning and hash function learning into two separate stages, thus making the proposed method equally applicable for both single-media and cross-media search. Extensive experiments on both single-media and cross-media retrieval tasks demonstrate the effectiveness of the proposed method