Ten Research Questions for Scalable Multimedia Analytics

International audienceThe scale and complexity of multimedia collections is ever increasing, as is the desire to harvest useful insight from the collections. To optimally support the complex quest for insight, multimedia ana-lytics has emerged as a new research area that combines concepts and techniques from multimedia analysis and visual analytics into a single framework. State of the art multimedia analytics solutions are highly interactive and give users freedom in how they perform their analytics task, but they do not scale well. State of the art scalable database management solutions, on the other hand, are not yet designed for multimedia analytics workloads. In this position paper we therefore argue the need for research on scalable multimedia analytics, a new research area built on the three pillars of visual analytics, multimedia analysis and database management. We propose a specific goal for scalable multimedia analyt-ics and present several important research questions that we believe must be addressed in order to achieve that goal

Visual Analytics of Gaze Data with Standard Multimedia Player

With the increasing number of studies, where participants’ eye movements are tracked while watching videos, the volume of gaze data records is growing tremendously. Unfortunately, in most cases, such data are collected in separate files in custom-made or proprietary data formats. These data are difficult to access even for experts and effectively inaccessible for non-experts. Normally expensive or custom-made software is necessary for their analysis. We address this problem by using existing multimedia container formats for distributing and archiving eye-tracking and gaze data bundled with the stimuli data. We define an exchange format that can be interpreted by standard multimedia players and can be streamed via the Internet. We convert several gaze data sets into our format, demonstrating the feasibility of our approach and allowing to visualize these data with standard multimedia players. We also introduce two VLC player add-ons, allowing for further visual analytics. We discuss the benefit of gaze data in a multimedia container and explain possible visual analytics approaches based on our implementations, converted datasets, and first user interviews

Steering data quality with visual analytics: The complexity challenge

Data quality management, especially data cleansing, has been extensively studied for many years in the areas of data management and visual analytics. In the paper, we first review and explore the relevant work from the research areas of data management, visual analytics and human-computer interaction. Then for different types of data such as multimedia data, textual data, trajectory data, and graph data, we summarize the common methods for improving data quality by leveraging data cleansing techniques at different analysis stages. Based on a thorough analysis, we propose a general visual analytics framework for interactively cleansing data. Finally, the challenges and opportunities are analyzed and discussed in the context of data and humans

Exploiting multimedia in creating and analysing multimedia Web archives

The data contained on the web and the social web are inherently multimedia and consist of a mixture of textual, visual and audio modalities. Community memories embodied on the web and social web contain a rich mixture of data from these modalities. In many ways, the web is the greatest resource ever created by human-kind. However, due to the dynamic and distributed nature of the web, its content changes, appears and disappears on a daily basis. Web archiving provides a way of capturing snapshots of (parts of) the web for preservation and future analysis. This paper provides an overview of techniques we have developed within the context of the EU funded ARCOMEM (ARchiving COmmunity MEMories) project to allow multimedia web content to be leveraged during the archival process and for post-archival analysis. Through a set of use cases, we explore several practical applications of multimedia analytics within the realm of web archiving, web archive analysis and multimedia data on the web in general

An improved model for sentiment analysis on luxury hotel review

This article proposes a heuristic model for sentiment analysis on luxury hotel reviews to analyse and explore marketing insights from attitudes and emotions expressed in reviews. We make several significant contributions to visual and multimedia analytics. This research will develop the practical application of visual and multimedia analytics as the research foundation is based on information analytics, geospatial analytics, statistical analytics and data management. Large amounts of data are generated by hotel customers on the Internet, which provides a good opportunity for managers and analysts to explore the hidden information. The analysis of luxury hotels involves different types of data, including real-world scale data, high-dimensional data and geospatial data. The diversity of data increases the difficulty of processing computational visual analytics. It leads to that some classical classification methods, which cost too much time and have high requirements for hardware, are excluded. The goal is to achieve a compromise between performance and cost. An experiment of this model is operated using data extracted from Booking.com. The entire framework of this experiment includes data collection, data preprocessing, feature engineering consisting of term frequency-inverse document frequency and Doc2Vec based feature generation and feature selection, Random Forest classification, data analysis and data visualization. The whole process combines statistical analysis, review sentiment analysis and visual analysis to make full use of this dataset and gain more decision-making information to improve luxury hotels' service quality. Compared with simple sentiment analysis, this integrated analytics in social media is expected to be used in practice to gain more insights. The result shows that luxury hotels should focus on staff training, cleanness of rooms and location choice to improve customer satisfaction. The sentiment distribution shows that scores are consistent with the emotion they show in reviews. Hotels in Spain have a much better average score than hotels in the other five countries. In the experiment, the sentiment analysis model is evaluated by receiver operating characteristic and precision-recall curve. It is proved that this model performs well. Twenty most essential features have been listed for future adjustments to the model

Big Data Analysis: A New Scheme for the Information Retrieving Based on the Content of Multimedia Documents

Big Data analysis is one of the hot topics now days for knowledge discovery in databases process. It’s considered as significant field of knowledge management. Roughly, the ¾ of organizations have been adopted some form of analytics today. The most posed question in big data analysis is how to manage and operate in it? In this study, we explain the concept of the proposed information system architecture for retrieving information. This system scheme operates basing on the content of the document.  Digitized visual media: images and videos captured from real time video surveillance system require high storage capacity. This work describes the steps of indexation and content modeling for retrieving and managing information in multimedia documents databases. Keywords: big data analysis, multimedia documents, indexing, modeling, classification, content representation

Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature

Integration of Exploration and Search: A Case Study of the M3 Model

International audienceEffective support for multimedia analytics applications requires exploration and search to be integrated seamlessly into a single interaction model. Media metadata can be seen as defining a multidimensional media space, casting multimedia analytics tasks as exploration, manipulation and augmentation of that space. We present an initial case study of integrating exploration and search within this multidimensional media space. We extend the M3 model, initially proposed as a pure exploration tool, and show that it can be elegantly extended to allow searching within an exploration context and exploring within a search context. We then evaluate the suitability of relational database management systems, as representatives of today’s data management technologies, for implementing the extended M3 model. Based on our results, we finally propose some research directions for scalability of multimedia analytics

Multimedia big data computing for in-depth event analysis

While the most part of ”big data” systems target text-based analytics, multimedia data, which makes up about 2/3 of internet traffic, provide unprecedented opportunities for understanding and responding to real world situations and challenges. Multimedia Big Data Computing is the new topic that focus on all aspects of distributed computing systems that enable massive scale image and video analytics. During the course of this paper we describe BPEM (Big Picture Event Monitor), a Multimedia Big Data Computing framework that operates over streams of digital photos generated by online communities, and enables monitoring the relationship between real world events and social media user reaction in real-time. As a case example, the paper examines publicly available social media data that relate to the Mobile World Congress 2014 that has been harvested and analyzed using the described system.Peer ReviewedPostprint (author's final draft