Extracting textual overlays from social media videos using neural networks

Textual overlays are often used in social media videos as people who watch them without the sound would otherwise miss essential information conveyed in the audio stream. This is why extraction of those overlays can serve as an important meta-data source, e.g. for content classification or retrieval tasks. In this work, we present a robust method for extracting textual overlays from videos that builds up on multiple neural network architectures. The proposed solution relies on several processing steps: keyframe extraction, text detection and text recognition. The main component of our system, i.e. the text recognition module, is inspired by a convolutional recurrent neural network architecture and we improve its performance using synthetically generated dataset of over 600,000 images with text prepared by authors specifically for this task. We also develop a filtering method that reduces the amount of overlapping text phrases using Levenshtein distance and further boosts system's performance. The final accuracy of our solution reaches over 80A% and is au pair with state-of-the-art methods.Comment: International Conference on Computer Vision and Graphics (ICCVG) 201

AMBIT: Semantic Engine Foundations for Knowledge Management in Context-dependent Applications

Context-aware application and services proposing potentially useful information to users are more and more widespread; however, their actual usefulness is often limited by the “syntactical” notion of context they adopt. The recently started AMBIT project aims to provide a general software architecture for developing semantic-based context-aware tools in a number of vertical case study applications. In this paper, we focus on the knowledge management foundations we are laying for the Semantic Engine of the AMBIT architecture. The proposed semantic analysis and similarity techniques: (a) exploit the textual information deeply characterizing both users and the information to be retrieved; (b) overcome the limits of syntactic methods by leveraging on the strengths of both classic information retrieval and knowledge-based analysis and classification, ultimately proposing information relevant to the user interests. The experimental evaluation of a preliminary implementation in an actual “cultural territorial enhancement” scenario already shows promising results

Modeling social information skills

In a modern economy, the most important resource consists in\ud human talent: competent, knowledgeable people. Locating the right person for\ud the task is often a prerequisite to complex problem-solving, and experienced\ud professionals possess the social skills required to find appropriate human\ud expertise. These skills can be reproduced more and more with specific\ud computer software, an approach defining the new field of social information\ud retrieval. We will analyze the social skills involved and show how to model\ud them on computer. Current methods will be described, notably information\ud retrieval techniques and social network theory. A generic architecture and its\ud functions will be outlined and compared with recent work. We will try in this\ud way to estimate the perspectives of this recent domain

A framework for interrogating social media images to reveal an emergent archive of war

The visual image has long been central to how war is seen, contested and legitimised, remembered and forgotten. Archives are pivotal to these ends as is their ownership and access, from state and other official repositories through to the countless photographs scattered and hidden from a collective understanding of what war looks like in individual collections and dusty attics. With the advent and rapid development of social media, however, the amateur and the professional, the illicit and the sanctioned, the personal and the official, and the past and the present, all seem to inhabit the same connected and chaotic space.However, to even begin to render intelligible the complexity, scale and volume of what war looks like in social media archives is a considerable task, given the limitations of any traditional human-based method of collection and analysis. We thus propose the production of a series of ‘snapshots’, using computer-aided extraction and identification techniques to try to offer an experimental way in to conceiving a new imaginary of war. We were particularly interested in testing to see if twentieth century wars, obviously initially captured via pre-digital means, had become more ‘settled’ over time in terms of their remediated presence today through their visual representations and connections on social media, compared with wars fought in digital media ecologies (i.e. those fought and initially represented amidst the volume and pervasiveness of social media images).To this end, we developed a framework for automatically extracting and analysing war images that appear in social media, using both the features of the images themselves, and the text and metadata associated with each image. The framework utilises a workflow comprising four core stages: (1) information retrieval, (2) data pre-processing, (3) feature extraction, and (4) machine learning. Our corpus was drawn from the social media platforms Facebook and Flickr