TagBook: A Semantic Video Representation without Supervision for Event Detection

We consider the problem of event detection in video for scenarios where only few, or even zero examples are available for training. For this challenging setting, the prevailing solutions in the literature rely on a semantic video representation obtained from thousands of pre-trained concept detectors. Different from existing work, we propose a new semantic video representation that is based on freely available social tagged videos only, without the need for training any intermediate concept detectors. We introduce a simple algorithm that propagates tags from a video's nearest neighbors, similar in spirit to the ones used for image retrieval, but redesign it for video event detection by including video source set refinement and varying the video tag assignment. We call our approach TagBook and study its construction, descriptiveness and detection performance on the TRECVID 2013 and 2014 multimedia event detection datasets and the Columbia Consumer Video dataset. Despite its simple nature, the proposed TagBook video representation is remarkably effective for few-example and zero-example event detection, even outperforming very recent state-of-the-art alternatives building on supervised representations.Comment: accepted for publication as a regular paper in the IEEE Transactions on Multimedi

Video content analysis and retrieval system using video storytelling and indexing techniques

Videos are used often for communicating ideas, concepts, experience, and situations, because of the significant advances made in video communication technology. The social media platforms enhanced the video usage expeditiously. At, present, recognition of a video is done, using the metadata like video title, video descriptions, and video thumbnails. There are situations like video searcher requires only a video clip on a specific topic from a long video. This paper proposes a novel methodology for the analysis of video content and using video storytelling and indexing techniques for the retrieval of the intended video clip from a long duration video. Video storytelling technique is used for video content analysis and to produce a description of the video. The video description thus created is used for preparation of an index using wormhole algorithm, guarantying the search of a keyword of definite length L, within the minimum worst-case time. This video index can be used by video searching algorithm to retrieve the relevant part of the video by virtue of the frequency of the word in the keyword search of the video index. Instead of downloading and transferring a whole video, the user can download or transfer the specifically necessary video clip. The network constraints associated with the transfer of videos are considerably addressed

Visualização de padrões temporais cíclicos em estudos de fenologia

Orientadores: Ricardo da Silva Torres, Leonor Patrícia Cerdeira MorellatoTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Em diversas aplicações, grandes volumes de dados multidimensionais têm sido gerados continuamente ao longo do tempo. Uma abordagem adequada para lidar com estas coleções consiste no uso de métodos de visualização de informação, a partir dos quais padrões de interesse podem ser identificados, possibilitando o entendimento de fenômenos temporais complexos. De fato, em diversos domínios, o desenvolvimento de ferramentas adequadas para apoiar análises complexas, por exemplo, aquelas baseadas na identificação de padrões de mudanças ou correlações existentes entre múltiplas variáveis ao longo do tempo é de suma importância. Em estudos de fenologia, por exemplo, especialistas observam as mudanças que ocorrem ao longo da vida de plantas e animais e investigam qual é a relação entre essas mudanças com variáveis ambientais. Neste cenário, especialistas em fenologia cada vez mais precisam de ferramentas para, adequadamente, visualizar séries temporais longas, com muitas variáveis e de diferentes tipos (por exemplo, texto e imagem), assim como identificar padrões temporais cíclicos. Embora diversas abordagens tenham sido propostas para visualizar dados que variam ao longo do tempo, muitas não são apropriadas ou aplicáveis para dados de fenologia, porque não são capazes de: (i) lidar com séries temporais longas, com muitas variáveis de diferentes tipos de dados e de uma ou mais dimensões; e (ii) permitir a identificação de padrões temporais cíclicos e drivers ambientais associados. Este trabalho aborda essas questões a partir da proposta de duas novas abordagens para apoiar a análise e visualização de dados temporais multidimensionais. Nossa primeira proposta combina estruturas visuais radiais com ritmos visuais. As estruturas radiais são usadas para fornecer informação contextual sobre fenômenos cíclicos, enquanto que o ritmo visual é usado para sumarizar séries temporais longas em representações compactas. Nós desenvolvemos, avaliamos e validamos nossa proposta com especialistas em fenologia em tarefas relacionadas à visualização de dados de observação direta da fenologia de plantas em nível tanto de indivíduos quanto de espécies. Nós também validamos a proposta usando dados temporais relacionados a imagens obtidas de sistemas de monitoramento de vegetação próxima à superfície. Nossa segunda abordagem é uma nova representação baseada em imagem, chamada Change Frequency Heatmap (CFH), usada para codificar mudanças temporais de dados numéricos multivariados. O método calcula histogramas de padrões de mudanças observados em sucessivos instantes de tempo. Nós validamos o uso do CFH a partir da criação de uma ferramenta de caracterização de mudanças no ciclo de vida de plantas de múltiplos indivíduos e espécies ao longo do tempo. Nós demonstramos o potencial do CFH para ajudar na identificação visual de padrões de mudanças temporais complexas, especialmente na identificação de variações entre indivíduos em estudos relacionados à fenologia de plantasAbstract: In several applications, large volumes of multidimensional data have been generated continuously over time. One suitable approach for handling those collections in a meaningful way consists in the use of information visualization methods, based on which patterns of interest can be identified, triggering the understanding of complex temporal phenomena. In fact, in several domains, the development of appropriate tools for supporting complex analysis based, for example, on the identification of change patterns in temporal data or existing correlations, over time, among multiple variables, is of paramount importance. In phenology studies, for instance, phenologists observe changes in the development of plants and animals throughout their lives and investigate what is the relationship between these changes with environmental changes. Therefore, phenologists increasingly need tools for visualizing appropriately long-term series with many variables of different data types, as well as for identifying cyclical temporal patterns. Although several approaches have been proposed to visualize data varying over time, most of them are not appropriate or applicable to phenology data, because they are not able: (i) to handle long-term series with many variables of different data types and one or more dimensions and (ii) to support the identification of cyclical temporal patterns and associated environmental drivers. This work addresses these shortcomings by presenting two new approaches to support the analysis and visualization of multidimensional temporal data. Our first proposal to visualize phenological data combines radial visual structures along with visual rhythms. Radial visual structures are used to provide contextual insights regarding cyclical phenomena, while the visual rhythm encoding is used to summarize long-term time series into compact representations. We developed, evaluated, and validated our proposal with phenology experts using plant phenology direct observational data both at individuals and species levels. Also we validated the proposal using image-related temporal data obtained from near-surface vegetation monitoring systems. Our second approach is a novel image-based representation, named Change Frequency Heatmap (CFH), used to encode temporal changes of multivariate numerical data. The method computes histograms of change patterns observed at successive timestamps. We validated the use of CFHs through the creation of a temporal change characterization tool to support complex plant phenology analysis, concerning the characterization of plant life cycle changes of multiple individuals and species over time. We demonstrated the potential of CFH to support visual identification of complex temporal change patterns, especially to decipher interindividual variations in plant phenologyDoutoradoCiência da ComputaçãoDoutora em Ciência da Computação162312/2015-62013/501550-0CNPQCAPESFAPES

Learning Visual Importance for Graphic Designs and Data Visualizations

Knowing where people look and click on visual designs can provide clues about how the designs are perceived, and where the most important or relevant content lies. The most important content of a visual design can be used for effective summarization or to facilitate retrieval from a database. We present automated models that predict the relative importance of different elements in data visualizations and graphic designs. Our models are neural networks trained on human clicks and importance annotations on hundreds of designs. We collected a new dataset of crowdsourced importance, and analyzed the predictions of our models with respect to ground truth importance and human eye movements. We demonstrate how such predictions of importance can be used for automatic design retargeting and thumbnailing. User studies with hundreds of MTurk participants validate that, with limited post-processing, our importance-driven applications are on par with, or outperform, current state-of-the-art methods, including natural image saliency. We also provide a demonstration of how our importance predictions can be built into interactive design tools to offer immediate feedback during the design process

Data-driven interaction techniques for improving navigation of educational videos

With an unprecedented scale of learners watching educational videos on online platforms such as MOOCs and YouTube, there is an opportunity to incorporate data generated from their interactions into the design of novel video interaction techniques. Interaction data has the potential to help not only instructors to improve their videos, but also to enrich the learning experience of educational video watchers. This paper explores the design space of data-driven interaction techniques for educational video navigation. We introduce a set of techniques that augment existing video interface widgets, including: a 2D video timeline with an embedded visualization of collective navigation traces; dynamic and non-linear timeline scrubbing; data-enhanced transcript search and keyword summary; automatic display of relevant still frames next to the video; and a visual summary representing points with high learner activity. To evaluate the feasibility of the techniques, we ran a laboratory user study with simulated learning tasks. Participants rated watching lecture videos with interaction data to be efficient and useful in completing the tasks. However, no significant differences were found in task performance, suggesting that interaction data may not always align with moment-by-moment information needs during the tasks.Engineering and Applied Science