Hierarchical Multimodal Attention for Deep Video Summarization

International audienceThe way people consume sports on TV has drastically evolved in the last years, particularly under the combined effects of the legalization of sport betting and the huge increase of sport analytics. Several companies are nowadays sending observers in the stadiums to collect live data of all the events happening on the field during the match. Those data contain meaningful information providing a very detailed description of all the actions occurring during the match to feed the coaches and staff, the fans, the viewers, and the gamblers. Exploiting all these data, sport broadcasters want to generate extra content such as match highlights, match summaries, players and teams analytics, etc., to appeal subscribers. This paper explores the problem of summarizing professional soccer matches as automatically as possible using both the aforementioned event-stream data collected from the field and the content broadcasted on TV. We have designed an architecture, introducing first (1) a Multiple Instance Learning method that takes into account the sequential dependency among events and then (2) a hierarchical multimodal attention layer that grasps the importance of each event in an action. We evaluate our approach on matches from two professional European soccer leagues, showing its capability to identify the best actions for automatic summarization by comparing with real summaries made by human operators

Profiling Actions for Sport Video Summarization: An attention signal analysis

International audienceCurrently, in broadcast companies many human operators select which actions should belong to the summary based on multiple rules they have built upon their own experience using different sources of information. These rules define the different profiles of actions of interest that help the operator to generate better customized summaries. Most of these profiles do not directly rely on broadcast video content but rather exploit metadata describing the course of the match. In this paper, we show how the signals produced by the attention layer of a recurrent neural network can be seen as a learned representation of these action profiles and provide a new tool to support operators’ work. The results in soccer matches show the capacity of our approach to transfer knowledge between datasets from different broadcasting companies, from different leagues, and the ability of the attention layer to learn meaningful action profiles

On the Variability Secrets of an Online Video Generator

International audienceWe relate an original experience concerning a popular online video service that offers to generate variants of an humorous video. To further the understanding of the generator, we have reverse engineered its general behavior, architecture, as well as its variation points and its configuration space. The reverse engineering also allows us to create a new generator and online configurator that proposes 18 variation points – instead of only 3 as in the original generator. We explain why and how we have collaborated and are collaborating with the original creators of the video generator. We also highlight how our reverse engineering work represents a threat to the original service and call for further investigating variability-aware security mechanisms

A Multi-stage deep architecture for summary generation of soccer videos

Video content is present in an ever-increasing number of fields, both scientific and commercial. Sports, particularly soccer, is one of the industries that has invested the most in the field of video analytics, due to the massive popularity of the game and the emergence of new markets. Previous state-of-the-art methods on soccer matches video summarization rely on handcrafted heuristics to generate summaries which are poorly generalizable, but these works have yet proven that multiple modalities help detect the best actions of the game. On the other hand, machine learning models with higher generalization potential have entered the field of summarization of general-purpose videos, offering several deep learning approaches. However, most of them exploit content specificities that are not appropriate for sport whole-match videos. Although video content has been for many years the main source for automatizing knowledge extraction in soccer, the data that records all the events happening on the field has become lately very important in sports analytics, since this event data provides richer context information and requires less processing. We propose a method to generate the summary of a soccer match exploiting both the audio and the event metadata. The results show that our method can detect the actions of the match, identify which of these actions should belong to the summary and then propose multiple candidate summaries which are similar enough but with relevant variability to provide different options to the final editor. Furthermore, we show the generalization capability of our work since it can transfer knowledge between datasets from different broadcasting companies, different competitions, acquired in different conditions, and corresponding to summaries of different length

A Multi-stage deep architecture for summary generation of soccer videos

Video content is present in an ever-increasing number of fields, both scientific and commercial. Sports, particularly soccer, is one of the industries that has invested the most in the field of video analytics, due to the massive popularity of the game and the emergence of new markets. Previous state-of-the-art methods on soccer matches video summarization rely on handcrafted heuristics to generate summaries which are poorly generalizable, but these works have yet proven that multiple modalities help detect the best actions of the game. On the other hand, machine learning models with higher generalization potential have entered the field of summarization of general-purpose videos, offering several deep learning approaches. However, most of them exploit content specificities that are not appropriate for sport whole-match videos. Although video content has been for many years the main source for automatizing knowledge extraction in soccer, the data that records all the events happening on the field has become lately very important in sports analytics, since this event data provides richer context information and requires less processing. We propose a method to generate the summary of a soccer match exploiting both the audio and the event metadata. The results show that our method can detect the actions of the match, identify which of these actions should belong to the summary and then propose multiple candidate summaries which are similar enough but with relevant variability to provide different options to the final editor. Furthermore, we show the generalization capability of our work since it can transfer knowledge between datasets from different broadcasting companies, different competitions, acquired in different conditions, and corresponding to summaries of different length

Revealing forms of iron in river-borne material from major tropical rivers of the Amazon Basin (Brazil).

The present study deals with the direct determination of colloidal forms of iron in river-borne solids from main rivers of the Amazon Basin. The contribution of different forms of colloidal iron have been assessed using ultrafiltration associated with various techniques including electron paramagnetic resonance spectroscopy (EPR), high resolution transmission electron microscopy (HRTEM), and micro proton-induced X ray emission analysis (μPIXE). EPR shows the presence of Fe3+ bound to organic matter (Fe3+-OM) and colloidal iron oxides. Quantitative estimate of Fe3+-OM content in colloidal matter ranges from 0.1 to 1.6 weight % of dried solids and decreases as the pH of the river increases in the range 4 to 6.8. The modeling of the field data with the Equilibrium Calculation of Speciation and Transport (ECOSAT) code demonstrates that this trend is indicative of a geochemical control resulting from the solubility equilibrium of Fe oxyhydroxide phase and Fe binding to organic matter. Combining EPR and μPIXE data quantitatively confirms the presence of colloidal iron phase (min. 35 to 65% of iron content), assuming no divalent Fe is present. In the Rio Negro, HRTEM specifies the nature of colloidal iron phase mainly as ferrihydrite particles of circa 20 to 50 Å associated with organic matter. The geochemical forms of colloidal iron differentiate the pedoclimatic regions drained by the different rivers, corresponding to different major weathering/erosion processes. Modeling allows the calculation of the speciation of iron as mineral, organic and dissolved phases in the studied rivers

Experimental and theoretical evidence for oriented aggregate crystal growth of CoO in a polyol

International audienceMonodispersed about 5 nm sized CoO crystals were prepared by forced hydrolysis of cobalt(II) acetate in diethyleneglycol (DEG) solvent. The adsorption of the solvent molecules on these primary nanocrystals caused their in-situ oriented aggregation resulting in the precipitation of textured submicrometer-sized polycrystals. X-ray diffraction, Infrared spectroscopy, Transmission Electron Microscopy and Thermogravimetry analyses coupled to ab-initio modeling were applied to understand the adsorption mechanism of the alcohol moieties and the role of the molecule-to-surface and molecule-to-molecule interactions in the crystal growth mechanism of these polycrystals. We showed that DEG moieties are mainly adsorbed at the top of the cobalt (100) surface atoms and the process does not involve the whole molecule but only one of its extreme oxygen atoms. As a consequence, adsorbed DEG molecules exhibit an extended configuration which is favorable to intermolecule hydrogen bonding from one covered nanocrystal to another. Interestingly, at high surface coverage, the energy required for DEG attachment to the crystal surface is found to be 18.6 kJ/mol per molecule, while that required for hydrogen bonding between a bearing molecule and a neighbor one is found to be 36,4 kJ/mol per molecule, meaning that the collective departure of an assembly of DEG from the surface of CoO nanocrystals is therodynamically easier, leading thus to the observed final morphology