Content-based video indexing for the support of digital library search

Presents a digital library search engine that combines efforts of the AMIS and DMW research projects, each covering significant parts of the problem of finding the required information in an enormous mass of data. The most important contributions of our work are the following: (1) We demonstrate a flexible solution for the extraction and querying of meta-data from multimedia documents in general. (2) Scalability and efficiency support are illustrated for full-text indexing and retrieval. (3) We show how, for a more limited domain, like an intranet, conceptual modelling can offer additional and more powerful query facilities. (4) In the limited domain case, we demonstrate how domain knowledge can be used to interpret low-level features into semantic content. In this short description, we focus on the first and fourth item

Semantic analysis of field sports video using a petri-net of audio-visual concepts

The most common approach to automatic summarisation and highlight detection in sports video is to train an automatic classifier to detect semantic highlights based on occurrences of low-level features such as action replays, excited commentators or changes in a scoreboard. We propose an alternative approach based on the detection of perception concepts (PCs) and the construction of Petri-Nets which can be used for both semantic description and event detection within sports videos. Low-level algorithms for the detection of perception concepts using visual, aural and motion characteristics are proposed, and a series of Petri-Nets composed of perception concepts is formally defined to describe video content. We call this a Perception Concept Network-Petri Net (PCN-PN) model. Using PCN-PNs, personalized high-level semantic descriptions of video highlights can be facilitated and queries on high-level semantics can be achieved. A particular strength of this framework is that we can easily build semantic detectors based on PCN-PNs to search within sports videos and locate interesting events. Experimental results based on recorded sports video data across three types of sports games (soccer, basketball and rugby), and each from multiple broadcasters, are used to illustrate the potential of this framework

High-level feature detection from video in TRECVid: a 5-year retrospective of achievements

Successful and effective content-based access to digital video requires fast, accurate and scalable methods to determine the video content automatically. A variety of contemporary approaches to this rely on text taken from speech within the video, or on matching one video frame against others using low-level characteristics like colour, texture, or shapes, or on determining and matching objects appearing within the video. Possibly the most important technique, however, is one which determines the presence or absence of a high-level or semantic feature, within a video clip or shot. By utilizing dozens, hundreds or even thousands of such semantic features we can support many kinds of content-based video navigation. Critically however, this depends on being able to determine whether each feature is or is not present in a video clip. The last 5 years have seen much progress in the development of techniques to determine the presence of semantic features within video. This progress can be tracked in the annual TRECVid benchmarking activity where dozens of research groups measure the effectiveness of their techniques on common data and using an open, metrics-based approach. In this chapter we summarise the work done on the TRECVid high-level feature task, showing the progress made year-on-year. This provides a fairly comprehensive statement on where the state-of-the-art is regarding this important task, not just for one research group or for one approach, but across the spectrum. We then use this past and on-going work as a basis for highlighting the trends that are emerging in this area, and the questions which remain to be addressed before we can achieve large-scale, fast and reliable high-level feature detection on video

On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances and Million-AID

The past years have witnessed great progress on remote sensing (RS) image interpretation and its wide applications. With RS images becoming more accessible than ever before, there is an increasing demand for the automatic interpretation of these images. In this context, the benchmark datasets serve as essential prerequisites for developing and testing intelligent interpretation algorithms. After reviewing existing benchmark datasets in the research community of RS image interpretation, this article discusses the problem of how to efficiently prepare a suitable benchmark dataset for RS image interpretation. Specifically, we first analyze the current challenges of developing intelligent algorithms for RS image interpretation with bibliometric investigations. We then present the general guidances on creating benchmark datasets in efficient manners. Following the presented guidances, we also provide an example on building RS image dataset, i.e., Million-AID, a new large-scale benchmark dataset containing a million instances for RS image scene classification. Several challenges and perspectives in RS image annotation are finally discussed to facilitate the research in benchmark dataset construction. We do hope this paper will provide the RS community an overall perspective on constructing large-scale and practical image datasets for further research, especially data-driven ones

Data-Driven Analytics for Decision Making in Game Sports

Performance analysis and good decision making in sports is important to maximize chances of winning. Over the last years the amount and quality of data which is available for the analysis has increased enormously due to technical developments like, e.g., of sensor technologies or computer vision technology. However, the data-driven analysis of athletes and team performances is very demanding. One reason is the so called semantic gap of sports analytics. This means that the concepts of coaches are seldomly represented in the data for the analysis. Furthermore, sports in general and game sports in particular present a huge challenge due to its dynamic characteristics and the multi-factorial influences on an athlete’s performance like, e.g., the numerous interaction processes during a match. This requires different types of analyses like, e.g., qualitative analyses and thus anecdotal descriptions of performances up to quantitative analyses with which performances can be described through statistics and indicators. Additionally, coaches and analysts have to work under an enormous time pressure and decisions have to be made very quickly. In order to facilitate the demanding task of game sports analysts and coaches we present a generic approach how to conceptualize and design a Data Analytics System (DAS) for an efficient support of the decision making processes in practice. We first introduce a theoretical model and present a way how to bridge the semantic gap of sports analytics. This ensures that DASs will provide relevant information for the decision makers. Moreover, we show that DASs need to combine qualitative and quantitative analyses as well as visualizations. Additionally, we introduce different query types which are required for a holistic retrieval of sports data. We furthermore show a model for the user-centered planning and designing of the User Experience (UX) of a DAS. Having introduced the theoretical basis we present SportSense, a DAS to support decision making in game sports. Its generic architecture allows a fast adaptation to the individual characteristics and requirements of different game sports. SportSense is novel with respect to the fact that it unites raw data, event data, and video data. Furthermore, it supports different query types including an intuitive sketch-based retrieval and seamlessly combines qualitative and quantitative analyses as well as several data visualization options. Moreover, we present the two applications SportSense Football and SportSense Ice Hockey which contain sport-specific concepts and cover (high-level) tactical analyses