Advanced content-based semantic scene analysis and information retrieval: the SCHEMA project

The aim of the SCHEMA Network of Excellence is to bring together a critical mass of universities, research centers, industrial partners and end users, in order to design a reference system for content-based semantic scene analysis, interpretation and understanding. Relevant research areas include: content-based multimedia analysis and automatic annotation of semantic multimedia content, combined textual and multimedia information retrieval, semantic -web, MPEG-7 and MPEG-21 standards, user interfaces and human factors. In this paper, recent advances in content-based analysis, indexing and retrieval of digital media within the SCHEMA Network are presented. These advances will be integrated in the SCHEMA module-based, expandable reference system

The aceToolbox: low-level audiovisual feature extraction for retrieval and classification

In this paper we present an overview of a software platform that has been developed within the aceMedia project, termed the aceToolbox, that provides global and local lowlevel feature extraction from audio-visual content. The toolbox is based on the MPEG-7 eXperimental Model (XM), with extensions to provide descriptor extraction from arbitrarily shaped image segments, thereby supporting local descriptors reflecting real image content. We describe the architecture of the toolbox as well as providing an overview of the descriptors supported to date. We also briefly describe the segmentation algorithm provided. We then demonstrate the usefulness of the toolbox in the context of two different content processing scenarios: similarity-based retrieval in large collections and scene-level classification of still images

Vision-Based Production of Personalized Video

In this paper we present a novel vision-based system for the automated production of personalised video souvenirs for visitors in leisure and cultural heritage venues. Visitors are visually identified and tracked through a camera network. The system produces a personalized DVD souvenir at the end of a visitor’s stay allowing visitors to relive their experiences. We analyze how we identify visitors by fusing facial and body features, how we track visitors, how the tracker recovers from failures due to occlusions, as well as how we annotate and compile the final product. Our experiments demonstrate the feasibility of the proposed approach

Simulated evaluation of faceted browsing based on feature selection

In this paper we explore the limitations of facet based browsing which uses sub-needs of an information need for querying and organising the search process in video retrieval. The underlying assumption of this approach is that the search effectiveness will be enhanced if such an approach is employed for interactive video retrieval using textual and visual features. We explore the performance bounds of a faceted system by carrying out a simulated user evaluation on TRECVid data sets, and also on the logs of a prior user experiment with the system. We first present a methodology to reduce the dimensionality of features by selecting the most important ones. Then, we discuss the simulated evaluation strategies employed in our evaluation and the effect on the use of both textual and visual features. Facets created by users are simulated by clustering video shots using textual and visual features. The experimental results of our study demonstrate that the faceted browser can potentially improve the search effectiveness

Deep active learning for suggestive segmentation of biomedical image stacks via optimisation of Dice scores and traced boundary length

Manual segmentation of stacks of 2D biomedical images (e.g., histology) is a time-consuming task which can be sped up with semi-automated techniques. In this article, we present a suggestive deep active learning framework that seeks to minimise the annotation effort required to achieve a certain level of accuracy when labelling such a stack. The framework suggests, at every iteration, a specific region of interest (ROI) in one of the images for manual delineation. Using a deep segmentation neural network and a mixed cross-entropy loss function, we propose a principled strategy to estimate class probabilities for the whole stack, conditioned on heterogeneous partial segmentations of the 2D images, as well as on weak supervision in the form of image indices that bound each ROI. Using the estimated probabilities, we propose a novel active learning criterion based on predictions for the estimated segmentation performance and delineation effort, measured with average Dice scores and total delineated boundary length, respectively, rather than common surrogates such as entropy. The query strategy suggests the ROI that is expected to maximise the ratio between performance and effort, while considering the adjacency of structures that may have already been labelled – which decrease the length of the boundary to trace. We provide quantitative results on synthetically deformed MRI scans and real histological data, showing that our framework can reduce labelling effort by up to 60–70% without compromising accuracy

The GalMer database: Galaxy Mergers in the Virtual Observatory

We present the GalMer database, a library of galaxy merger simulations, made available to users through tools compatible with the Virtual Observatory (VO) standards adapted specially for this theoretical database. To investigate the physics of galaxy formation through hierarchical merging, it is necessary to simulate galaxy interactions varying a large number of parameters: morphological types, mass ratios, orbital configurations, etc. On one side, these simulations have to be run in a cosmological context, able to provide a large number of galaxy pairs, with boundary conditions given by the large-scale simulations, on the other side the resolution has to be high enough at galaxy scales, to provide realistic physics. The GalMer database is a library of thousands simulations of galaxy mergers at moderate spatial resolution and it is a compromise between the diversity of initial conditions and the details of underlying physics. We provide all coordinates and data of simulated particles in FITS binary tables. The main advantages of the database are VO access interfaces and value-added services which allow users to compare the results of the simulations directly to observations: stellar population modelling, dust extinction, spectra, images, visualisation using dedicated VO tools. The GalMer value-added services can be used as virtual telescope producing broadband images, 1D spectra, 3D spectral datacubes, thus making our database oriented towards the usage by observers. We present several examples of the GalMer database scientific usage obtained from the analysis of simulations and modelling their stellar population properties, including: (1) studies of the star formation efficiency in interactions; (2) creation of old counter-rotating components; (3) reshaping metallicity profiles in elliptical galaxies; (4) orbital to internal angular momentum transfer; (5) reproducing observed colour bimodality of galaxies.Comment: 15 pages, 11 figures, 10 tables accepted to A&A. Visualisation of GalMer simulations, access to snapshot files and value-added tools described in the paper are available at http://galmer.obspm.fr