Active Frame, Location, and Detector Selection for Automated and Manual Video Annotation

We describe an information-driven active selection ap-proach to determine which detectors to deploy at which lo-cation in which frame of a video to minimize semantic class label uncertainty at every pixel, with the smallest computa-tional cost that ensures a given uncertainty bound. We show minimal performance reduction compared to a “paragon” algorithm running all detectors at all locations in all frames, at a small fraction of the computational cost. Our method can handle uncertainty in the labeling mechanism, so it can handle both “oracles ” (manual annotation) or noisy detec-tors (automated annotation). 1

Fully automated segmentation and tracking of the intima media thickness in ultrasound video sequences of the common carotid artery

Abstract—The robust identification and measurement of the intima media thickness (IMT) has a high clinical relevance because it represents one of the most precise predictors used in the assessment of potential future cardiovascular events. To facilitate the analysis of arterial wall thickening in serial clinical investigations, in this paper we have developed a novel fully automatic algorithm for the segmentation, measurement, and tracking of the intima media complex (IMC) in B-mode ultrasound video sequences. The proposed algorithm entails a two-stage image analysis process that initially addresses the segmentation of the IMC in the first frame of the ultrasound video sequence using a model-based approach; in the second step, a novel customized tracking procedure is applied to robustly detect the IMC in the subsequent frames. For the video tracking procedure, we introduce a spatially coherent algorithm called adaptive normalized correlation that prevents the tracking process from converging to wrong arterial interfaces. This represents the main contribution of this paper and was developed to deal with inconsistencies in the appearance of the IMC over the cardiac cycle. The quantitative evaluation has been carried out on 40 ultrasound video sequences of the common carotid artery (CCA) by comparing the results returned by the developed algorithm with respect to ground truth data that has been manually annotated by clinical experts. The measured IMTmean ± standard deviation recorded by the proposed algorithm is 0.60 mm ± 0.10, with a mean coefficient of variation (CV) of 2.05%, whereas the corresponding result obtained for the manually annotated ground truth data is 0.60 mm ± 0.11 with a mean CV equal to 5.60%. The numerical results reported in this paper indicate that the proposed algorithm is able to correctly segment and track the IMC in ultrasound CCA video sequences, and we were encouraged by the stability of our technique when applied to data captured under different imaging conditions. Future clinical studies will focus on the evaluation of patients that are affected by advanced cardiovascular conditions such as focal thickening and arterial plaques

Automatic Neuron Detection in Calcium Imaging Data Using Convolutional Networks

Calcium imaging is an important technique for monitoring the activity of thousands of neurons simultaneously. As calcium imaging datasets grow in size, automated detection of individual neurons is becoming important. Here we apply a supervised learning approach to this problem and show that convolutional networks can achieve near-human accuracy and superhuman speed. Accuracy is superior to the popular PCA/ICA method based on precision and recall relative to ground truth annotation by a human expert. These results suggest that convolutional networks are an efficient and flexible tool for the analysis of large-scale calcium imaging data.Comment: 9 pages, 5 figures, 2 ancillary files; minor changes for camera-ready version. appears in Advances in Neural Information Processing Systems 29 (NIPS 2016

Spott : on-the-spot e-commerce for television using deep learning-based video analysis techniques

Spott is an innovative second screen mobile multimedia application which offers viewers relevant information on objects (e.g., clothing, furniture, food) they see and like on their television screens. The application enables interaction between TV audiences and brands, so producers and advertisers can offer potential consumers tailored promotions, e-shop items, and/or free samples. In line with the current views on innovation management, the technological excellence of the Spott application is coupled with iterative user involvement throughout the entire development process. This article discusses both of these aspects and how they impact each other. First, we focus on the technological building blocks that facilitate the (semi-) automatic interactive tagging process of objects in the video streams. The majority of these building blocks extensively make use of novel and state-of-the-art deep learning concepts and methodologies. We show how these deep learning based video analysis techniques facilitate video summarization, semantic keyframe clustering, and (similar) object retrieval. Secondly, we provide insights in user tests that have been performed to evaluate and optimize the application's user experience. The lessons learned from these open field tests have already been an essential input in the technology development and will further shape the future modifications to the Spott application

Search Tracker: Human-derived object tracking in-the-wild through large-scale search and retrieval

Humans use context and scene knowledge to easily localize moving objects in conditions of complex illumination changes, scene clutter and occlusions. In this paper, we present a method to leverage human knowledge in the form of annotated video libraries in a novel search and retrieval based setting to track objects in unseen video sequences. For every video sequence, a document that represents motion information is generated. Documents of the unseen video are queried against the library at multiple scales to find videos with similar motion characteristics. This provides us with coarse localization of objects in the unseen video. We further adapt these retrieved object locations to the new video using an efficient warping scheme. The proposed method is validated on in-the-wild video surveillance datasets where we outperform state-of-the-art appearance-based trackers. We also introduce a new challenging dataset with complex object appearance changes.Comment: Under review with the IEEE Transactions on Circuits and Systems for Video Technolog

Automatic annotation for weakly supervised learning of detectors

PhDObject detection in images and action detection in videos are among the most widely studied computer vision problems, with applications in consumer photography, surveillance, and automatic media tagging. Typically, these standard detectors are fully supervised, that is they require a large body of training data where the locations of the objects/actions in images/videos have been manually annotated. With the emergence of digital media, and the rise of high-speed internet, raw images and video are available for little to no cost. However, the manual annotation of object and action locations remains tedious, slow, and expensive. As a result there has been a great interest in training detectors with weak supervision where only the presence or absence of object/action in image/video is needed, not the location. This thesis presents approaches for weakly supervised learning of object/action detectors with a focus on automatically annotating object and action locations in images/videos using only binary weak labels indicating the presence or absence of object/action in images/videos. First, a framework for weakly supervised learning of object detectors in images is presented. In the proposed approach, a variation of multiple instance learning (MIL) technique for automatically annotating object locations in weakly labelled data is presented which, unlike existing approaches, uses inter-class and intra-class cue fusion to obtain the initial annotation. The initial annotation is then used to start an iterative process in which standard object detectors are used to refine the location annotation. Finally, to ensure that the iterative training of detectors do not drift from the object of interest, a scheme for detecting model drift is also presented. Furthermore, unlike most other methods, our weakly supervised approach is evaluated on data without manual pose (object orientation) annotation. Second, an analysis of the initial annotation of objects, using inter-class and intra-class cues, is carried out. From the analysis, a new method based on negative mining (NegMine) is presented for the initial annotation of both object and action data. The NegMine based approach is a much simpler formulation using only inter-class measure and requires no complex combinatorial optimisation but can still meet or outperform existing approaches including the previously pre3 sented inter-intra class cue fusion approach. Furthermore, NegMine can be fused with existing approaches to boost their performance. Finally, the thesis will take a step back and look at the use of generic object detectors as prior knowledge in weakly supervised learning of object detectors. These generic object detectors are typically based on sampling saliency maps that indicate if a pixel belongs to the background or foreground. A new approach to generating saliency maps is presented that, unlike existing approaches, looks beyond the current image of interest and into images similar to the current image. We show that our generic object proposal method can be used by itself to annotate the weakly labelled object data with surprisingly high accuracy

Modeling cell migration in quantitative image analysis

Tese de mestrado em Tecnologias da Informação aplicadas às Ciências Biológicas e Médicas, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2012All biological phenomena are dynamic and movement is an essential function in cellular systems but their regulation, characteristics and physiological meaning are not fully known. Measurement of the cell movements provides quantitative information that is inevitable for understanding the cellular system. Cell migration is a field of intense current research generating high amounts of image data that need to be quantitatively analyzed with efficiency, consistency and completeness. To accomplish, computerized motion analysis is rapidly becoming a requisite. Since all the existing algorithms for these purposes are often not robust, effective and optimal enough to yield satisfactory results, new and alternative methods must be developed. The aim of this work is to find and develop an alternative to the tracking of individual cells in order to, visualize, characterize and quantify the migration characteristics of cell population. This alternative comprises the implementation of a simple and automated algorithm to obtain qualitative and quantitative information from image sequences of cell migration in a fast, easy and inexpensive computationally way. After an extensive literature review, it became clear that all the methodologies and approaches employed to make the quantitative analysis of cell migration only presented solutions that involved object tracking. And the new method developed estimates the probability density functions for cell migration and was implemented as a plugin (Migration) for ImageJ, as cross platform open source application. In the evaluation of the developed algorithm was taken in to account his applicability, efficiency, consistency, completeness and validity. It can be used to in image sequences to extract information regarding the distribution of the future positions of all particles in a determined time point in the future and is quick when is executing. The results obtained with this method were satisfactory. Comparing to existing approaches to study the cell migration this method adds an improvement, it can deal with complex situation, such as overlapping of particles or other occlusions.Todos os fenómenos biológicos são dinâmicos e o movimento é uma função essencial nos sistemas celulares, mas a sua regulação, características e significado fisiológico não são totalmente conhecidos. A medição dos movimentos das células providencia informação quantitativa para compreender o sistema celular. A migração de células é um campo de intensa investigação gerando grandes quantidades de dados que necessitam de ser quantitativamente analisados com eficiência, consistência e de maneira completa. Para tal, a análise do movimento através dos sistemas de informação está a tornar-se cada vez mais num requisito. Dado que os algoritmos disponíveis para este propósito não são muitas vezes robustos, eficientes e óptimos para proporcionarem resultados satisfatórios, métodos alternativos devem ser desenvolvidos e implementados. O objectivo deste trabalho é encontrar e desenvolver uma alternativa para o tracking de células de modo a se visualizar, caracterizar e quantificar a migração de células. Esta alternativa requer a implementação de um algoritmo simples e automático para obter a informação, quer qualitativa, quer quantitativa de um vídeo, com imagens da migração de células, de um modo rápido e fácil. Depois de uma revisão bibliográfica extensa, verificou-se que todos os métodos implementados para fazer a análise quantitativa da migração de células eram soluções de tracking de partículas. O novo método aqui desenvolvido estima as funções de densidade de probabilidade para a migração de células e foi implementado como um plugin (Migration) para o ImageJ. A avaliação do algoritmo desenvolvido teve em conta a sua aplicabilidade, eficiência, consistência e validade. Pode ser usado em vídeos e extrair informação relativa à estimação da distribuição das posições de todas as partículas num determinado momento no tempo, executando de maneira rápida. Todos os resultados obtidos com este novo método são satisfatórios. Comparando com as abordagens conhecidas da literatura, este método apresenta uma melhoria, pode lidar com situações complexas, tais como sobreposição de partículas e outras oclusões