Segmentation-based mesh design for motion estimation

Dans la plupart des codec vidéo standard, l'estimation des mouvements entre deux images se fait généralement par l'algorithme de concordance des blocs ou encore BMA pour « Block Matching Algorithm ». BMA permet de représenter l'évolution du contenu des images en décomposant normalement une image par blocs 2D en mouvement translationnel. Cette technique de prédiction conduit habituellement à de sévères distorsions de 1'artefact de bloc lorsque Ie mouvement est important. De plus, la décomposition systématique en blocs réguliers ne dent pas compte nullement du contenu de l'image. Certains paramètres associes aux blocs, mais inutiles, doivent être transmis; ce qui résulte d'une augmentation de débit de transmission. Pour paillier a ces défauts de BMA, on considère les deux objectifs importants dans Ie codage vidéo, qui sont de recevoir une bonne qualité d'une part et de réduire la transmission a très bas débit d'autre part. Dans Ie but de combiner les deux exigences quasi contradictoires, il est nécessaire d'utiliser une technique de compensation de mouvement qui donne, comme transformation, de bonnes caractéristiques subjectives et requiert uniquement, pour la transmission, l'information de mouvement. Ce mémoire propose une technique de compensation de mouvement en concevant des mailles 2D triangulaires a partir d'une segmentation de l'image. La décomposition des mailles est construite a partir des nœuds repartis irrégulièrement Ie long des contours dans l'image. La décomposition résultant est ainsi basée sur Ie contenu de l'image. De plus, étant donné la même méthode de sélection des nœuds appliquée à l'encodage et au décodage, la seule information requise est leurs vecteurs de mouvement et un très bas débit de transmission peut ainsi être réalise. Notre approche, comparée avec BMA, améliore à la fois la qualité subjective et objective avec beaucoup moins d'informations de mouvement. Dans la premier chapitre, une introduction au projet sera présentée. Dans Ie deuxième chapitre, on analysera quelques techniques de compression dans les codec standard et, surtout, la populaire BMA et ses défauts. Dans Ie troisième chapitre, notre algorithme propose et appelé la conception active des mailles a base de segmentation, sera discute en détail. Ensuite, les estimation et compensation de mouvement seront décrites dans Ie chapitre 4. Finalement, au chapitre 5, les résultats de simulation et la conclusion seront présentés.Abstract: In most video compression standards today, the generally accepted method for temporal prediction is motion compensation using block matching algorithm (BMA). BMA represents the scene content evolution with 2-D rigid translational moving blocks. This kind of predictive scheme usually leads to distortions such as block artefacts especially when the motion is important. The two most important aims in video coding are to receive a good quality on one hand and a low bit-rate on the other. This thesis proposes a motion compensation scheme using segmentation-based 2-D triangular mesh design method. The mesh is constructed by irregularly spread nodal points selected along image contour. Based on this, the generated mesh is, to a great extent, image content based. Moreover, the nodes are selected with the same method on the encoder and decoder sides, so that the only information that has to be transmitted are their motion vectors, and thus very low bit-rate can be achieved. Compared with BMA, our approach could improve subjective and objective quality with much less motion information."--Résumé abrégé par UM

Segmentation of Moving Object with Uncovered Background, Temporary Poses and GMOB

AbstractVideo has to be segmented into objects for content-based processing. A number of video object segmentation algorithms have been proposed such as semiautomatic and automatic. Semiautomatic methods adds burden to users and also not suitable for some applications. Automatic segmentation systems are still a challenge, although they are required by many applications. The proposed work aims at contributing to identify the gaps that are present in the current segmentation system and also to give the possible solutions to overcome those gaps so that the accurate and efficient video segmentation system can be developed. The proposed system aims to resolve the issue of uncovered background, Temporary poses and Global motion of background

Object-based video representations: shape compression and object segmentation

Object-based video representations are considered to be useful for easing the process of multimedia content production and enhancing user interactivity in multimedia productions. Object-based video presents several new technical challenges, however. Firstly, as with conventional video representations, compression of the video data is a requirement. For object-based representations, it is necessary to compress the shape of each video object as it moves in time. This amounts to the compression of moving binary images. This is achieved by the use of a technique called context-based arithmetic encoding. The technique is utilised by applying it to rectangular pixel blocks and as such it is consistent with the standard tools of video compression. The blockbased application also facilitates well the exploitation of temporal redundancy in the sequence of binary shapes. For the first time, context-based arithmetic encoding is used in conjunction with motion compensation to provide inter-frame compression. The method, described in this thesis, has been thoroughly tested throughout the MPEG-4 core experiment process and due to favourable results, it has been adopted as part of the MPEG-4 video standard. The second challenge lies in the acquisition of the video objects. Under normal conditions, a video sequence is captured as a sequence of frames and there is no inherent information about what objects are in the sequence, not to mention information relating to the shape of each object. Some means for segmenting semantic objects from general video sequences is required. For this purpose, several image analysis tools may be of help and in particular, it is believed that video object tracking algorithms will be important. A new tracking algorithm is developed based on piecewise polynomial motion representations and statistical estimation tools, e.g. the expectationmaximisation method and the minimum description length principle

Semi-automatic video object segmentation for multimedia applications

A semi-automatic video object segmentation tool is presented for segmenting both still pictures and image sequences. The approach comprises both automatic segmentation algorithms and manual user interaction. The still image segmentation component is comprised of a conventional spatial segmentation algorithm (Recursive Shortest Spanning Tree (RSST)), a hierarchical segmentation representation method (Binary Partition Tree (BPT)), and user interaction. An initial segmentation partition of homogeneous regions is created using RSST. The BPT technique is then used to merge these regions and hierarchically represent the segmentation in a binary tree. The semantic objects are then manually built by selectively clicking on image regions. A video object-tracking component enables image sequence segmentation, and this subsystem is based on motion estimation, spatial segmentation, object projection, region classification, and user interaction. The motion between the previous frame and the current frame is estimated, and the previous object is then projected onto the current partition. A region classification technique is used to determine which regions in the current partition belong to the projected object. User interaction is allowed for object re-initialisation when the segmentation results become inaccurate. The combination of all these components enables offline video sequence segmentation. The results presented on standard test sequences illustrate the potential use of this system for object-based coding and representation of multimedia

Video coding for compression and content-based functionality

The lifetime of this research project has seen two dramatic developments in the area of digital video coding. The first has been the progress of compression research leading to a factor of two improvement over existing standards, much wider deployment possibilities and the development of the new international ITU-T Recommendation H.263. The second has been a radical change in the approach to video content production with the introduction of the content-based coding concept and the addition of scene composition information to the encoded bit-stream. Content-based coding is central to the latest international standards efforts from the ISO/IEC MPEG working group. This thesis reports on extensions to existing compression techniques exploiting a priori knowledge about scene content. Existing, standardised, block-based compression coding techniques were extended with work on arithmetic entropy coding and intra-block prediction. These both form part of the H.263 and MPEG-4 specifications respectively. Object-based coding techniques were developed within a collaborative simulation model, known as SIMOC, then extended with ideas on grid motion vector modelling and vector accuracy confidence estimation. An improved confidence measure for encouraging motion smoothness is proposed. Object-based coding ideas, with those from other model and layer-based coding approaches, influenced the development of content-based coding within MPEG-4. This standard made considerable progress in this newly adopted content based video coding field defining normative techniques for arbitrary shape and texture coding. The means to generate this information, the analysis problem, for the content to be coded was intentionally not specified. Further research work in this area concentrated on video segmentation and analysis techniques to exploit the benefits of content based coding for generic frame based video. The work reported here introduces the use of a clustering algorithm on raw data features for providing initial segmentation of video data and subsequent tracking of those image regions through video sequences. Collaborative video analysis frameworks from COST 21 l qual and MPEG-4, combining results from many other segmentation schemes, are also introduced

Video object segmentation for future multimedia applications

An efficient representation of two-dimensional visual objects is specified by an emerging audiovisual compression standard known as MPEG-4. It incorporates the advantages of segmentation-based video compression (whereby objects are encoded independently, facilitating content-based functionalities), and also the advantages of more traditional block-based approaches (such as low delay and compression efficiency). What is not specified, however, is the method of extracting semantic objects from a scene corresponding to a video segmentation task. An accurate, robust and flexible solution to this is essential to enable the future multimedia applications possible with MPEG-4. Two categories of video segmentation approaches can be identified: supervised and unsupervised. A representative set of unsupervised approaches is discussed. These approaches are found to be suitable for real-time MPEG-4 applications. However, they are not suitable for off-line applications which require very accurate segmentations of entire semantic objects. This is because an automatic segmentation process cannot solve the ill-posed problem of extracting semantic meaning from a scene. Supervised segmentation incorporates user interaction so that semantic objects in a scene can be defined. A representative set of supervised approaches with greater or lesser degrees of interaction is discussed. Three new approaches to the problem, each more sophisticated than the last, are presented by the author. The most sophisticated is an object-based approach in which an automatic segmentation and tracking algorithm is used to perform a segmentation of a scene in terms of the semantic objects defined by the user. The approach relies on maximum likelihood estimation of the parameters of mixtures of multimodal multivariate probability distribution functions. The approach is an enhanced and modified version of an existing approach yielding more sophisticated object modelling. The segmentation results obtained are comparable to those of existing approaches and in many cases better. It is concluded that the author’s approach is ideal as a content extraction tool for future off-line MPEG-4 applications

Video object segmentation.

Wei Wei.Thesis submitted in: December 2005.Thesis (M.Phil.)--Chinese University of Hong Kong, 2006.Includes bibliographical references (leaves 112-122).Abstracts in English and Chinese.Abstract --- p.IIList of Abbreviations --- p.IVChapter Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Overview of Content-based Video Standard --- p.1Chapter 1.2 --- Video Object Segmentation --- p.4Chapter 1.2.1 --- Video Object Plane (VOP) --- p.4Chapter 1.2.2 --- Object Segmentation --- p.5Chapter 1.3 --- Problems of Video Object Segmentation --- p.6Chapter 1.4 --- Objective of the research work --- p.7Chapter 1.5 --- Organization of This Thesis --- p.8Chapter 1.6 --- Notes on Publication --- p.8Chapter Chapter 2 --- Literature Review --- p.10Chapter 2.1 --- What is segmentation? --- p.10Chapter 2.1.1 --- Manual Segmentation --- p.10Chapter 2.1.2 --- Automatic Segmentation --- p.11Chapter 2.1.3 --- Semi-automatic segmentation --- p.12Chapter 2.2 --- Segmentation Strategy --- p.14Chapter 2.3 --- Segmentation of Moving Objects --- p.17Chapter 2.3.1 --- Motion --- p.18Chapter 2.3.2 --- Motion Field Representation --- p.19Chapter 2.3.3 --- Video Object Segmentation --- p.25Chapter 2.4 --- Summary --- p.35Chapter Chapter 3 --- Automatic Video Object Segmentation Algorithm --- p.37Chapter 3.1 --- Spatial Segmentation --- p.38Chapter 3.1.1 --- k:-Medians Clustering Algorithm --- p.39Chapter 3.1.2 --- Cluster Number Estimation --- p.41Chapter 3.1.2 --- Region Merging --- p.46Chapter 3.2 --- Foreground Detection --- p.48Chapter 3.2.1 --- Global Motion Estimation --- p.49Chapter 3.2.2 --- Detection of Moving Objects --- p.50Chapter 3.3 --- Object Tracking and Extracting --- p.50Chapter 3.3.1 --- Binary Model Tracking --- p.51Chapter 3.3.1.2 --- Initial Model Extraction --- p.53Chapter 3.3.2 --- Region Descriptor Tracking --- p.59Chapter 3.4 --- Results and Discussions --- p.65Chapter 3.4.1 --- Objective Evaluation --- p.65Chapter 3.4.2 --- Subjective Evaluation --- p.66Chapter 3.5 --- Conclusion --- p.74Chapter Chapter 4 --- Disparity Estimation and its Application in Video Object Segmentation --- p.76Chapter 4.1 --- Disparity Estimation --- p.79Chapter 4.1.1. --- Seed Selection --- p.80Chapter 4.1.2. --- Edge-based Matching by Propagation --- p.82Chapter 4.2 --- Remedy Matching Sparseness by Interpolation --- p.84Chapter 4.2 --- Disparity Applications in Video Conference Segmentation --- p.92Chapter 4.3 --- Conclusion --- p.106Chapter Chapter 5 --- Conclusion and Future Work --- p.108Chapter 5.1 --- Conclusion and Contribution --- p.108Chapter 5.2 --- Future work --- p.109Reference --- p.11

NEW CHANGE DETECTION MODELS FOR OBJECT-BASED ENCODING OF PATIENT MONITORING VIDEO

The goal of this thesis is to find a highly efficient algorithm to compress patient monitoring video. This type of video mainly contains local motions and a large percentage of idle periods. To specifically utilize these features, we present an object-based approach, which decomposes input video into three objects representing background, slow-motion foreground and fast-motion foreground. Encoding these three video objects with different temporal scalabilities significantly improves the coding efficiency in terms of bitrate vs. visual quality. The video decomposition is built upon change detection which identifies content changes between video frames. To improve the robustness of capturing small changes, we contribute two new change detection models. The model built upon Markov random theory discriminates foreground containing the patient being monitored. The other model, called covariance test method, identifies constantly changing content by exploiting temporal correlation in multiple video frames. Both models show great effectiveness in constructing the defined video objects. We present detailed algorithms of video object construction, as well as experimental results on the object-based coding of patient monitoring video

Unsupervised behavioral classification with 3D pose data from tethered Drosophila melanogaster

Tese de mestrado integrado em Engenharia Biomédica e Biofísica (Biofísica Médica e Fisiologia de Sistemas), Universidade de Lisboa, Faculdade de Ciências, 2020O comportamento animal e guiado por instruções geneticamente codificadas, com contribuições do meio envolvente e experiências antecedentes. O mesmo pode ser considerado como o derradeiro output da atividade neuronal, pelo que o estudo do comportamento animal constitui um meio de compreensão dos mecanismos subjacentes ao funcionamento do cérebro animal. Para desvendar a correspondência entre cérebro e comportamento são necessárias ferramentas que consigam medir um comportamento de forma precisa, apreciável e coerente. O domínio científico responsável pelo estudo dos comportamentos dos animais denomina-se Etologia. No início do seculo XX, os etólogos categorizavam comportamentos animais com recurso as suas próprias intuições e experiência. Consequentemente, as suas avaliações eram subjetivas e desprovidas de comportamentos que os etólogos não considerassem a priori. Com o ressurgimento de novas técnicas de captura e analise de comportamentos, os etólogos transitaram para paradigmas mais objetivos, quantitativos da medição de comportamentos. Tais ferramentas analíticas fomentaram a construção de datasets comportamentais que, por sua vez, promoveram o desenvolvimento de softwares para a quantificação de comportamentos: rastreamento de trajetórias, classificação de ações, analise de padrões comportamentais em grandes escalas consistem nos exemplos mais preeminentes. Este trabalho encontra-se inserido na segunda categoria referida (classificação de ações). Os classificadores de ações dividem-se consoante são supervisionados ou não-supervisionados. A primeira categoria compreende classificadores treinados para reconhecer padrões específicos, definidos por um especialista humano. Esta categoria de classificadores e encontra-se limitada por: 1) necessitar de um processo extenuado de anotação de frames para treino do classificador; 2) subjetividade face ao especialista que classifica os mesmos frames, 3) baixa dimensionalidade, na medida em que a classificação reduz os complexos comportamentos a um só rotulo; 4) assunções erróneas; 5) preconceito humano face aos comportamentos observados. Por sua vez, os classificadores não-supervisionados seguem exaustivamente uma formula: 1) computer vision e empregue para a extração das características posturais do animal; 2) dá-se o pré-processamento dos dados, que inclui um modulo vital que envolve a construção de uma representação dinâmico-postural das ações do animal, de forma a capturar os elementos dinâmicos do comportamento; 3) segue-se um modulo opcional de redução de dimensionalidade, caso o utilizador deseje visualizar diretamente os dados num espaço de reduzidas dimensões; 4) efetua-se a atribuição de um rótulo a cada elemento dos dados, por via de um algoritmo que opera quer diretamente no espaço de alta dimensão, ou no de baixa dimensão, resultante do passo anterior. O objetivo deste trabalho passa por alcançar uma classificação objetiva e reproduzível, de forma não-supervisionada de frames de Drosophila melanogaster suspensas numa bola que flutua no ar, tentando minimizar o número de intuições requeridas para o efeito e, se possível, dissipar a influência dos aspetos morfológicos de cada individuo (garantindo assim uma classificação generalizada dos comportamentos destes insetos). Para alcançar tal classificação, este estudo recorre a uma ferramenta recém desenvolvida que regista a pose tridimensional de Drosophila fixas, o DeepFly3D, para construir um dataset com as coordenadas x-, y- e z-, ao longo do tempo, das posições de referência de um conjunto de três genótipos de Drosophila melanogaster (linhas aDN>CsChrimson, MDN-GAL4/+ e aDNGAL4/+). Sucede-se uma operação inovadora de normalização que recorre ao cálculo de ângulos entre pontos de referência adjacentes, como as articulações, antenas e riscas dorsais das moscas, por via de relações trigonométricas e a definição dos planos anatómicos das moscas, que visa atenuar os pesos das diferenças morfológicas das moscas, ou a sua orientação relativa as camaras do DeepFly3D, para o classificador. O modulo de normalização e sucedido por outro de analise de frequência, focado na extração das frequências relevantes nas series temporais dos ângulos calculados, bem como dos seus pesos relativos. O produto final do pré-processamento consiste numa matriz com a norma dos ditos pesos – a matriz de expressão do espaço dinâmico-postural. Subsequentemente, seguem-se os módulos de redução de dimensionalidade e de atribuição de clusters (pontos 3) e 4) do paragrafo anterior). Para os mesmos, são propostas seis configurações possíveis de algoritmos, submetidas de imediato a uma anélise comparativa, de forma a determinar a mais apta para classificar este tipo de dados. Os algoritmos de redução de dimensionalidade aqui postos a prova são o t-SNE (t-distributed Stochastic Neighbor Embedding) e o PCA (Principal Component Analysis), enquanto que os algoritmos de clustering comparados são o Watershed, GMM-posterior probability assignment e o HDBSCAN (Hierarchical Density Based Spatial Clustering of Applications with Noise). Cada uma das pipelines candidatas e finalmente avaliada mediante a observação dos vídeos inclusos nos clusters produzidos e, dado o vasto numero destes vídeos, bem como a possibilidade de uma validação subjetiva face a observadores distintos, com o auxilio de métricas que expressam determinados critérios abrangentes de qualidade dos clusters: 1) Fly uncompactness, que avalia a eficiência do modulo de normalização com ângulos de referencia da mosca; 2) Homogeneity, que procura garantir que os clusters não refletem a identidade ou o genótipo das moscas; 3) Cluster entropy, que afere a previsibilidade das transições entre os clusters; 4) Mean dwell time, que pondera o tempo que um individuo demora em media a realizar uma Acão. Dois critérios auxiliares extra são ainda considerados: o número de parâmetros que foram estimados pelo utilizador (quanto maior, mais limitada e a reprodutibilidade da pipeline) e o tempo de execução do algoritmo (que deve ser igualmente minimizado). Apesar de manter alguma subjetividade face aquilo a que o utilizador considera um “bom” cluster, a inclusão das métricas aproxima esta abordagem a um cenário ideal de completa autonomia entre a conceção de uma definição de comportamento, e a validação dos resultados que decorrem das suas conjeturas. Os desempenhos das pipelines candidatas divergiram largamente: os espaços resultantes das operações de redução de dimensionalidade demonstram-se heterogéneos e anisotrópicos, com a presença de sequências de pontos que tomam formas vermiformes, ao invés de um antecipado conglomerado de pontos desassociados. Estas trajetórias vermiformes limitam o desempenho dos algoritmos de clustering que operam nos espaços de baixas (duas, neste caso) dimensões. A ausência de um passo intermedio de amostragem do espaço dinâmico-postural explica a génese destas trajetórias vermiformes. Não obstante, as pipelines que praticam redução de dimensionalidade geraram melhores resultados que a pipeline que recorre a clustering com HDBSCAN diretamente sobre a matriz de expressão do espaço dinâmico-postural. A combinação mais fortuita de módulos de redução de dimensionalidade e clustering adveio da pipeline PCA30-t-SNE2-GMM. Embora não sejam absolutamente consistentes, os clusters resultantes desta pipeline incluem um comportamento que se sobressai face aos demais que se encontram inseridos no mesmo cluster (erroneamente). Lacunas destes clusters envolvem sobretudo a ocasional fusão de dois comportamentos distintos no mesmo cluster, ou a presença inoportuna de sequências de comportamentos nas quais a mosca se encontra imóvel (provavelmente o resultado de pequenos erros de deteção produzidos pelo DeepFly3D). Para mais, a pipeline PCA30-t-SNE2-GMM foi capaz de reconhecer diferenças no fenótipo comportamental de moscas, validadas pelas linhas genéticas das mesmas. Apesar dos resultados obtidos manifestarem visíveis melhorias face aqueles produzidos por abordagens semelhantes, sobretudo a nível de vídeos dos clusters, uma vez que só uma das abordagens inclui métricas de sucesso dos clusters, alguns aspetos desta abordagem requerem correções: a inclusão de uma etapa de amostragem, sucedida de um novo algoritmo que fosse capaz de realizar reduções de dimensionalidade consistentes, de forma a reunir todos os pontos no mesmo espaço embutido será possivelmente a característica mais capaz de acrescentar valor a esta abordagem. Futuras abordagens não deverão descurar o contributo de múltiplas representações comportamentais que possam vir a validar-se mutuamente, substituindo a necessidade de métricas de sucesso definidas pelos utilizadores.One of the preeminent challenges of Behavioral Neuroscience is the understanding of how the brain works and how it ultimately commands an animal’s behavior. Solving this brain-behavior linkage requires, on one end, precise, meaningful and coherent techniques for measuring behavior. Rapid technical developments in tools for collecting and analyzing behavioral data, paired with the immaturity of current approaches, motivate an ongoing search for systematic, unbiased behavioral classification techniques. To accomplish such a classification, this study employs a state-of-the-art tool for tracking 3D pose of tethered Drosophila, DeepFly3D, to collect a dataset of x-, y- and z- landmark positions over time, from tethered Drosophila melanogaster moving over an air-suspended ball. This is succeeded by unprecedented normalization across individual flies by computing the angles between adjoining landmarks, followed by standard wavelet analysis. Subsequently, six unsupervised behavior classification techniques are compared - four of which follow proven formulas, while the remaining two are experimental. Lastly, their performances are evaluated via meaningful metric scores along with cluster video assessment, as to ensure a fully unbiased cycle - from the conjecturing of a definition of behavior to the corroboration of the results that stem from its assumptions. Performances from different techniques varied significantly. Techniques that perform clustering in embedded low- (two-) dimensional spaces struggled with their heterogeneous and anisotropic nature. High-dimensional clustering techniques revealed that these properties emerged from the original highdimensional posture-dynamics spaces. Nonetheless, high and low-dimensional spaces disagree on the arrangement of their elements, with embedded data points showing hierarchical organization, which was lacking prior to their embedding. Low-dimensional clustering techniques were globally a better match against these spatial features and yielded more suitable results. Their candidate embedding algorithms alone were capable of revealing dissimilarities in preferred behaviors among contrasting genotypes of Drosophila. Lastly, the top-ranking classification technique produced satisfactory behavioral cluster videos (despite the irregular allocation of rest labels) in a consistent and repeatable manner, while requiring a marginal number of hand tuned parameters