Multiple Media Correlation: Theory and Applications

This thesis introduces multiple media correlation, a new technology for the automatic alignment of multiple media objects such as text, audio, and video. This research began with the question: what can be learned when multiple multimedia components are analyzed simultaneously? Most ongoing research in computational multimedia has focused on queries, indexing, and retrieval within a single media type. Video is compressed and searched independently of audio, text is indexed without regard to temporal relationships it may have to other media data. Multiple media correlation provides a framework for locating and exploiting correlations between multiple, potentially heterogeneous, media streams. The goal is computed synchronization, the determination of temporal and spatial alignments that optimize a correlation function and indicate commonality and synchronization between media objects. The model also provides a basis for comparison of media in unrelated domains. There are many real-world applications for this technology, including speaker localization, musical score alignment, and degraded media realignment. Two applications, text-to-speech alignment and parallel text alignment, are described in detail with experimental validation. Text-to-speech alignment computes the alignment between a textual transcript and speech-based audio. The presented solutions are effective for a wide variety of content and are useful not only for retrieval of content, but in support of automatic captioning of movies and video. Parallel text alignment provides a tool for the comparison of alternative translations of the same document that is particularly useful to the classics scholar interested in comparing translation techniques or styles. The results presented in this thesis include (a) new media models more useful in analysis applications, (b) a theoretical model for multiple media correlation, (c) two practical application solutions that have wide-spread applicability, and (d) Xtrieve, a multimedia database retrieval system that demonstrates this new technology and demonstrates application of multiple media correlation to information retrieval. This thesis demonstrates that computed alignment of media objects is practical and can provide immediate solutions to many information retrieval and content presentation problems. It also introduces a new area for research in media data analysis

Indexing and Retrieval of 3D Articulated Geometry Models

In this PhD research study, we focus on building a content-based search engine for 3D articulated geometry models. 3D models are essential components in nowadays graphic applications, and are widely used in the game, animation and movies production industry. With the increasing number of these models, a search engine not only provides an entrance to explore such a huge dataset, it also facilitates sharing and reusing among different users. In general, it reduces production costs and time to develop these 3D models. Though a lot of retrieval systems have been proposed in recent years, search engines for 3D articulated geometry models are still in their infancies. Among all the works that we have surveyed, reliability and efficiency are the two main issues that hinder the popularity of such systems. In this research, we have focused our attention mainly to address these two issues. We have discovered that most existing works design features and matching algorithms in order to reflect the intrinsic properties of these 3D models. For instance, to handle 3D articulated geometry models, it is common to extract skeletons and use graph matching algorithms to compute the similarity. However, since this kind of feature representation is complex, it leads to high complexity of the matching algorithms. As an example, sub-graph isomorphism can be NP-hard for model graph matching. Our solution is based on the understanding that skeletal matching seeks correspondences between the two comparing models. If we can define descriptive features, the correspondence problem can be solved by bag-based matching where fast algorithms are available. In the first part of the research, we propose a feature extraction algorithm to extract such descriptive features. We then convert the skeletal matching problems into bag-based matching. We further define metric similarity measure so as to support fast search. We demonstrate the advantages of this idea in our experiments. The improvement on precision is 12\% better at high recall. The indexing search of 3D model is 24 times faster than the state of the art if only the first relevant result is returned. However, improving the quality of descriptive features pays the price of high dimensionality. Curse of dimensionality is a notorious problem on large multimedia databases. The computation time scales exponentially as the dimension increases, and indexing techniques may not be useful in such situation. In the second part of the research, we focus ourselves on developing an embedding retrieval framework to solve the high dimensionality problem. We first argue that our proposed matching method projects 3D models on manifolds. We then use manifold learning technique to reduce dimensionality and maximize intra-class distances. We further propose a numerical method to sub-sample and fast search databases. To preserve retrieval accuracy using fewer landmark objects, we propose an alignment method which is also beneficial to existing works for fast search. The advantages of the retrieval framework are demonstrated in our experiments that it alleviates the problem of curse of dimensionality. It also improves the efficiency (3.4 times faster) and accuracy (30\% more accurate) of our matching algorithm proposed above. In the third part of the research, we also study a closely related area, 3D motions. 3D motions are captured by sticking sensor on human beings. These captured data are real human motions that are used to animate 3D articulated geometry models. Creating realistic 3D motions is an expensive and tedious task. Although 3D motions are very different from 3D articulated geometry models, we observe that existing works also suffer from the problem of temporal structure matching. This also leads to low efficiency in the matching algorithms. We apply the same idea of bag-based matching into the work of 3D motions. From our experiments, the proposed method has a 13\% improvement on precision at high recall and is 12 times faster than existing works. As a summary, we have developed algorithms for 3D articulated geometry models and 3D motions, covering feature extraction, feature matching, indexing and fast search methods. Through various experiments, our idea of converting restricted matching to bag-based matching improves matching efficiency and reliability. These have been shown in both 3D articulated geometry models and 3D motions. We have also connected 3D matching to the area of manifold learning. The embedding retrieval framework not only improves efficiency and accuracy, but has also opened a new area of research

Affect-based indexing and retrieval of multimedia data

Digital multimedia systems are creating many new opportunities for rapid access to content archives. In order to explore these collections using search, the content must be annotated with significant features. An important and often overlooked aspect o f human interpretation o f multimedia data is the affective dimension. The hypothesis o f this thesis is that affective labels o f content can be extracted automatically from within multimedia data streams, and that these can then be used for content-based retrieval and browsing. A novel system is presented for extracting affective features from video content and mapping it onto a set o f keywords with predetermined emotional interpretations. These labels are then used to demonstrate affect-based retrieval on a range o f feature films. Because o f the subjective nature o f the words people use to describe emotions, an approach towards an open vocabulary query system utilizing the electronic lexical database WordNet is also presented. This gives flexibility for search queries to be extended to include keywords without predetermined emotional interpretations using a word-similarity measure. The thesis presents the framework and design for the affectbased indexing and retrieval system along with experiments, analysis, and conclusions

Improving Content Based Video Retrieval Performance by Using Hadoop-MapReduce Model

In this paper, we present a distributed Content- Based Video Retrieval (CBVR) system based on MapReduce pro- gramming model. A CBVR system called Bounded Coordinate of Motion Histogram (BCMH) has been implemented as case study by using Hadoop framework. Our work consists of proposing a distributed model to extract videos signatures and compute similarity with the BCMH system based on a set of Mapreduce jobs assigned to multiple nodes of the Hadoop cluster in order to reduce computation time of training process. The proposed approach is tested on HOLLYWOOD2 dataset and the obtained results demonstrate efficiency of the proposed approach

Geo-Tagged Video Management: Storage, Queries and Streaming

Ph.DDOCTOR OF PHILOSOPH

Model- and image-based scene representation.

Lee Kam Sum.Thesis (M.Phil.)--Chinese University of Hong Kong, 1999.Includes bibliographical references (leaves 97-101).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.2Chapter 1.1 --- Video representation using panorama mosaic and 3D face model --- p.2Chapter 1.2 --- Mosaic-based Video Representation --- p.3Chapter 1.3 --- "3D Human Face modeling ," --- p.7Chapter 2 --- Background --- p.13Chapter 2.1 --- Video Representation using Mosaic Image --- p.13Chapter 2.1.1 --- Traditional Video Compression --- p.17Chapter 2.2 --- 3D Face model Reconstruction via Multiple Views --- p.19Chapter 2.2.1 --- Shape from Silhouettes --- p.19Chapter 2.2.2 --- Head and Face Model Reconstruction --- p.22Chapter 2.2.3 --- Reconstruction using Generic Model --- p.24Chapter 3 --- System Overview --- p.27Chapter 3.1 --- Panoramic Video Coding Process --- p.27Chapter 3.2 --- 3D Face model Reconstruction Process --- p.28Chapter 4 --- Panoramic Video Representation --- p.32Chapter 4.1 --- Mosaic Construction --- p.32Chapter 4.1.1 --- Cylindrical Panorama Mosaic --- p.32Chapter 4.1.2 --- Cylindrical Projection of Mosaic Image --- p.34Chapter 4.2 --- Foreground Segmentation and Registration --- p.37Chapter 4.2.1 --- Segmentation Using Panorama Mosaic --- p.37Chapter 4.2.2 --- Determination of Background by Local Processing --- p.38Chapter 4.2.3 --- Segmentation from Frame-Mosaic Comparison --- p.40Chapter 4.3 --- Compression of the Foreground Regions --- p.44Chapter 4.3.1 --- MPEG-1 Compression --- p.44Chapter 4.3.2 --- MPEG Coding Method: I/P/B Frames --- p.45Chapter 4.4 --- Video Stream Reconstruction --- p.48Chapter 5 --- Three Dimensional Human Face modeling --- p.52Chapter 5.1 --- Capturing Images for 3D Face modeling --- p.53Chapter 5.2 --- Shape Estimation and Model Deformation --- p.55Chapter 5.2.1 --- Head Shape Estimation and Model deformation --- p.55Chapter 5.2.2 --- Face organs shaping and positioning --- p.58Chapter 5.2.3 --- Reconstruction with both intrinsic and extrinsic parameters --- p.59Chapter 5.2.4 --- Reconstruction with only Intrinsic Parameter --- p.63Chapter 5.2.5 --- Essential Matrix --- p.65Chapter 5.2.6 --- Estimation of Essential Matrix --- p.66Chapter 5.2.7 --- Recovery of 3D Coordinates from Essential Matrix --- p.67Chapter 5.3 --- Integration of Head Shape and Face Organs --- p.70Chapter 5.4 --- Texture-Mapping --- p.71Chapter 6 --- Experimental Result & Discussion --- p.74Chapter 6.1 --- Panoramic Video Representation --- p.74Chapter 6.1.1 --- Compression Improvement from Foreground Extraction --- p.76Chapter 6.1.2 --- Video Compression Performance --- p.78Chapter 6.1.3 --- Quality of Reconstructed Video Sequence --- p.80Chapter 6.2 --- 3D Face model Reconstruction --- p.91Chapter 7 --- Conclusion and Future Direction --- p.94Bibliography --- p.10

Context-based motion retrieval using vector space model

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 87-89).Motion retrieval is the problem of retrieving highly relevant motions in a timely manner. The principal challenge is to characterize the similarity between two motions effectively, which is tightly related to the gap between the motion data's representation and its semantics. Our approach uses vector space model to measure the similarities among motions, which are made discrete using the vocabulary technique and transformation invariant using the relational feature model. In our approach, relational features are first extracted from motion data. then such features are clustered into a motion vocabulary. Finally motions are turned into bag of words and retrieved using vector-space model. We implemented this new system and tested it on two benchmark databases composed of real world data. Two existing methods, the dynamics time warping method and the binary feature method, are implemented for comparison. The results shows that our system are comparable in effectiveness with the dynamic time warping system, but runs 100 to 400 times faster. In comparison to retrieval with binary features, it is just as fast but more accurate and practical.The success of our system points to several additional improvements. Our experiments reveal that the velocity features improve the relevance of retrieved results, but more effort should be dedicated to determining the best set of features for motion retrieval. The same experiments should be performed on large databases and in particular to test how this performance generalizes on test motions outside the original database. The alternative vocabulary organizations, such as vocabulary tree and random forest, should be investigated because they can improve our approach by providing more flexibility to the similarity scoring model and reducing the approximation error of the vocabulary. Because the bag of words model ignores the temporal ordering of key features, a wavelet model should also be explored as a mechanism to encode features across different time scales.(cont.) The alternative vocabulary organizations, such as vocabulary tree and random forest, should be investigated because they can improve our approach by providing more flexibility to the similarity scoring model and reducing the approximation error of the vocabulary. Because the bag of words model ignores the temporal ordering of key features, a wavelet model should also be explored as a mechanism to encode features across different time scales.by Zhunping 'Justin' Zhang.S.M