Enabling arbitrary rotation camera-motion using multi-sprites with minimum coding cost

Object-oriented coding in the MPEG-4 standard enables the separate processing of foreground objects and the scene background (sprite). Since the background sprite only has to be sent once, transmission bandwidth can be saved.We have found that the counter-intuitive approach of splitting the background into several independent parts can reduce the overall amount of data. Furthermore, we show that in the general case, the synthesis of a single background sprite is even impossible and that the scene background must be sent as multiple sprites instead. For this reason, we propose an algorithm that provides an optimal partitioning of a video sequence into independent background sprites (a multisprite), resulting in a significant reduction of the involved coding cost. Additionally, our sprite-generation algorithm ensures that the sprite resolution is kept high enough to preserve all details of the input sequence, which is a problem especially during camera zoom-in operations. Even though our sprite generation algorithm creates multiple sprites instead of only a single background sprite, it is fully compatible with the existing MPEG-4 standard. The algorithm has been evaluated with several test sequences, including the well-known Table-tennis and Stefan sequences. The total coding cost for the sprite VOP is reduced by a factor of about 2.6 or even higher, depending on the sequence

Automatic video segmentation employing object/camera modeling techniques

Practically established video compression and storage techniques still process video sequences as rectangular images without further semantic structure. However, humans watching a video sequence immediately recognize acting objects as semantic units. This semantic object separation is currently not reflected in the technical system, making it difficult to manipulate the video at the object level. The realization of object-based manipulation will introduce many new possibilities for working with videos like composing new scenes from pre-existing video objects or enabling user-interaction with the scene. Moreover, object-based video compression, as defined in the MPEG-4 standard, can provide high compression ratios because the foreground objects can be sent independently from the background. In the case that the scene background is static, the background views can even be combined into a large panoramic sprite image, from which the current camera view is extracted. This results in a higher compression ratio since the sprite image for each scene only has to be sent once. A prerequisite for employing object-based video processing is automatic (or at least user-assisted semi-automatic) segmentation of the input video into semantic units, the video objects. This segmentation is a difficult problem because the computer does not have the vast amount of pre-knowledge that humans subconsciously use for object detection. Thus, even the simple definition of the desired output of a segmentation system is difficult. The subject of this thesis is to provide algorithms for segmentation that are applicable to common video material and that are computationally efficient. The thesis is conceptually separated into three parts. In Part I, an automatic segmentation system for general video content is described in detail. Part II introduces object models as a tool to incorporate userdefined knowledge about the objects to be extracted into the segmentation process. Part III concentrates on the modeling of camera motion in order to relate the observed camera motion to real-world camera parameters. The segmentation system that is described in Part I is based on a background-subtraction technique. The pure background image that is required for this technique is synthesized from the input video itself. Sequences that contain rotational camera motion can also be processed since the camera motion is estimated and the input images are aligned into a panoramic scene-background. This approach is fully compatible to the MPEG-4 video-encoding framework, such that the segmentation system can be easily combined with an object-based MPEG-4 video codec. After an introduction to the theory of projective geometry in Chapter 2, which is required for the derivation of camera-motion models, the estimation of camera motion is discussed in Chapters 3 and 4. It is important that the camera-motion estimation is not influenced by foreground object motion. At the same time, the estimation should provide accurate motion parameters such that all input frames can be combined seamlessly into a background image. The core motion estimation is based on a feature-based approach where the motion parameters are determined with a robust-estimation algorithm (RANSAC) in order to distinguish the camera motion from simultaneously visible object motion. Our experiments showed that the robustness of the original RANSAC algorithm in practice does not reach the theoretically predicted performance. An analysis of the problem has revealed that this is caused by numerical instabilities that can be significantly reduced by a modification that we describe in Chapter 4. The synthetization of static-background images is discussed in Chapter 5. In particular, we present a new algorithm for the removal of the foreground objects from the background image such that a pure scene background remains. The proposed algorithm is optimized to synthesize the background even for difficult scenes in which the background is only visible for short periods of time. The problem is solved by clustering the image content for each region over time, such that each cluster comprises static content. Furthermore, it is exploited that the times, in which foreground objects appear in an image region, are similar to the corresponding times of neighboring image areas. The reconstructed background could be used directly as the sprite image in an MPEG-4 video coder. However, we have discovered that the counterintuitive approach of splitting the background into several independent parts can reduce the overall amount of data. In the case of general camera motion, the construction of a single sprite image is even impossible. In Chapter 6, a multi-sprite partitioning algorithm is presented, which separates the video sequence into a number of segments, for which independent sprites are synthesized. The partitioning is computed in such a way that the total area of the resulting sprites is minimized, while simultaneously satisfying additional constraints. These include a limited sprite-buffer size at the decoder, and the restriction that the image resolution in the sprite should never fall below the input-image resolution. The described multisprite approach is fully compatible to the MPEG-4 standard, but provides three advantages. First, any arbitrary rotational camera motion can be processed. Second, the coding-cost for transmitting the sprite images is lower, and finally, the quality of the decoded sprite images is better than in previously proposed sprite-generation algorithms. Segmentation masks for the foreground objects are computed with a change-detection algorithm that compares the pure background image with the input images. A special effect that occurs in the change detection is the problem of image misregistration. Since the change detection compares co-located image pixels in the camera-motion compensated images, a small error in the motion estimation can introduce segmentation errors because non-corresponding pixels are compared. We approach this problem in Chapter 7 by integrating risk-maps into the segmentation algorithm that identify pixels for which misregistration would probably result in errors. For these image areas, the change-detection algorithm is modified to disregard the difference values for the pixels marked in the risk-map. This modification significantly reduces the number of false object detections in fine-textured image areas. The algorithmic building-blocks described above can be combined into a segmentation system in various ways, depending on whether camera motion has to be considered or whether real-time execution is required. These different systems and example applications are discussed in Chapter 8. Part II of the thesis extends the described segmentation system to consider object models in the analysis. Object models allow the user to specify which objects should be extracted from the video. In Chapters 9 and 10, a graph-based object model is presented in which the features of the main object regions are summarized in the graph nodes, and the spatial relations between these regions are expressed with the graph edges. The segmentation algorithm is extended by an object-detection algorithm that searches the input image for the user-defined object model. We provide two objectdetection algorithms. The first one is specific for cartoon sequences and uses an efficient sub-graph matching algorithm, whereas the second processes natural video sequences. With the object-model extension, the segmentation system can be controlled to extract individual objects, even if the input sequence comprises many objects. Chapter 11 proposes an alternative approach to incorporate object models into a segmentation algorithm. The chapter describes a semi-automatic segmentation algorithm, in which the user coarsely marks the object and the computer refines this to the exact object boundary. Afterwards, the object is tracked automatically through the sequence. In this algorithm, the object model is defined as the texture along the object contour. This texture is extracted in the first frame and then used during the object tracking to localize the original object. The core of the algorithm uses a graph representation of the image and a newly developed algorithm for computing shortest circular-paths in planar graphs. The proposed algorithm is faster than the currently known algorithms for this problem, and it can also be applied to many alternative problems like shape matching. Part III of the thesis elaborates on different techniques to derive information about the physical 3-D world from the camera motion. In the segmentation system, we employ camera-motion estimation, but the obtained parameters have no direct physical meaning. Chapter 12 discusses an extension to the camera-motion estimation to factorize the motion parameters into physically meaningful parameters (rotation angles, focal-length) using camera autocalibration techniques. The speciality of the algorithm is that it can process camera motion that spans several sprites by employing the above multi-sprite technique. Consequently, the algorithm can be applied to arbitrary rotational camera motion. For the analysis of video sequences, it is often required to determine and follow the position of the objects. Clearly, the object position in image coordinates provides little information if the viewing direction of the camera is not known. Chapter 13 provides a new algorithm to deduce the transformation between the image coordinates and the real-world coordinates for the special application of sport-video analysis. In sport videos, the camera view can be derived from markings on the playing field. For this reason, we employ a model of the playing field that describes the arrangement of lines. After detecting significant lines in the input image, a combinatorial search is carried out to establish correspondences between lines in the input image and lines in the model. The algorithm requires no information about the specific color of the playing field and it is very robust to occlusions or poor lighting conditions. Moreover, the algorithm is generic in the sense that it can be applied to any type of sport by simply exchanging the model of the playing field. In Chapter 14, we again consider panoramic background images and particularly focus ib their visualization. Apart from the planar backgroundsprites discussed previously, a frequently-used visualization technique for panoramic images are projections onto a cylinder surface which is unwrapped into a rectangular image. However, the disadvantage of this approach is that the viewer has no good orientation in the panoramic image because he looks into all directions at the same time. In order to provide a more intuitive presentation of wide-angle views, we have developed a visualization technique specialized for the case of indoor environments. We present an algorithm to determine the 3-D shape of the room in which the image was captured, or, more generally, to compute a complete floor plan if several panoramic images captured in each of the rooms are provided. Based on the obtained 3-D geometry, a graphical model of the rooms is constructed, where the walls are displayed with textures that are extracted from the panoramic images. This representation enables to conduct virtual walk-throughs in the reconstructed room and therefore, provides a better orientation for the user. Summarizing, we can conclude that all segmentation techniques employ some definition of foreground objects. These definitions are either explicit, using object models like in Part II of this thesis, or they are implicitly defined like in the background synthetization in Part I. The results of this thesis show that implicit descriptions, which extract their definition from video content, work well when the sequence is long enough to extract this information reliably. However, high-level semantics are difficult to integrate into the segmentation approaches that are based on implicit models. Intead, those semantics should be added as postprocessing steps. On the other hand, explicit object models apply semantic pre-knowledge at early stages of the segmentation. Moreover, they can be applied to short video sequences or even still pictures since no background model has to be extracted from the video. The definition of a general object-modeling technique that is widely applicable and that also enables an accurate segmentation remains an important yet challenging problem for further research

Adaptive decoding of MPEG-4 sprites for memory-constrained embedded systems

Background sprite decoding is an essential part of object-based video coding.The composition and rendering of a final scene involves the placing of individual video objects in a predefined way superimposed on the decoded background image. The MPEG-4 standard includes the decoding algorithm for background image decoding, but this algorithm is not suitable for implementation on a memory-constrained platform. In this paper we present a modification of the decoding algorithm that decodes MPEG-4 sequences while fulfilling the requirements of a memory-constrained multiprocessor system with only 17% extra overhead of computation. Our algorithm reduces the memory cost of such decoding with a factor of four. Additionally, our algorithm offers the possibility of high level data parallelism and consequently contributes to an increase of throughput rate

RNA promotes the formation of spatial compartments in the nucleus

The nucleus is a highly organized arrangement of RNA, DNA, and protein molecules that are compartmentalized within three-dimensional (3D) structures involved in shared functional and regulatory processes. Although RNA has long been proposed to play a global role in organizing nuclear structure, exploring the role of RNA in shaping nuclear structure has remained a challenge because no existing methods can simultaneously measure RNA-RNA, RNA-DNA, and DNA-DNA contacts within 3D structures. To address this, we developed RNA & DNA SPRITE (RD-SPRITE) to comprehensively map the location of all RNAs relative to DNA and other RNAs. Using this approach, we identify many RNAs that are localized near their transcriptional loci (RNA-DNA) together with other diffusible ncRNAs (RNA-RNA) within higher-order DNA structures (DNA-DNA). These RNA-chromatin compartments span three major classes of nuclear functions: RNA processing (including ribosome biogenesis, mRNA splicing, snRNA biogenesis, and histone mRNA processing), heterochromatin assembly, and gene regulation. More generally, we identify hundreds of ncRNAs that form stable nuclear compartments in spatial proximity to their transcriptional loci. We find that dozens of nuclear compartments require RNA to guide protein regulators into these 3D structures, and focusing on several ncRNAs, we show that these ncRNAs specifically regulate heterochromatin assembly and the expression of genes contained within these compartments. Together, our results demonstrate a unique mechanism by which RNA acts to shape nuclear structure by forming high concentration territories immediately upon transcription, binding to diffusible regulators, and guiding them into spatial compartments to regulate a wide range of essential nuclear functions

RNA promotes the formation of spatial compartments in the nucleus

The nucleus is a highly organized arrangement of RNA, DNA, and protein molecules that are compartmentalized within three-dimensional (3D) structures involved in shared functional and regulatory processes. Although RNA has long been proposed to play a global role in organizing nuclear structure, exploring the role of RNA in shaping nuclear structure has remained a challenge because no existing methods can simultaneously measure RNA-RNA, RNA-DNA, and DNA-DNA contacts within 3D structures. To address this, we developed RNA & DNA SPRITE (RD-SPRITE) to comprehensively map the location of all RNAs relative to DNA and other RNAs. Using this approach, we identify many RNAs that are localized near their transcriptional loci (RNA-DNA) together with other diffusible ncRNAs (RNA-RNA) within higher-order DNA structures (DNA-DNA). These RNA-chromatin compartments span three major classes of nuclear functions: RNA processing (including ribosome biogenesis, mRNA splicing, snRNA biogenesis, and histone mRNA processing), heterochromatin assembly, and gene regulation. More generally, we identify hundreds of ncRNAs that form stable nuclear compartments in spatial proximity to their transcriptional loci. We find that dozens of nuclear compartments require RNA to guide protein regulators into these 3D structures, and focusing on several ncRNAs, we show that these ncRNAs specifically regulate heterochromatin assembly and the expression of genes contained within these compartments. Together, our results demonstrate a unique mechanism by which RNA acts to shape nuclear structure by forming high concentration territories immediately upon transcription, binding to diffusible regulators, and guiding them into spatial compartments to regulate a wide range of essential nuclear functions

Role of Cis-regulatory Elements in Transcriptional Regulation: From Evolution to 4D Interactions

Transcriptional regulation is the principal mechanism in establishing cell-type specific gene activity by exploring an almost infinite space of different combinations of regulatory elements, transcription factors with high precision. Recent efforts have mapped thousands of candidate regulatory elements, of which a great portion is cell-type specific yet it is still unclear as to what fraction of these elements is functional, what genes these elements regulate, or how they are established in a cell-type specific manner. In this dissertation, I will discuss methods and approaches I developed to better understand the role of regulatory elements and transcription factors in gene expression regulation. First, by comparing the transcriptome and chromatin landscape between mouse and human innate immune cells I showed specific gene expression programs are regulated by highly conserved regulatory elements that contain a set of constrained sequence motifs, which can successfully classify gene-induction in both species. Next, using chromatin interactions I accurately defined functional enhancers and their target genes. This fine mapping dramatically improved the prediction of transcriptional changes. Finally, we built a supervised learning approach to detect the short DNA sequences motifs that regulate the activation of regulatory elements following LPS stimulation. This approach detected several transcription factors to be critical in remodeling the epigenetic landscape both across time and individuals. Overall this thesis addresses several important aspects of cis-regulatory elements in transcriptional regulation and started to derive principles and models of gene-expression regulation that address the fundamental question: “How do cis-regulatory elements drive cell-type-specific transcription?

Detection of 3D Genome Folding at Multiple Scales

Understanding 3D genome structure is crucial to learn how chromatin folds and how genes are regulated through the spatial organization of regulatory elements. Various technologies have been developed to investigate genome architecture. These technologies include ligation-based 3C Methodologies such as Hi-C and Micro-C, ligation-based pull-down methods like Proximity Ligation-Assisted ChIP-seq (PLAC Seq) and Paired-end tag sequencing (ChIA PET), and ligation-free methods like Split-Pool Recognition of Interactions by Tag Extension (SPRITE) and Genome Architecture Mapping (GAM). Although these technologies have provided great insight into chromatin organization, a systematic evaluation of these technologies is lacking. Among these technologies, Hi-C has been one of the most widely used methods to map genome-wide chromatin interactions for over a decade. To understand how the choice of experimental parameters determines the ability to detect and quantify the features of chromosome folding, we have first systematically evaluated two critical parameters in the Hi-C protocol: cross-linking and digestion of chromatin. We found that different protocols capture distinct 3D genome features with different efficiencies depending on the cell type (Chapter 2). Use of the updated Hi-C protocol with new parameters, which we call Hi-C 3.0, was subsequently evaluated and found to provide the best loop detection compared to all previous Hi-C protocols as well as better compartment quantification compared to Micro-C (Chapter 3). Finally, to understand how the aforementioned technologies (Hi-C, Micro-C, PLAC-Seq, ChIA-PET, SPRITE, GAM) that measure 3D organization could provide a comprehensive understanding of the genome structure, we have performed a comparison of these technologies. We found that each of these methods captures different aspects of the chromatin folding (Chapter 4). Collectively, these studies suggest that improving the 3D methodologies and integrative analyses of these methods will reveal unprecedented details of the genome structure and function

Higher-Order Inter-chromosomal Hubs Shape 3D Genome Organization in the Nucleus

Eukaryotic genomes are packaged into a 3-dimensional structure in the nucleus. Current methods for studying genome-wide structure are based on proximity ligation. However, this approach can fail to detect known structures, such as interactions with nuclear bodies, because these DNA regions can be too far apart to directly ligate. Accordingly, our overall understanding of genome organization remains incomplete. Here, we develop split-pool recognition of interactions by tag extension (SPRITE), a method that enables genome-wide detection of higher-order interactions within the nucleus. Using SPRITE, we recapitulate known structures identified by proximity ligation and identify additional interactions occurring across larger distances, including two hubs of inter-chromosomal interactions that are arranged around the nucleolus and nuclear speckles. We show that a substantial fraction of the genome exhibits preferential organization relative to these nuclear bodies. Our results generate a global model whereby nuclear bodies act as inter-chromosomal hubs that shape the overall packaging of DNA in the nucleus

Exploration of Pervasive Games in Relation to Mobile Technologies

The project is an exploration of Pervasive Games in relation to mobile technologies, with the intention of developing a pervasive game engine. Pervasive Games are interactive games where the participants drive the game play by playing the game in both the real world and a virtual environment. This is an area of gaming that has rapidly evolved over the last few years. The initial research involved establishing several key elements common to existing pervasive applications, defining real world / virtual world considerations for game play (both positive and negative) and identifying the technical requirements needed to implement play elements on a mobile device. After comparing several platforms the Windows 7 platform was selected for development purposes. The requirements for establishing a working development platform (with delivery mechanism) was investigated and a working environment set-up. A pervasive games engine was then developed in the format of 67 code stubs (coding solutions) that allow the implementation of solutions to gaming elements required in the development of pervasive applications. Two new helper classes were in addition developed containing solutions to topics related to run-time data storage (StorageUtils.cs) and generic gaming tasks (GameCode.cs). A pervasive game was implemented to test a cross section of functionality in the engine. The basic principle behind the game was to overlay various layers video, backgrounds, sprite and text, to build up an immersive pervasive environment with a player in the centre of the game imagery, game domain and real world. The intention of the game was to see how the pervasive game experience could be reflected in the game mechanics and pervasive interaction, while utilising the engine functionality

Higher-Order RNA and DNA Hubs Shape Genome Organization in the Nucleus

Although the entire genome is present within the nucleus of every cell, distinct genes need to be accessed and expressed in different cellular conditions. Accordingly, the nucleus of each cell is a highly organized arrangement of DNA, RNA, and protein that is dynamically assembled and regulated in different cellular states. These dynamic nuclear structures are largely arranged around functionally related roles and often occur across multiple chromosomes. These include large nuclear bodies (i.e., nucleolus, nuclear speckle), smaller nuclear bodies (i.e., Cajal bodies and histone locus bodies), and gene-gene interactions (i.e., transcription compartments and loops). Yet, what molecular components are involved in establishing this dynamic organization have been largely unknown due to a lack of methods to measure the RNA and DNA components of nuclear bodies and their spatial arrangements in the nucleus. Here, we present Split-Pool Recognition of Interactions by Tag Extension (SPRITE), which enables genome-wide detection of higher-order interactions within the nucleus. In the second chapter, we introduce SPRITE and recapitulate known structures identified by proximity ligation and identify additional interactions occurring across larger distances, including two hubs of inter-chromosomal interactions that are arranged around the nucleolus and nuclear speckles. We show that a substantial fraction of the genome exhibits preferential organization relative to these nuclear bodies. Our results generate a global model whereby nuclear bodies act as inter-chromosomal hubs that shape the overall packaging of DNA in the nucleus. In the third chapter, we provide a detailed experimental protocol for performing SPRITE and an automated computational pipeline for analyzing SPRITE data. Finally, in the fourth chapter, we present a dramatically improved implementation of the SPRITE method that enables comprehensive mapping of all classes of RNA in the nucleus, from abundant RNAs encoded from DNA repeats to low abundance RNAs such as nascent pre-mRNAs and lncRNAs. We find that RNAs localize broadly across the nucleus, with individual RNAs localizing within discrete territories ranging from nuclear bodies to individual topologically associated domains. We uncover that nascent mRNAs interact in structures corresponding to nascent mRNA chromosome territories and compartments. Together, these results uncover a central and widespread role for non-coding RNA in demarcating 3D nuclear structures within the nucleus.</p