Loss-resilient Coding of Texture and Depth for Free-viewpoint Video Conferencing

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is commonly synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this end, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will erase errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Experimental results show that the proposed scheme can outperform the use of a traditional feedback channel by up to 0.82 dB on average at 8% packet loss rate, and by as much as 3 dB for particular frames

Segmentation Based Image Scanning

The submitted paper deals with separate scanning of individual image segments. A new image processing approach based on image segmentation and segment scanning is presented. The resulting individual segments 1-dimensional representation provides higher neighbor pixel similarity than the 1-dimensional representation of the original image. This increased adjacent pixel similarity was achieved even without application of different recursive 2-dimensional scanning methods [4], such as Peano-Hilbert scanning method [1]. The resulting 1-dimensional image representation provides a good base for applying lossless compression methods, such as the entropic coding. The paper contains also results analysis of the traditional method scanned segment pixels and adjacent pixel differences from the entropy point of view. As these results indicate the lossy compression methods could be applicable using this approach as well and might improve the final results as confirmed by simple prediction algorithm results presented in this paper. More complex and sophisticated lossy compression algorithms application will be a part of the future work

Error-resilient multi-view video plus depth based 3-D video coding

Three Dimensional (3-D) video, by definition, is a collection of signals that can provide depth perception of a 3-D scene. With the development of 3-D display technologies and interactive multimedia systems, 3-D video has attracted significant interest from both industries and academia with a variety of applications. In order to provide desired services in various 3-D video applications, the multiview video plus depth (MVD) representation, which can facilitate the generation of virtual views, has been determined to be the best format for 3-D video data. Similar to 2-D video, compressed 3-D video is highly sensitive to transmission errors due to errors propagated from the current frame to the future predicted frames. Moreover, since the virtual views required for auto-stereoscopic displays are rendered from the compressed texture videos and depth maps, transmission errors of the distorted texture videos and depth maps can be further propagated to the virtual views. Besides, the distortions in texture and depth show different effects on the rendering views. Therefore, compared to the reliability of the transmission of the 2-D video, error-resilient texture video and depth map coding are facing major new challenges. This research concentrates on improving the error resilience performance of MVD-based 3-D video in packet loss scenarios. Based on the analysis of the propagating behaviour of transmission errors, a Wyner-Ziv (WZ)-based error-resilient algorithm is first designed for coding of the multi-view video data or depth data. In this scheme, an auxiliary redundant stream encoded according to WZ principle is employed to protect a primary stream encoded with standard multi-view video coding codec. Then, considering the fact that different combinations of texture and depth coding mode will exhibit varying robustness to transmission errors, a rate-distortion optimized mode switching scheme is proposed to strike the optimal trade-off between robustness and compression effciency. In this approach, the texture and depth modes are jointly optimized by minimizing the overall distortion of both the coded and synthesized views subject to a given bit rate. Finally, this study extends the research on the reliable transmission of view synthesis prediction (VSP)-based 3-D video. In order to mitigate the prediction position error caused by packet losses in the depth map, a novel disparity vector correction algorithm is developed, where the corrected disparity vector is calculated from the depth error. To facilitate decoder error concealment, the depth error is recursively estimated at the decoder. The contributions of this dissertation are multifold. First, the proposed WZbased error-resilient algorithm can accurately characterize the effect of transmission error on multi-view distortion at the transform domain in consideration of both temporal and inter-view error propagation, and based on the estimated distortion, this algorithm can perform optimal WZ bit allocation at the encoder through explicitly developing a sophisticated rate allocation strategy. This proposed algorithm is able to provide a finer granularity in performing rate adaptivity and unequal error protection for multi-view data, not only at the frame level, but also at the bit-plane level. Secondly, in the proposed mode switching scheme, a new analytic model is formulated to optimally estimate the view synthesis distortion due to packet losses, in which the compound impact of the transmission distortions of both the texture video and the depth map on the quality of the synthesized view is mathematically analysed. The accuracy of this view synthesis distortion model is demonstrated via simulation results and, further, the estimated distortion is integrated into a rate-distortion framework for optimal mode switching to achieve substantial performance gains over state-of-the-art algorithms. Last, but not least, this dissertation provides a preliminary investigation of VSP-based 3-D video over unreliable channel. In the proposed disparity vector correction algorithm, the pixel-level depth map error can be precisely estimated at the decoder without the deterministic knowledge of the error-free reconstructed depth. The approximation of the innovation term involved in depth error estimation is proved theoretically. This algorithm is very useful to conceal the position-erroneous pixels whose disparity vectors are correctly received

Robust and efficient video/image transmission

The Internet has become a primary medium for information transmission. The unreliability of channel conditions, limited channel bandwidth and explosive growth of information transmission requests, however, hinder its further development. Hence, research on robust and efficient delivery of video/image content is demanding nowadays. Three aspects of this task, error burst correction, efficient rate allocation and random error protection are investigated in this dissertation. A novel technique, called successive packing, is proposed for combating multi-dimensional (M-D) bursts of errors. A new concept of basis interleaving array is introduced. By combining different basis arrays, effective M-D interleaving can be realized. It has been shown that this algorithm can be implemented only once and yet optimal for a set of error bursts having different sizes for a given two-dimensional (2-D) array. To adapt to variable channel conditions, a novel rate allocation technique is proposed for FineGranular Scalability (FGS) coded video, in which real data based rate-distortion modeling is developed, constant quality constraint is adopted and sliding window approach is proposed to adapt to the variable channel conditions. By using the proposed technique, constant quality is realized among frames by solving a set of linear functions. Thus, significant computational simplification is achieved compared with the state-of-the-art techniques. The reduction of the overall distortion is obtained at the same time. To combat the random error during the transmission, an unequal error protection (UEP) method and a robust error-concealment strategy are proposed for scalable coded video bitstreams

Digital encoding of black and white facsimile signals

As the costs of digital signal processing and memory hardware are decreasing each year compared to those of transmission, it is increasingly economical to apply sophisticated source encoding techniques to reduce the transmission time for facsimile documents. With this intent, information lossy encoding schemes have been investigated in which the encoder is divided into two stages. Firstly, preprocessing, which removes redundant information from the original documents, and secondly, actual encoding of the preprocessed documents. [Continues.

Slice-Level Trading of Quality and Performance in Decoding H.264 Video: Slice-basiertes Abwägen zwischen Qualität und Leistung beim Dekodieren von H.264-Video

When a demanding video decoding task requires more CPU resources then available, playback degrades ungracefully today: The decoder skips frames selected arbitrarily or by simple heuristics, which is noticed by the viewer as jerky motion in the good case or as images completely breaking up in the bad case. The latter can happen due to missing reference frames. This thesis provides a way to schedule individual decoding tasks based on a cost for performance trade. Therefore, I will present a way to preprocess a video, generating estimates for the cost in terms of execution time and the performance in terms of perceived visual quality. The granularity of the scheduling decision is a single slice, which leads to a much more fine-grained approach than dealing with entire frames. Together with an actual scheduler implementation that uses the generated estimates, this work allows for higher perceived quality video playback in case of CPU overload.Wenn eine anspruchsvolle Video-Dekodierung mehr Prozessor-Ressourcen benötigt, als verfügbar sind, dann verschlechtert sich die Abspielqualität mit aktuellen Methoden drastisch: Willkürlich oder mit einfachen Heuristiken ausgewählten Bilder werden nicht dekodiert. Diese Auslassung nimmt der Betrachter im günstigsten Fall nur als ruckelnde Bewegung wahr, im ungünstigen Fall jedoch als komplettes Zusammenbrechen nachfolgender Bilder durch Folgefehler im Dekodierprozess. Meine Arbeit ermöglicht es, einzelne Teilaufgaben des Dekodierprozesses anhand einer Kosten-Nutzen-Analyse einzuplanen. Dafür ermittle ich die Kosten im Sinne von Rechenzeitbedarf und den Nutzen im Sinne von visueller Qualität für einzelne Slices eines H.264 Videos. Zusammen mit einer Implementierung eines Schedulers, der diese Werte nutzt, erlaubt meine Arbeit höhere vom Betrachter wahrgenommene Videoqualität bei knapper Prozessorzeit

Reflections on en-teaching: Dewey, Heidegger and Lao Tzu

Reflecting on my past two unsettling journeys of teaching in China and America produces questions about the teaching of truth in chapter one. The question of truth as it relates to the teacher’s role in the classroom raises not only issues of what and how we should teach, but challenges the very purpose of teaching. When I explored Martin Heidegger’s phenomenological perspective on (un)truth for insights into taken-for-granted assumptions about education and the purposes of teaching and learning, I noticed a strong resonance between his notion of “clearing” and the essential spirit of Taoism, “the Tao of inaction.” This led me to coin the word, “en-teaching” to express my idea of how teachers can teach through paradoxically non-teaching, without implying a binary opposition between teaching and non-teaching. In reviewing selected literature critical of the “teaching-as-telling” in America and China, I suggest in chapter two that the traditional direct teaching of truth has been entrenched in the public school systems in both countries as not only a teaching method but an implicit educational culture. The essence of this “teaching-as-telling” in both countries is the same – the will to control. What alternatives might there be to the method of teaching-as-telling? How can we teach otherwise? Or can we? Since I struggle with the question of truth related to teaching and this question “assumes the greatest urgency in Heidegger’s thought” (Sallis, 20), chapter three focuses on Heidegger’s complex explorations of (un)truth in clearings between brightness and darkness along with his concept of “let learn” through “always-being-in-the-world.” In chapter four, I go further to explore my notion of en-teaching based upon Heidegger’s thoughts of teaching and learning with insights from Lao Tzu and Dewey. In the last chapter, I try to not only reflect upon all previous chapters but respond to the practical question, “What does en-teaching mean to me as I face my class on Monday morning?

Service quality assurance for the IPTV networks

The objective of the proposed research is to design and evaluate end-to-end solutions to support the Quality of Experience (QoE) for the Internet Protocol Television (IPTV) service. IPTV is a system that integrates voice, video, and data delivery into a single Internet Protocol (IP) framework to enable interactive broadcasting services at the subscribers. It promises significant advantages for both service providers and subscribers. For instance, unlike conventional broadcasting systems, IPTV broadcasts will not be restricted by the limited number of channels in the broadcast/radio spectrum. Furthermore, IPTV will provide its subscribers with the opportunity to access and interact with a wide variety of high-quality on-demand video content over the Internet. However, these advantages come at the expense of stricter quality of service (QoS) requirements than traditional Internet applications. Since IPTV is considered as a real-time broadcast service over the Internet, the success of the IPTV service depends on the QoE perceived by the end-users. The characteristics of the video traffic as well as the high-quality requirements of the IPTV broadcast impose strict requirements on transmission delay. IPTV framework has to provide mechanisms to satisfy the stringent delay, jitter, and packet loss requirements of the IPTV service over lossy transmission channels with varying characteristics. The proposed research focuses on error recovery and channel change latency problems in IPTV networks. Our specific aim is to develop a content delivery framework that integrates content features, IPTV application requirements, and network characteristics in such a way that the network resource utilization can be optimized for the given constraints on the user perceived service quality. To achieve the desired QoE levels, the proposed research focuses on the design of resource optimal server-based and peer-assisted delivery techniques. First, by analyzing the tradeoffs on the use of proactive and reactive repair techniques, a solution that optimizes the error recovery overhead is proposed. Further analysis on the proposed solution is performed by also focusing on the use of multicast error recovery techniques. By investigating the tradeoffs on the use of network-assisted and client-based channel change solutions, distributed content delivery frameworks are proposed to optimize the error recovery performance. Next, bandwidth and latency tradeoffs associated with the use of concurrent delivery streams to support the IPTV channel change are analyzed, and the results are used to develop a resource-optimal channel change framework that greatly improves the latency performance in the network. For both problems studied in this research, scalability concerns for the IPTV service are addressed by properly integrating peer-based delivery techniques into server-based solutions.Ph.D