Bitstream Processing and Analysis for Video Communication Systems

在本論文中，我們介紹了在視訊通訊系統中位元流處理的演算法分析與架構設計。由於視訊資料是以位元流的格式來存放，許多重要的資訊隱藏在其中，因此位元流在現今數位多媒體時代扮演相當重要的角色。 首先，我們提出一個視訊串流系統。為了實現一個內容感知的視訊串流系統，我們提出了一個新的傳輸模型，其主要的觀念是將視訊編碼與視訊加強分開。串流伺服器可依據編碼端所提供之物件資訊，以及接收端所提供之使用者自定的區域，動態調整欲加強畫質的區域。除此以外，畫面中物件移動的資訊也可以當作參考。所提出的系統可提供給使用者較好的主觀視訊品質，而沒有增加額外的負擔。 接下來，我們設計了一個適用於視訊串流系統的MPEG-4 FGS編碼器。我們分析了FGS編碼的運算複雜度，重新調整其編碼流程，能提早得到整個畫面最高的位元平面，並動態取得所需的位元平面資料。所提出的設計完全相容於MPEG-4。藉由所提出之以硬體設計為導向的方法，我們實現了FGS編碼器的設計。 此外，我們也對在解碼端位元流解碼的運算作分析，並提出一個有效率且彈性的位元流解碼處理器。針對有作資料分割（data partitioned）的位元流，我們提出有效的方式來處理。在指令集的設計上，由於DCT係數佔位元流較大的比例，因此也對其解碼作加速。實作上，所提出的設計成功的整合至MPEG-4解碼系統中，可達到即時的位元流解碼。 最後，對於傳輸時會有錯誤的情形發生，我們以統計的方式來分析錯誤對MPEG-4資料分割位元流的影響。對不同錯誤偵測的條件，藉由位元流的結構與其中的欄位，以找出主要錯誤的來源。當錯誤在motion的部份被偵測到時，較高的可能會是motion marker發生錯誤；當錯誤在DCT係數的部份被偵測到時，較高的可能會是DCT係數本身發生錯誤。根據這些特性，我們提出相對應的策略以提高錯誤發生位置判斷的準確率。In this dissertation, the algorithm analysis, architecture design and applications for bitstream processing for video communication systems are presented. Bitstream plays an important role in digital multimedia era since all video content are represented in bitstream and a lot of important and useful information is hidden in it. First, we proposed an efficient video streaming system with the help of the coded bitstream. A new transmission model is proposed for the realization of a content-aware video streaming. We separate enhancement from encoding. The streaming server can adaptively decide quality-enhanced region by selective enhancement according to both object information from the encoding side and user-defined region from the receiver side. Besides, the motion information in the content itself can also be used as reference. The proposed system provides better quality in users' interest regions with no bit-rate or complexity overhead. In addition, we design an MPEG-4 FGS encoder for this streaming system. The computational complexity of FGS coding is analyzed to explore an efficient FGS encoder implementation. We reorder current MPEG-4 FGS coding flow such that the picture-level maximum can be acquired in advance and bit-plane data can be dynamically adapted. It is completely compatible with MPEG-4. With several proposed hardware-oriented optimization approaches, a hardwired FGS block-level processing core is proposed. Next, we present the bitstream parsing analysis and an efficient and flexible bitstream parsing processor design. The critical part in bitstream parsing is explored. We proposed novel approaches to parse the data partitioned bitstreams. An efficient instruction set optimized for bitstream processing, especially for DCT coefficient decoding, is designed. In our implementation, it is integrated into an MPEG-4 video decoding system successfully and achieves real time bitstream decoding under the specification of MPEG-4 Advanced Simple Profile Level 5. Both flexibility and computation power are provided at the same time. Moreover, we perform statistical analysis to model the error propagation in MPEG-4 bitstream with data partitioning. With the help of the symbols in the bitstream and its structure, we explore the error propagation effect under various error detection conditions. It is shown that errors detected in forward section of texture data may be propagated from motion data, while those in DCT coefficients mostly result from themselves. Furthermore, the motion marker is the major error source for several error conditions detected in motion part. According to these characteristics, we propose motion marker assumption and backtracking-based strategies. The experimental results show that more accurate error localization in bitstream domain is achieved.Abstract ix 1 Introduction 1 1.1 Digital Video and Video Compression . . . . . . . . . . . . . . . 1 1.2 Video Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Video Bitstream Processing . . . . . . . . . . . . . . . . . . . . . 5 1.4 Background and Motivation . . . . . . . . . . . . . . . . . . . . 8 1.5 Research Approaches . . . . . . . . . . . . . . . . . . . . . . . . 10 1.6 Dissertation Organization . . . . . . . . . . . . . . . . . . . . . . 11 2 Interactive Content-aware Video Streaming System with Fine Granularity Scalability 13 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 MPEG-4 FGS Functional Description . . . . . . . . . . . . . . . 15 2.3 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.4 Evolution of Transmission Model . . . . . . . . . . . . . . . . . 20 2.5 Proposed Transmission Model for Video Streaming . . . . . . . . 24 2.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . 28 2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3 Architecture Design for MPEG-4 FGS Encoder 39 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.3 Analysis on MPEG-4 FGS Coding Algorithm . . . . . . . . . . . 42 3.3.1 Encoding Operation Profiling . . . . . . . . . . . . . . . 42 3.3.2 FGS Coding Flow Analysis . . . . . . . . . . . . . . . . 45 3.4 Proposed Implementation Approaches for MPEG-4 FGS Encoder 46 3.4.1 Hardware-oriented Optimization Approaches . . . . . . . 46 3.4.2 Support for the Proposed Transmission Model for Video Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.5 Architecture Design and Implementation . . . . . . . . . . . . . . 54 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4 Bitstream Parsing Processor for MPEG-4 Video Decoding System 57 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.3 Bitstream Parsing Analysis and Proposed Algorithms . . . . . . . 62 4.3.1 Codeword Type Distribution . . . . . . . . . . . . . . . . 62 4.3.2 Proposed DCT Coefficients Parsing Approach . . . . . . . 63 4.3.3 Proposed Data Partitioned Parsing Approach . . . . . . . 65 4.4 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.4.1 Proposed Instruction Set . . . . . . . . . . . . . . . . . . 68 4.4.2 Register File Organization . . . . . . . . . . . . . . . . . 72 4.4.3 Bitstream Processor Architecture . . . . . . . . . . . . . 74 4.4.4 Stream Handler . . . . . . . . . . . . . . . . . . . . . . . 79 4.5 Implementation and Comparison . . . . . . . . . . . . . . . . . . 79 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5 Error Localization by Error Propagation Analysis for Data Partitioned Bitstream 83 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5.3 Data Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.3.1 Bitstream Structure . . . . . . . . . . . . . . . . . . . . . 86 5.3.2 Consideration . . . . . . . . . . . . . . . . . . . . . . . . 87 5.4 Error Propagation Analysis . . . . . . . . . . . . . . . . . . . . . 88 5.4.1 Analysis Approach . . . . . . . . . . . . . . . . . . . . . 89 5.4.2 Analysis Results . . . . . . . . . . . . . . . . . . . . . . 91 5.5 Proposed Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . 98 5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6 Conclusions 101 6.1 Principal Contributions . . . . . . . . . . . . . . . . . . . . . . . 101 6.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . 10