Algorithms and Hardware Co-Design of HEVC Intra Encoders

Digital video is becoming extremely important nowadays and its importance has greatly increased in the last two decades. Due to the rapid development of information and communication technologies, the demand for Ultra-High Definition (UHD) video applications is becoming stronger. However, the most prevalent video compression standard H.264/AVC released in 2003 is inefficient when it comes to UHD videos. The increasing desire for superior compression efficiency to H.264/AVC leads to the standardization of High Efficiency Video Coding (HEVC). Compared with the H.264/AVC standard, HEVC offers a double compression ratio at the same level of video quality or substantial improvement of video quality at the same video bitrate. Yet, HE-VC/H.265 possesses superior compression efficiency, its complexity is several times more than H.264/AVC, impeding its high throughput implementation. Currently, most of the researchers have focused merely on algorithm level adaptations of HEVC/H.265 standard to reduce computational intensity without considering the hardware feasibility. What’s more, the exploration of efficient hardware architecture design is not exhaustive. Only a few research works have been conducted to explore efficient hardware architectures of HEVC/H.265 standard. In this dissertation, we investigate efficient algorithm adaptations and hardware architecture design of HEVC intra encoders. We also explore the deep learning approach in mode prediction. From the algorithm point of view, we propose three efficient hardware-oriented algorithm adaptations, including mode reduction, fast coding unit (CU) cost estimation, and group-based CABAC (context-adaptive binary arithmetic coding) rate estimation. Mode reduction aims to reduce mode candidates of each prediction unit (PU) in the rate-distortion optimization (RDO) process, which is both computation-intensive and time-consuming. Fast CU cost estimation is applied to reduce the complexity in rate-distortion (RD) calculation of each CU. Group-based CABAC rate estimation is proposed to parallelize syntax elements processing to greatly improve rate estimation throughput. From the hardware design perspective, a fully parallel hardware architecture of HEVC intra encoder is developed to sustain UHD video compression at 4K@30fps. The fully parallel architecture introduces four prediction engines (PE) and each PE performs the full cycle of mode prediction, transform, quantization, inverse quantization, inverse transform, reconstruction, rate-distortion estimation independently. PU blocks with different PU sizes will be processed by the different prediction engines (PE) simultaneously. Also, an efficient hardware implementation of a group-based CABAC rate estimator is incorporated into the proposed HEVC intra encoder for accurate and high-throughput rate estimation. To take advantage of the deep learning approach, we also propose a fully connected layer based neural network (FCLNN) mode preselection scheme to reduce the number of RDO modes of luma prediction blocks. All angular prediction modes are classified into 7 prediction groups. Each group contains 3-5 prediction modes that exhibit a similar prediction angle. A rough angle detection algorithm is designed to determine the prediction direction of the current block, then a small scale FCLNN is exploited to refine the mode prediction

Error resilience and concealment techniques for high-efficiency video coding

This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods

Approximate and timing-speculative hardware design for high-performance and energy-efficient video processing

Since the end of transistor scaling in 2-D appeared on the horizon, innovative circuit design paradigms have been on the rise to go beyond the well-established and ultraconservative exact computing. Many compute-intensive applications – such as video processing – exhibit an intrinsic error resilience and do not necessarily require perfect accuracy in their numerical operations. Approximate computing (AxC) is emerging as a design alternative to improve the performance and energy-efficiency requirements for many applications by trading its intrinsic error tolerance with algorithm and circuit efficiency. Exact computing also imposes a worst-case timing to the conventional design of hardware accelerators to ensure reliability, leading to an efficiency loss. Conversely, the timing-speculative (TS) hardware design paradigm allows increasing the frequency or decreasing the voltage beyond the limits determined by static timing analysis (STA), thereby narrowing pessimistic safety margins that conventional design methods implement to prevent hardware timing errors. Timing errors should be evaluated by an accurate gate-level simulation, but a significant gap remains: How these timing errors propagate from the underlying hardware all the way up to the entire algorithm behavior, where they just may degrade the performance and quality of service of the application at stake? This thesis tackles this issue by developing and demonstrating a cross-layer framework capable of performing investigations of both AxC (i.e., from approximate arithmetic operators, approximate synthesis, gate-level pruning) and TS hardware design (i.e., from voltage over-scaling, frequency over-clocking, temperature rising, and device aging). The cross-layer framework can simulate both timing errors and logic errors at the gate-level by crossing them dynamically, linking the hardware result with the algorithm-level, and vice versa during the evolution of the application’s runtime. Existing frameworks perform investigations of AxC and TS techniques at circuit-level (i.e., at the output of the accelerator) agnostic to the ultimate impact at the application level (i.e., where the impact is truly manifested), leading to less optimization. Unlike state of the art, the framework proposed offers a holistic approach to assessing the tradeoff of AxC and TS techniques at the application-level. This framework maximizes energy efficiency and performance by identifying the maximum approximation levels at the application level to fulfill the required good enough quality. This thesis evaluates the framework with an 8-way SAD (Sum of Absolute Differences) hardware accelerator operating into an HEVC encoder as a case study. Application-level results showed that the SAD based on the approximate adders achieve savings of up to 45% of energy/operation with an increase of only 1.9% in BD-BR. On the other hand, VOS (Voltage Over-Scaling) applied to the SAD generates savings of up to 16.5% in energy/operation with around 6% of increase in BD-BR. The framework also reveals that the boost of about 6.96% (at 50°) to 17.41% (at 75° with 10- Y aging) in the maximum clock frequency achieved with TS hardware design is totally lost by the processing overhead from 8.06% to 46.96% when choosing an unreliable algorithm to the blocking match algorithm (BMA). We also show that the overhead can be avoided by adopting a reliable BMA. This thesis also shows approximate DTT (Discrete Tchebichef Transform) hardware proposals by exploring a transform matrix approximation, truncation and pruning. The results show that the approximate DTT hardware proposal increases the maximum frequency up to 64%, minimizes the circuit area in up to 43.6%, and saves up to 65.4% in power dissipation. The DTT proposal mapped for FPGA shows an increase of up to 58.9% on the maximum frequency and savings of about 28.7% and 32.2% on slices and dynamic power, respectively compared with stat

Hardware based High Accuracy Integer Motion Estimation and Merge Mode Estimation

학위논문 (박사)-- 서울대학교 대학원 공과대학 전기·컴퓨터공학부, 2017. 8. 이혁재.HEVC는 H.264/AVC 대비 2배의 뛰어난 압축 효율을 가지지만, 많은 압축 기술이 사용됨으로써, 인코더 측의 계산 복잡도를 크게 증가시켰다. HEVC의 높은 계산 복잡도를 줄이기 위한 많은 연구들이 이루어졌지만, 대부분의 연구들은 H.264/AVC를 위한 계산 복잡도 감소 방법을 확장 적용하는 데에 그쳐, 만족스럽지 않은 계산 복잡도 감소 성능을 보이거나, 지나치게 큰 압축 효율 손실을 동반하여 HEVC의 최대 압축 성능을 끌어내지 못했다. 특히 앞서 연구된 하드웨어 기반의 인코더는 실시간 인코더의 실현이 우선되어 압축 효율의 희생이 매우 크다. 그러므로, 본 연구에서는 하드웨어 기반 Inter prediction의 고속화를 이룸과 동시에 HEVC가 가진 압축 성능의 손실을 최소화하고, 실시간 코딩이 가능한 하드웨어 구조를 제안하였다. 본 연구에서 제안한 bottom-up MV 예측 방법은 기존의 공간적, 시간적으로 인접한 PU로부터 MV를 예측하는 방법이 아닌, HEVC의 계층적으로 인접한 PU로부터 MV를 예측하는 방법을 제안하여 MV 예측의 정확도를 큰 폭으로 향상시켰다. 결과적으로 압축 효율의 변화 없이 IME의 계산 복잡도를 67% 감소시킬 수 있었다. 또한, 본 연구에서는 제안된 bottom-up IME 알고리즘을 적용하여 실시간 동작이 가능한 하드웨어 기반의 IME를 제안하였다. 기존의 하드웨어 기반 IME는 고속 IME 알고리즘이 갖는 단계별 의존성으로 인한 idle cycle의 발생과 참조 데이터 접근 문제로 인해, 고속 IME 알고리즘을 사용하지 않거나 또는 하드웨어에 맞게 고속 IME 알고리즘을 수정하였기 때문에 압축 효율의 저하가 수 퍼센트 이상으로 매우 컸다. 그러나 본 연구에서는 고속 IME 알고리즘인 TZS 알고리즘을 채택하여 TZS 알고리즘의 계산 복잡도 감소 성능을 훼손하지 않는 하드웨어 기반의 IME를 제안하였다. 고속 IME 알고리즘을 하드웨어에서 사용하기 위해서 다음 세 가지 사항을 제안하고 하드웨어에 적용하였다. 첫 째로, 고속 IME 알고리즘의 고질적 문제인 idle cycle 발생 문제를 서로 다른 참조 픽쳐와 서로 다른 depth에 대한 IME를 컨텍스트 스위칭을 통해 해결하였다. 둘 째로, 참조 데이터로의 빠르고 자유로운 접근을 위해 참조 데이터의 locality 이용한 multi bank SRAM 구조를 제안하였다. 셋 째로, 지나치게 자유로운 참조 데이터 접근이 발생시키는 대량의 스위칭 mux의 사용을 피하기 위해 탐색 중심을 기준으로 하는 제한된 자유도의 참조 데이터 접근을 제안하였다. 결과 제안된 IME 하드웨어는 HEVC의 모든 블록 크기를 지원하면서, 참조 픽처 4장를 사용하여, 4k UHD 영상을 60fps의 속도로 처리할 수 있으며 이 때 압축 효율의 손실은 0.11%로 거의 나타나지 않는다. 이 때 사용되는 하드웨어 리소스는 1.27M gates이다. HEVC에 새로이 채택된 merge mode estimation은 압축 효율 개선 효과가 뛰어난 새로운 기술이지만, 매 PU 마다 계산 복잡도의 변동 폭이 커서 하드웨어로 구현되는 경우 하드웨어 리소스의 낭비가 많다. 그러므로 본 연구에서는 효율적인 하드웨어 기반 MME 방법과 하드웨어 구조를 함께 제안하였다. 기존 MME 방식은 이웃 PU에 의해 보간 필터 적용 여부가 결정되기 때문에, 보간 필터의 사용률은 50% 이하를 나타낸다. 그럼에도 불구하고 하드웨어는 보간 필터를 사용하는 경우에 맞추어 설계되어왔기 때문에 하드웨어 리소스의 사용 효율이 낮았다. 본 연구에서는 가장 하드웨어 리소스를 많이 사용하는 세로 방향 보간 필터를 절반 크기로 줄인 두 개의 데이터 패스를 갖는 MME 하드웨어 구조를 제안하였고, 높은 하드웨어 사용률을 유지하면서 압축 효율 손실을 최소화 하는 merge 후보 할당 알고리즘을 제안하였다. 결과, 기존 하드웨어 기반 MME 보다 24% 적은 하드웨어 리소스를 사용하면서도 7.4% 더 빠른 수행 시간을 갖는 새로운 하드웨어 기반의 MME를 달성하였다. 제안된 하드웨어 기반의 MME는 460.8K gates의 하드웨어 리소스를 사용하고 4k UHD 영상을 30 fps의 속도로 처리할 수 있다.제 1 장 서 론 1 1.1 연구 배경 1 1.2 연구 내용 3 1.3 공통 실험 환경 5 1.4 논문 구성 6 제 2 장 관련 연구 7 2.1 HEVC 표준 7 2.1.1 쿼드-트리 기반의 계층적 블록 구조 7 2.1.2 HEVC 의 Inter Prediction 9 2.2 화면 간 예측의 속도 향상을 위한 이전 연구 17 2.2.1 고속 Integer Motion Estimation 알고리즘 17 2.2.2 고속 Merge Mode Estimation 알고리즘 20 2.3 화면 간 예측 하드웨어 구조에 대한 이전 연구 21 2.3.1 하드웨어 기반 Integer Motion Estimation 연구 21 2.3.2 하드웨어 기반 Merge Mode Estimation 연구 25 제 3 장 Bottom-up Integer Motion Estimation 26 3.1 서로 다른 계층 간의 Motion Vector 관계 관찰 26 3.1.1 서로 다른 계층 간의 Motion Vector 관계 분석 26 3.1.2 Top-down 및 Bottom-up 방향의 Motion Vector 관계 분석 30 3.2 Bottom-up Motion Vector Prediction 33 3.3 Bottom-up Integer Motion Estimation 37 3.3.1 Bottom-up Integer Motion Estimation - Single MVP 37 3.3.2 Bottom-up Integer Motion Estimation - Multiple MVP 38 3.4 실험 결과 40 제 4 장 하드웨어 기반 Integer Motion Estimation 46 4.1 Bottom-up Integer Motion Estimation의 하드웨어 적용 46 4.2 하드웨어를 위한 수정된 Test Zone Search 47 4.2.1 SAD-tree를 활용한 CU 내 PU의 병렬 처리 47 4.2.2 Grid 기반의 Sampled Raster Search 53 4.2.3 서로 다른 PU 간의 중복 연산 제거 55 4.3 Idle cycle이 감소된 5-stage 파이프라인 스케줄 56 4.3.1 파이프라인 스테이지 별 동작 56 4.3.2 Test Zone Search의 의존성으로 인한 Idle cycle 도입 58 4.3.3 컨텍스트 스위칭을 통한 Idle cycle 감소 60 4.4 고속 동작을 위한 참조 데이터 공급 방법 63 4.4.1 참조 데이터 접근 패턴 및 접근 지연 발생 시 문제점 63 4.4.2 Search Points의 Locality를 활용한 참조 데이터 접근 64 4.4.3 단일 cycle 참조 데이터 접근을 위한 Multi Bank 메모리 구조 66 4.4.4 참조 데이터 접근의 자유도 제어를 통한 스위칭 복잡도 저감 방법 68 4.5 하드웨어 구조 72 4.5.1 전체 하드웨어 구조 72 4.5.2 하드웨어 세부 스케줄 78 4.6 하드웨어 구현 결과 및 실험 결과 82 4.6.1 하드웨어 구현 결과 82 4.6.2 수행 시간 및 압축 효율 84 4.6.3 제안 방법 적용 단계 별 성능 변화 88 4.6.4 이전 연구와의 비교 91 제 5 장 하드웨어 기반 Merge Mode Estimation 96 5.1 기존 Merge Mode Estimation의 하드웨어 관점에서의 고찰 96 5.1.1 기존 Merge Mode Estimation 96 5.1.2 기존 Merge Mode Estimation 하드웨어 구조 및 분석 98 5.1.3 기존 Merge Mode Estimation의 하드웨어 사용률 저하 문제 100 5.2 연산량 변동폭을 감소시킨 새로운 Merge Mode Estimation 103 5.3 새로운 Merge Mode Estimation의 하드웨어 구현 106 5.3.1 후보 타입 별 독립적 path를 갖는 하드웨어 구조 106 5.3.2 하드웨어 사용률을 높이기 위한 적응적 후보 할당 방법 109 5.3.3 적응적 후보 할당 방법을 적용한 하드웨어 스케줄 111 5.4 실험 결과 및 하드웨어 구현 결과 114 5.4.1 수행 시간 및 압축 효율 변화 114 5.4.2 하드웨어 구현 결과 116 제 6 장 Overall Inter Prediction 117 6.1 CTU 단위의 3-stage 파이프라인 Inter Prediction 117 6.2 Two-way Encoding Order 119 6.2.1 Top-down 인코딩 순서와 Bottom-up 인코딩 순서 119 6.2.2 기존 고속 알고리즘과 호환되는 Two-way Encoding Order 120 6.2.3 기존 고속 알고리즘과 결합 및 비교 실험 결과 123 제 7 장 Next Generation Video Coding으로의 확장 127 7.1 Bottom-up Motion Vector Prediction의 확장 127 7.2 Bottom-up Integer Motion Estimation의 확장 130 제 8 장 결 론 132Docto

Data-driven visual quality estimation using machine learning

Heutzutage werden viele visuelle Inhalte erstellt und sind zugänglich, was auf Verbesserungen der Technologie wie Smartphones und das Internet zurückzuführen ist. Es ist daher notwendig, die von den Nutzern wahrgenommene Qualität zu bewerten, um das Erlebnis weiter zu verbessern. Allerdings sind nur wenige der aktuellen Qualitätsmodelle speziell für höhere Auflösungen konzipiert, sagen mehr als nur den Mean Opinion Score vorher oder nutzen maschinelles Lernen. Ein Ziel dieser Arbeit ist es, solche maschinellen Modelle für höhere Auflösungen mit verschiedenen Datensätzen zu trainieren und zu evaluieren. Als Erstes wird eine objektive Analyse der Bildqualität bei höheren Auflösungen durchgeführt. Die Bilder wurden mit Video-Encodern komprimiert, hierbei weist AV1 die beste Qualität und Kompression auf. Anschließend werden die Ergebnisse eines Crowd-Sourcing-Tests mit einem Labortest bezüglich Bildqualität verglichen. Weiterhin werden auf Deep Learning basierende Modelle für die Vorhersage von Bild- und Videoqualität beschrieben. Das auf Deep Learning basierende Modell ist aufgrund der benötigten Ressourcen für die Vorhersage der Videoqualität in der Praxis nicht anwendbar. Aus diesem Grund werden pixelbasierte Videoqualitätsmodelle vorgeschlagen und ausgewertet, die aussagekräftige Features verwenden, welche Bild- und Bewegungsaspekte abdecken. Diese Modelle können zur Vorhersage von Mean Opinion Scores für Videos oder sogar für anderer Werte im Zusammenhang mit der Videoqualität verwendet werden, wie z.B. einer Bewertungsverteilung. Die vorgestellte Modellarchitektur kann auf andere Videoprobleme angewandt werden, wie z.B. Videoklassifizierung, Vorhersage der Qualität von Spielevideos, Klassifikation von Spielegenres oder der Klassifikation von Kodierungsparametern. Ein wichtiger Aspekt ist auch die Verarbeitungszeit solcher Modelle. Daher wird ein allgemeiner Ansatz zur Beschleunigung von State-of-the-Art-Videoqualitätsmodellen vorgestellt, der zeigt, dass ein erheblicher Teil der Verarbeitungszeit eingespart werden kann, während eine ähnliche Vorhersagegenauigkeit erhalten bleibt. Die Modelle sind als Open Source veröffentlicht, so dass die entwickelten Frameworks für weitere Forschungsarbeiten genutzt werden können. Außerdem können die vorgestellten Ansätze als Bausteine für neuere Medienformate verwendet werden.Today a lot of visual content is accessible and produced, due to improvements in technology such as smartphones and the internet. This results in a need to assess the quality perceived by users to further improve the experience. However, only a few of the state-of-the-art quality models are specifically designed for higher resolutions, predict more than mean opinion score, or use machine learning. One goal of the thesis is to train and evaluate such machine learning models of higher resolutions with several datasets. At first, an objective evaluation of image quality in case of higher resolutions is performed. The images are compressed using video encoders, and it is shown that AV1 is best considering quality and compression. This evaluation is followed by the analysis of a crowdsourcing test in comparison with a lab test investigating image quality. Afterward, deep learning-based models for image quality prediction and an extension for video quality are proposed. However, the deep learning-based video quality model is not practically usable because of performance constrains. For this reason, pixel-based video quality models using well-motivated features covering image and motion aspects are proposed and evaluated. These models can be used to predict mean opinion scores for videos, or even to predict other video quality-related information, such as a rating distributions. The introduced model architecture can be applied to other video problems, such as video classification, gaming video quality prediction, gaming genre classification or encoding parameter estimation. Furthermore, one important aspect is the processing time of such models. Hence, a generic approach to speed up state-of-the-art video quality models is introduced, which shows that a significant amount of processing time can be saved, while achieving similar prediction accuracy. The models have been made publicly available as open source so that the developed frameworks can be used for further research. Moreover, the presented approaches may be usable as building blocks for newer media formats

End to end Multi-Objective Optimisation of H.264 and HEVC Codecs

All multimedia devices now incorporate video CODECs that comply with international video coding standards such as H.264 / MPEG4-AVC and the new High Efficiency Video Coding Standard (HEVC) otherwise known as H.265. Although the standard CODECs have been designed to include algorithms with optimal efficiency, large number of coding parameters can be used to fine tune their operation, within known constraints of for e.g., available computational power, bandwidth, consumer QoS requirements, etc. With large number of such parameters involved, determining which parameters will play a significant role in providing optimal quality of service within given constraints is a further challenge that needs to be met. Further how to select the values of the significant parameters so that the CODEC performs optimally under the given constraints is a further important question to be answered. This thesis proposes a framework that uses machine learning algorithms to model the performance of a video CODEC based on the significant coding parameters. Means of modelling both the Encoder and Decoder performance is proposed. We define objective functions that can be used to model the performance related properties of a CODEC, i.e., video quality, bit-rate and CPU time. We show that these objective functions can be practically utilised in video Encoder/Decoder designs, in particular in their performance optimisation within given operational and practical constraints. A Multi-objective Optimisation framework based on Genetic Algorithms is thus proposed to optimise the performance of a video codec. The framework is designed to jointly minimize the CPU Time, Bit-rate and to maximize the quality of the compressed video stream. The thesis presents the use of this framework in the performance modelling and multi-objective optimisation of the most widely used video coding standard in practice at present, H.264 and the latest video coding standard, H.265/HEVC. When a communication network is used to transmit video, performance related parameters of the communication channel will impact the end-to-end performance of the video CODEC. Network delays and packet loss will impact the quality of the video that is received at the decoder via the communication channel, i.e., even if a video CODEC is optimally configured network conditions will make the experience sub-optimal. Given the above the thesis proposes a design, integration and testing of a novel approach to simulating a wired network and the use of UDP protocol for the transmission of video data. This network is subsequently used to simulate the impact of packet loss and network delays on optimally coded video based on the framework previously proposed for the modelling and optimisation of video CODECs. The quality of received video under different levels of packet loss and network delay is simulated, concluding the impact on transmitted video based on their content and features

Video compression algorithms for HEVC and beyond

PhDDue to the increasing number of new services and devices that allow the creation, distribution and consumption of video content, the amount of video information being transmitted all over the world is constantly growing. Video compression technology is essential to cope with the ever increasing volume of digital video data being distributed in today's networks, as more e cient video compression techniques allow support for higher volumes of video data under the same memory/bandwidth constraints. This is especially relevant with the introduction of new and more immersive video formats associated with signi cantly higher amounts of data. In this thesis, novel techniques for improving the e ciency of current and future video coding technologies are investigated. Several aspects that in uence the way conventional video coding methods work are considered. In particular, the properties and limitations of the Human Visual System are exploited to tune the performance of video encoders towards better subjective quality. Additionally, it is shown how the visibility of speci c types of visual artefacts can be prevented during the video encoding process, in order to avoid subjective quality degradations in the compressed content. Techniques for higher video compression e ciency are also explored, targeting to improve the compression capabilities of state-of-the-art video coding standards. Finally, the application of video coding technologies to practical use-cases is considered. Accurate estimation models are devised to control the encoding time and bit rate associated with compressed video signals, in order to meet speci c encoding time and transmission time restrictions

High throughput image compression and decompression on GPUs

Diese Arbeit befasst sich mit der Entwicklung eines GPU-freundlichen, intra-only, Wavelet-basierten Videokompressionsverfahrens mit hohem Durchsatz, das für visuell verlustfreie Anwendungen optimiert ist. Ausgehend von der Beobachtung, dass der JPEG 2000 Entropie-Kodierer ein Flaschenhals ist, werden verschiedene algorithmische Änderungen vorgeschlagen und bewertet. Zunächst wird der JPEG 2000 Selective Arithmetic Coding Mode auf der GPU realisiert, wobei sich die Erhöhung des Durchsatzes hierdurch als begrenzt zeigt. Stattdessen werden zwei nicht standard-kompatible Änderungen vorgeschlagen, die (1) jede Bitebebene in nur einem einzelnen Pass verarbeiten (Single-Pass-Modus) und (2) einen echten Rohcodierungsmodus einführen, der sample-weise parallelisierbar ist und keine aufwendige Kontextmodellierung erfordert. Als nächstes wird ein alternativer Entropiekodierer aus der Literatur, der Bitplane Coder with Parallel Coefficient Processing (BPC-PaCo), evaluiert. Er gibt Signaladaptivität zu Gunsten von höherer Parallelität auf und daher wird hier untersucht und gezeigt, dass ein aus verschiedensten Testsequenzen gemitteltes statisches Wahrscheinlichkeitsmodell eine kompetitive Kompressionseffizienz erreicht. Es wird zudem eine Kombination von BPC-PaCo mit dem Single-Pass-Modus vorgeschlagen, der den Speedup gegenüber dem JPEG 2000 Entropiekodierer von 2,15x (BPC-PaCo mit zwei Pässen) auf 2,6x (BPC-PaCo mit Single-Pass-Modus) erhöht auf Kosten eines um 0,3 dB auf 1,0 dB erhöhten Spitzen-Signal-Rausch-Verhältnis (PSNR). Weiter wird ein paralleler Algorithmus zur Post-Compression Ratenkontrolle vorgestellt sowie eine parallele Codestream-Erstellung auf der GPU. Es wird weiterhin ein theoretisches Laufzeitmodell formuliert, das es durch Benchmarking von einer GPU ermöglicht die Laufzeit einer Routine auf einer anderen GPU vorherzusagen. Schließlich wird der erste JPEG XS GPU Decoder vorgestellt und evaluiert. JPEG XS wurde als Low Complexity Codec konzipiert und forderte erstmals explizit GPU-Freundlichkeit bereits im Call for Proposals. Ab Bitraten über 1 bpp ist der Decoder etwa 2x schneller im Vergleich zu JPEG 2000 und 1,5x schneller als der schnellste hier vorgestellte Entropiekodierer (BPC-PaCo mit Single-Pass-Modus). Mit einer GeForce GTX 1080 wird ein Decoder Durchsatz von rund 200 fps für eine UHD-4:4:4-Sequenz erreicht.This work investigates possibilities to create a high throughput, GPU-friendly, intra-only, Wavelet-based video compression algorithm optimized for visually lossless applications. Addressing the key observation that JPEG 2000’s entropy coder is a bottleneck and might be overly complex for a high bit rate scenario, various algorithmic alterations are proposed. First, JPEG 2000’s Selective Arithmetic Coding mode is realized on the GPU, but the gains in terms of an increased throughput are shown to be limited. Instead, two independent alterations not compliant to the standard are proposed, that (1) give up the concept of intra-bit plane truncation points and (2) introduce a true raw-coding mode that is fully parallelizable and does not require any context modeling. Next, an alternative block coder from the literature, the Bitplane Coder with Parallel Coefficient Processing (BPC-PaCo), is evaluated. Since it trades signal adaptiveness for increased parallelism, it is shown here how a stationary probability model averaged from a set of test sequences yields competitive compression efficiency. A combination of BPC-PaCo with the single-pass mode is proposed and shown to increase the speedup with respect to the original JPEG 2000 entropy coder from 2.15x (BPC-PaCo with two passes) to 2.6x (proposed BPC-PaCo with single-pass mode) at the marginal cost of increasing the PSNR penalty by 0.3 dB to at most 1 dB. Furthermore, a parallel algorithm is presented that determines the optimal code block bit stream truncation points (given an available bit rate budget) and builds the entire code stream on the GPU, reducing the amount of data that has to be transferred back into host memory to a minimum. A theoretical runtime model is formulated that allows, based on benchmarking results on one GPU, to predict the runtime of a kernel on another GPU. Lastly, the first ever JPEG XS GPU-decoder realization is presented. JPEG XS was designed to be a low complexity codec and for the first time explicitly demanded GPU-friendliness already in the call for proposals. Starting at bit rates above 1 bpp, the decoder is around 2x faster compared to the original JPEG 2000 and 1.5x faster compared to JPEG 2000 with the fastest evaluated entropy coder (BPC-PaCo with single-pass mode). With a GeForce GTX 1080, a decoding throughput of around 200 fps is achieved for a UHD 4:4:4 sequence

Robust Subspace Estimation via Low-Rank and Sparse Decomposition and Applications in Computer Vision

PhDRecent advances in robust subspace estimation have made dimensionality reduction and noise and outlier suppression an area of interest for research, along with continuous improvements in computer vision applications. Due to the nature of image and video signals that need a high dimensional representation, often storage, processing, transmission, and analysis of such signals is a difficult task. It is therefore desirable to obtain a low-dimensional representation for such signals, and at the same time correct for corruptions, errors, and outliers, so that the signals could be readily used for later processing. Major recent advances in low-rank modelling in this context were initiated by the work of Cand`es et al. [17] where the authors provided a solution for the long-standing problem of decomposing a matrix into low-rank and sparse components in a Robust Principal Component Analysis (RPCA) framework. However, for computer vision applications RPCA is often too complex, and/or may not yield desirable results. The low-rank component obtained by the RPCA has usually an unnecessarily high rank, while in certain tasks lower dimensional representations are required. The RPCA has the ability to robustly estimate noise and outliers and separate them from the low-rank component, by a sparse part. But, it has no mechanism of providing an insight into the structure of the sparse solution, nor a way to further decompose the sparse part into a random noise and a structured sparse component that would be advantageous in many computer vision tasks. As videos signals are usually captured by a camera that is moving, obtaining a low-rank component by RPCA becomes impossible. In this thesis, novel Approximated RPCA algorithms are presented, targeting different shortcomings of the RPCA. The Approximated RPCA was analysed to identify the most time consuming RPCA solutions, and replace them with simpler yet tractable alternative solutions. The proposed method is able to obtain the exact desired rank for the low-rank component while estimating a global transformation to describe camera-induced motion. Furthermore, it is able to decompose the sparse part into a foreground sparse component, and a random noise part that contains no useful information for computer vision processing. The foreground sparse component is obtained by several novel structured sparsity-inducing norms, that better encapsulate the needed pixel structure in visual signals. Moreover, algorithms for reducing complexity of low-rank estimation have been proposed that achieve significant complexity reduction without sacrificing the visual representation of video and image information. The proposed algorithms are applied to several fundamental computer vision tasks, namely, high efficiency video coding, batch image alignment, inpainting, and recovery, video stabilisation, background modelling and foreground segmentation, robust subspace clustering and motion estimation, face recognition, and ultra high definition image and video super-resolution. The algorithms proposed in this thesis including batch image alignment and recovery, background modelling and foreground segmentation, robust subspace clustering and motion segmentation, and ultra high definition image and video super-resolution achieve either state-of-the-art or comparable results to existing methods