Real-time complexity constrained encoding

Complex software appliances can be deployed on hardware with limited available computational resources. This computational boundary puts an additional constraint on software applications. This can be an issue for real-time applications with a fixed time constraint such as low delay video encoding. In the context of High Efficiency Video Coding (HEVC), a limited number of publications have focused on controlling the complexity of an HEVC video encoder. In this paper, a technique is proposed to control complexity by deciding between 2Nx2N merge mode and full encoding, at different Coding Unit (CU) depths. The technique is demonstrated in two encoders. The results demonstrate fast convergence to a given complexity threshold, and a limited loss in rate-distortion performance (on average 2.84% Bjontegaard delta rate for 40% complexity reduction)

Rate-Accuracy Trade-Off In Video Classification With Deep Convolutional Neural Networks

Advanced video classification systems decode video frames to derive the necessary texture and motion representations for ingestion and analysis by spatio-temporal deep convolutional neural networks (CNNs). However, when considering visual Internet-of-Things applications, surveillance systems and semantic crawlers of large video repositories, the video capture and the CNN-based semantic analysis parts do not tend to be co-located. This necessitates the transport of compressed video over networks and incurs significant overhead in bandwidth and energy consumption, thereby significantly undermining the deployment potential of such systems. In this paper, we investigate the trade-off between the encoding bitrate and the achievable accuracy of CNN-based video classification models that directly ingest AVC/H.264 and HEVC encoded videos. Instead of retaining entire compressed video bitstreams and applying complex optical flow calculations prior to CNN processing, we only retain motion vector and select texture information at significantly-reduced bitrates and apply no additional processing prior to CNN ingestion. Based on three CNN architectures and two action recognition datasets, we achieve 11%-94% saving in bitrate with marginal effect on classification accuracy. A model-based selection between multiple CNNs increases these savings further, to the point where, if up to 7% loss of accuracy can be tolerated, video classification can take place with as little as 3 kbps for the transport of the required compressed video information to the system implementing the CNN models

Dynamic Switching of GOP Configurations in High Efficiency Video Coding (HEVC) using Relational Databases for Multi-objective Optimization

Our current technological era is flooded with smart devices that provide significant computational resources that require optimal video communications solutions. Optimal and dynamic management of video bitrate, quality and energy needs to take into account their inter-dependencies. With emerging network generations providing higher bandwidth rates, there is also a growing need to communicate video with the best quality subject to the availability of resources such as computational power and available bandwidth. Similarly, for accommodating multiple users, there is a need to minimize bitrate requirements while sustaining video quality for reasonable encoding times. This thesis focuses on providing an efficient mechanism for deriving optimal solutions for High Efficiency Video Coding (HEVC) based on dynamic switching of GOP configurations. The approach provides a basic system for multi-objective optimization approach with constraints on power, video quality and bitrate. This is accomplished by utilizing a recently introduced framework known as Dynamically Reconfigurable Architectures for Time-varying Image Constraints (DRASTIC) in HEVC/H.265 encoder with six different GOP configurations to support optimization modes for minimum rate, maximum quality and minimum computational time (minimum energy in constant power configuration) mode of operation. Pareto-optimal GOP configurations are used in implementing the DRASTIC modes. Additionally, this thesis also presents a relational database formulation for supporting multiple devices that are characterized by different screen resolutions and computational resources. This approach is applicable to internet-based video streaming to different devices where the videos have been pre-compressed. Here, the video configuration modes are determined based on the application of database queries applied to relational databases. The database queries are used to retrieve a Pareto-optimal configuration based on real-time user requirements, device, and network constraints