5,477 research outputs found

    OVSNet : Towards One-Pass Real-Time Video Object Segmentation

    Full text link
    Video object segmentation aims at accurately segmenting the target object regions across consecutive frames. It is technically challenging for coping with complicated factors (e.g., shape deformations, occlusion and out of the lens). Recent approaches have largely solved them by using backforth re-identification and bi-directional mask propagation. However, their methods are extremely slow and only support offline inference, which in principle cannot be applied in real time. Motivated by this observation, we propose a efficient detection-based paradigm for video object segmentation. We propose an unified One-Pass Video Segmentation framework (OVS-Net) for modeling spatial-temporal representation in a unified pipeline, which seamlessly integrates object detection, object segmentation, and object re-identification. The proposed framework lends itself to one-pass inference that effectively and efficiently performs video object segmentation. Moreover, we propose a maskguided attention module for modeling the multi-scale object boundary and multi-level feature fusion. Experiments on the challenging DAVIS 2017 demonstrate the effectiveness of the proposed framework with comparable performance to the state-of-the-art, and the great efficiency about 11.5 FPS towards pioneering real-time work to our knowledge, more than 5 times faster than other state-of-the-art methods.Comment: 10 pages, 6 figure

    Fast Multi-frame Stereo Scene Flow with Motion Segmentation

    Full text link
    We propose a new multi-frame method for efficiently computing scene flow (dense depth and optical flow) and camera ego-motion for a dynamic scene observed from a moving stereo camera rig. Our technique also segments out moving objects from the rigid scene. In our method, we first estimate the disparity map and the 6-DOF camera motion using stereo matching and visual odometry. We then identify regions inconsistent with the estimated camera motion and compute per-pixel optical flow only at these regions. This flow proposal is fused with the camera motion-based flow proposal using fusion moves to obtain the final optical flow and motion segmentation. This unified framework benefits all four tasks - stereo, optical flow, visual odometry and motion segmentation leading to overall higher accuracy and efficiency. Our method is currently ranked third on the KITTI 2015 scene flow benchmark. Furthermore, our CPU implementation runs in 2-3 seconds per frame which is 1-3 orders of magnitude faster than the top six methods. We also report a thorough evaluation on challenging Sintel sequences with fast camera and object motion, where our method consistently outperforms OSF [Menze and Geiger, 2015], which is currently ranked second on the KITTI benchmark.Comment: 15 pages. To appear at IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). Our results were submitted to KITTI 2015 Stereo Scene Flow Benchmark in November 201

    StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences

    Full text link
    Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. The inherent ambiguity introduced by occlusions directly violates the brightness constancy constraint and considerably hinders pixel-to-pixel matching. To address this issue, multi-frame optical flow methods leverage adjacent frames to mitigate the local ambiguity. Nevertheless, prior multi-frame methods predominantly adopt recursive flow estimation, resulting in a considerable computational overlap. In contrast, we propose a streamlined in-batch framework that eliminates the need for extensive redundant recursive computations while concurrently developing effective spatio-temporal modeling approaches under in-batch estimation constraints. Specifically, we present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. Furthermore, we introduce an efficient Integrative Spatio-temporal Coherence (ISC) modeling method for effective spatio-temporal modeling during the encoding phase, which introduces no additional parameter overhead. Additionally, we devise a Global Temporal Regressor (GTR) that effectively explores temporal relations during decoding. Benefiting from the efficient SIM pipeline and effective modules, StreamFlow not only excels in terms of performance on the challenging KITTI and Sintel datasets, with particular improvement in occluded areas but also attains a remarkable 63.82%63.82\% enhancement in speed compared with previous multi-frame methods. The code will be available soon at https://github.com/littlespray/StreamFlow

    ๋น„๋””์˜ค ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„์„ ์œ„ํ•œ ๋‹ค์ค‘ ๋ฒกํ„ฐ ๊ธฐ๋ฐ˜์˜ MEMC ๋ฐ ์‹ฌ์ธต CNN

    Get PDF
    ํ•™์œ„๋…ผ๋ฌธ (๋ฐ•์‚ฌ)-- ์„œ์šธ๋Œ€ํ•™๊ต ๋Œ€ํ•™์› : ๊ณต๊ณผ๋Œ€ํ•™ ์ „๊ธฐยท์ •๋ณด๊ณตํ•™๋ถ€, 2019. 2. ์ดํ˜์žฌ.Block-based hierarchical motion estimations are widely used and are successful in generating high-quality interpolation. However, it still fails in the motion estimation of small objects when a background region moves in a different direction. This is because the motion of small objects is neglected by the down-sampling and over-smoothing operations at the top level of image pyramids in the maximum a posterior (MAP) method. Consequently, the motion vector of small objects cannot be detected at the bottom level, and therefore, the small objects often appear deformed in an interpolated frame. This thesis proposes a novel algorithm that preserves the motion vector of the small objects by adding a secondary motion vector candidate that represents the movement of the small objects. This additional candidate is always propagated from the top to the bottom layers of the image pyramid. Experimental results demonstrate that the intermediate frame interpolated by the proposed algorithm significantly improves the visual quality when compared with conventional MAP-based frame interpolation. In motion compensated frame interpolation, a repetition pattern in an image makes it difficult to derive an accurate motion vector because multiple similar local minima exist in the search space of the matching cost for motion estimation. In order to improve the accuracy of motion estimation in a repetition region, this thesis attempts a semi-global approach that exploits both local and global characteristics of a repetition region. A histogram of the motion vector candidates is built by using a voter based voting system that is more reliable than an elector based voting system. Experimental results demonstrate that the proposed method significantly outperforms the previous local approach in term of both objective peak signal-to-noise ratio (PSNR) and subjective visual quality. In video frame interpolation or motion-compensated frame rate up-conversion (MC-FRUC), motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this research presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates bidirectional motion vectors from two unidirectional motion vector fields (forward and backward) obtained from the unidirectional motion estimations. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in the case when the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly. Video frame interpolation with a deep convolutional neural network (CNN) is also investigated in this thesis. Optical flow and video frame interpolation are considered as a chicken-egg problem such that one problem affects the other and vice versa. This thesis presents a stack of networks that are trained to estimate intermediate optical flows from the very first intermediate synthesized frame and later the very end interpolated frame is generated by the second synthesis network that is fed by stacking the very first one and two learned intermediate optical flows based warped frames. The primary benefit is that it glues two problems into one comprehensive framework that learns altogether by using both an analysis-by-synthesis technique for optical flow estimation and vice versa, CNN kernels based synthesis-by-analysis. The proposed network is the first attempt to bridge two branches of previous approaches, optical flow based synthesis and CNN kernels based synthesis into a comprehensive network. Experiments are carried out with various challenging datasets, all showing that the proposed network outperforms the state-of-the-art methods with significant margins for video frame interpolation and the estimated optical flows are accurate for challenging movements. The proposed deep video frame interpolation network to post-processing is applied to the improvement of the coding efficiency of the state-of-art video compress standard, HEVC/H.265 and experimental results prove the efficiency of the proposed network.๋ธ”๋ก ๊ธฐ๋ฐ˜ ๊ณ„์ธต์  ์›€์ง์ž„ ์ถ”์ •์€ ๊ณ ํ™”์งˆ์˜ ๋ณด๊ฐ„ ์ด๋ฏธ์ง€๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ์–ด ํญ๋„“๊ฒŒ ์‚ฌ์šฉ๋˜๊ณ  ์žˆ๋‹ค. ํ•˜์ง€๋งŒ, ๋ฐฐ๊ฒฝ ์˜์—ญ์ด ์›€์ง์ผ ๋•Œ, ์ž‘์€ ๋ฌผ์ฒด์— ๋Œ€ํ•œ ์›€์ง์ž„ ์ถ”์ • ์„ฑ๋Šฅ์€ ์—ฌ์ „ํžˆ ์ข‹์ง€ ์•Š๋‹ค. ์ด๋Š” maximum a posterior (MAP) ๋ฐฉ์‹์œผ๋กœ ์ด๋ฏธ์ง€ ํ”ผ๋ผ๋ฏธ๋“œ์˜ ์ตœ์ƒ์œ„ ๋ ˆ๋ฒจ์—์„œ down-sampling๊ณผ over-smoothing์œผ๋กœ ์ธํ•ด ์ž‘์€ ๋ฌผ์ฒด์˜ ์›€์ง์ž„์ด ๋ฌด์‹œ๋˜๊ธฐ ๋•Œ๋ฌธ์ด๋‹ค. ๊ฒฐ๊ณผ์ ์œผ๋กœ ์ด๋ฏธ์ง€ ํ”ผ๋ผ๋ฏธ๋“œ์˜ ์ตœํ•˜์œ„ ๋ ˆ๋ฒจ์—์„œ ์ž‘์€ ๋ฌผ์ฒด์˜ ์›€์ง์ž„ ๋ฒกํ„ฐ๋Š” ๊ฒ€์ถœ๋  ์ˆ˜ ์—†์–ด ๋ณด๊ฐ„ ์ด๋ฏธ์ง€์—์„œ ์ž‘์€ ๋ฌผ์ฒด๋Š” ์ข…์ข… ๋ณ€ํ˜•๋œ ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ธ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์—์„œ๋Š” ์ž‘์€ ๋ฌผ์ฒด์˜ ์›€์ง์ž„์„ ๋‚˜ํƒ€๋‚ด๋Š” 2์ฐจ ์›€์ง์ž„ ๋ฒกํ„ฐ ํ›„๋ณด๋ฅผ ์ถ”๊ฐ€ํ•˜์—ฌ ์ž‘์€ ๋ฌผ์ฒด์˜ ์›€์ง์ž„ ๋ฒกํ„ฐ๋ฅผ ๋ณด์กดํ•˜๋Š” ์ƒˆ๋กœ์šด ์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ์ œ์•ˆํ•œ๋‹ค. ์ถ”๊ฐ€๋œ ์›€์ง์ž„ ๋ฒกํ„ฐ ํ›„๋ณด๋Š” ํ•ญ์ƒ ์ด๋ฏธ์ง€ ํ”ผ๋ผ๋ฏธ๋“œ์˜ ์ตœ์ƒ์œ„์—์„œ ์ตœํ•˜์œ„ ๋ ˆ๋ฒจ๋กœ ์ „ํŒŒ๋œ๋‹ค. ์‹คํ—˜ ๊ฒฐ๊ณผ๋Š” ์ œ์•ˆ๋œ ์•Œ๊ณ ๋ฆฌ์ฆ˜์˜ ๋ณด๊ฐ„ ์ƒ์„ฑ ํ”„๋ ˆ์ž„์ด ๊ธฐ์กด MAP ๊ธฐ๋ฐ˜ ๋ณด๊ฐ„ ๋ฐฉ์‹์œผ๋กœ ์ƒ์„ฑ๋œ ํ”„๋ ˆ์ž„๋ณด๋‹ค ์ด๋ฏธ์ง€ ํ™”์งˆ์ด ์ƒ๋‹นํžˆ ํ–ฅ์ƒ๋จ์„ ๋ณด์—ฌ์ค€๋‹ค. ์›€์ง์ž„ ๋ณด์ƒ ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„์—์„œ, ์ด๋ฏธ์ง€ ๋‚ด์˜ ๋ฐ˜๋ณต ํŒจํ„ด์€ ์›€์ง์ž„ ์ถ”์ •์„ ์œ„ํ•œ ์ •ํ•ฉ ์˜ค์ฐจ ํƒ์ƒ‰ ์‹œ ๋‹ค์ˆ˜์˜ ์œ ์‚ฌ local minima๊ฐ€ ์กด์žฌํ•˜๊ธฐ ๋•Œ๋ฌธ์— ์ •ํ™•ํ•œ ์›€์ง์ž„ ๋ฒกํ„ฐ ์œ ๋„๋ฅผ ์–ด๋ ต๊ฒŒ ํ•œ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์€ ๋ฐ˜๋ณต ํŒจํ„ด์—์„œ์˜ ์›€์ง์ž„ ์ถ”์ •์˜ ์ •ํ™•๋„๋ฅผ ํ–ฅ์ƒ์‹œํ‚ค๊ธฐ ์œ„ํ•ด ๋ฐ˜๋ณต ์˜์—ญ์˜ localํ•œ ํŠน์„ฑ๊ณผ globalํ•œ ํŠน์„ฑ์„ ๋™์‹œ์— ํ™œ์šฉํ•˜๋Š” semi-globalํ•œ ์ ‘๊ทผ์„ ์‹œ๋„ํ•œ๋‹ค. ์›€์ง์ž„ ๋ฒกํ„ฐ ํ›„๋ณด์˜ ํžˆ์Šคํ† ๊ทธ๋žจ์€ ์„ ๊ฑฐ ๊ธฐ๋ฐ˜ ํˆฌํ‘œ ์‹œ์Šคํ…œ๋ณด๋‹ค ์‹ ๋ขฐํ•  ์ˆ˜ ์žˆ๋Š” ์œ ๊ถŒ์ž ๊ธฐ๋ฐ˜ ํˆฌํ‘œ ์‹œ์Šคํ…œ ๊ธฐ๋ฐ˜์œผ๋กœ ํ˜•์„ฑ๋œ๋‹ค. ์‹คํ—˜ ๊ฒฐ๊ณผ๋Š” ์ œ์•ˆ๋œ ๋ฐฉ๋ฒ•์ด ์ด์ „์˜ localํ•œ ์ ‘๊ทผ๋ฒ•๋ณด๋‹ค peak signal-to-noise ratio (PSNR)์™€ ์ฃผ๊ด€์  ํ™”์งˆ ํŒ๋‹จ ๊ด€์ ์—์„œ ์ƒ๋‹นํžˆ ์šฐ์ˆ˜ํ•จ์„ ๋ณด์—ฌ์ค€๋‹ค. ๋น„๋””์˜ค ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„ ๋˜๋Š” ์›€์ง์ž„ ๋ณด์ƒ ํ”„๋ ˆ์ž„์œจ ์ƒํ–ฅ ๋ณ€ํ™˜ (MC-FRUC)์—์„œ, ๋‹จ๋ฐฉํ–ฅ ์›€์ง์ž„ ๊ถค์ ์— ๋”ฐ๋ฅธ ์›€์ง์ž„ ๋ณด์ƒ์€ overlap๊ณผ hole ๋ฌธ์ œ๋ฅผ ์ผ์œผํ‚จ๋‹ค. ๋ณธ ์—ฐ๊ตฌ์—์„œ ์ด๋Ÿฌํ•œ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด ์–‘๋ฐฉํ–ฅ ์›€์ง์ž„ ๋ณด์ƒ ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„์„ ์œ„ํ•œ ์ƒˆ๋กœ์šด ์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ์ œ์‹œํ•œ๋‹ค. ๋จผ์ €, ์ œ์•ˆ๋œ ๋ฐฉ๋ฒ•์€ ๋‹จ๋ฐฉํ–ฅ ์›€์ง์ž„ ์ถ”์ •์œผ๋กœ๋ถ€ํ„ฐ ์–ป์–ด์ง„ ๋‘ ๊ฐœ์˜ ๋‹จ๋ฐฉํ–ฅ ์›€์ง์ž„ ์˜์—ญ(์ „๋ฐฉ ๋ฐ ํ›„๋ฐฉ)์œผ๋กœ๋ถ€ํ„ฐ ์–‘๋ฐฉํ–ฅ ์›€์ง์ž„ ๋ฒกํ„ฐ๋ฅผ ์ƒ์„ฑํ•œ๋‹ค. ์ด๋Š” ์ „๋ฐฉ ๋ฐ ํ›„๋ฐฉ ์›€์ง์ž„ ๋ฒกํ„ฐ๋ฅผ ๋ณด๊ฐ„ ํ”„๋ ˆ์ž„์— ํˆฌ์˜ํ•จ์œผ๋กœ์จ ์ˆ˜ํ–‰๋œ๋‹ค. ๋ณด๊ฐ„๋œ ๋ธ”๋ก์— ์—ฌ๋Ÿฌ ๊ฐœ์˜ ํˆฌ์˜๋œ ๋ธ”๋ก์ด ์žˆ๋Š” ๊ฒฝ์šฐ, ํˆฌ์˜๋œ ๋ธ”๋ก๊ณผ ๋ณด๊ฐ„๋œ ๋ธ”๋ก ์‚ฌ์ด์˜ ๊ฑฐ๋ฆฌ๋ฅผ ํ™•์žฅํ•˜๋Š” ๊ธฐ์ค€์ด ๊ฐ€์ค‘ ๊ณ„์ˆ˜๋ฅผ ๊ณ„์‚ฐํ•˜๊ธฐ ์œ„ํ•ด ์ œ์•ˆ๋œ๋‹ค. Hole์€ hole์ด ์•„๋‹Œ ์ด์›ƒ ๋ธ”๋ก์˜ vector median filter๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ์ฒ˜๋ฆฌ๋œ๋‹ค. ์ œ์•ˆ ๋ฐฉ๋ฒ•์€ ๊ธฐ์กด์˜ MC-FRUC๋ณด๋‹ค ์„ฑ๋Šฅ์ด ์šฐ์ˆ˜ํ•˜๋ฉฐ, ๋ธ”๋ก ์—ดํ™”๋ฅผ ์ƒ๋‹นํžˆ ์ œ๊ฑฐํ•œ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์—์„œ๋Š” CNN์„ ์ด์šฉํ•œ ๋น„๋””์˜ค ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„์— ๋Œ€ํ•ด์„œ๋„ ๋‹ค๋ฃฌ๋‹ค. Optical flow ๋ฐ ๋น„๋””์˜ค ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„์€ ํ•œ ๊ฐ€์ง€ ๋ฌธ์ œ๊ฐ€ ๋‹ค๋ฅธ ๋ฌธ์ œ์— ์˜ํ–ฅ์„ ๋ฏธ์น˜๋Š” chicken-egg ๋ฌธ์ œ๋กœ ๊ฐ„์ฃผ๋œ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์—์„œ๋Š” ์ค‘๊ฐ„ optical flow ๋ฅผ ๊ณ„์‚ฐํ•˜๋Š” ๋„คํŠธ์›Œํฌ์™€ ๋ณด๊ฐ„ ํ”„๋ ˆ์ž„์„ ํ•ฉ์„ฑ ํ•˜๋Š” ๋‘ ๊ฐ€์ง€ ๋„คํŠธ์›Œํฌ๋กœ ์ด๋ฃจ์–ด์ง„ ํ•˜๋‚˜์˜ ๋„คํŠธ์›Œํฌ ์Šคํƒ์„ ๊ตฌ์กฐ๋ฅผ ์ œ์•ˆํ•œ๋‹ค. The final ๋ณด๊ฐ„ ํ”„๋ ˆ์ž„์„ ์ƒ์„ฑํ•˜๋Š” ๋„คํŠธ์›Œํฌ์˜ ๊ฒฝ์šฐ ์ฒซ ๋ฒˆ์งธ ๋„คํŠธ์›Œํฌ์˜ ์ถœ๋ ฅ์ธ ๋ณด๊ฐ„ ํ”„๋ ˆ์ž„ ์™€ ์ค‘๊ฐ„ optical flow based warped frames์„ ์ž…๋ ฅ์œผ๋กœ ๋ฐ›์•„์„œ ํ”„๋ ˆ์ž„์„ ์ƒ์„ฑํ•œ๋‹ค. ์ œ์•ˆ๋œ ๊ตฌ์กฐ์˜ ๊ฐ€์žฅ ํฐ ํŠน์ง•์€ optical flow ๊ณ„์‚ฐ์„ ์œ„ํ•œ ํ•ฉ์„ฑ์— ์˜ํ•œ ๋ถ„์„๋ฒ•๊ณผ CNN ๊ธฐ๋ฐ˜์˜ ๋ถ„์„์— ์˜ํ•œ ํ•ฉ์„ฑ๋ฒ•์„ ๋ชจ๋‘ ์ด์šฉํ•˜์—ฌ ํ•˜๋‚˜์˜ ์ข…ํ•ฉ์ ์ธ framework๋กœ ๊ฒฐํ•ฉํ•˜์˜€๋‹ค๋Š” ๊ฒƒ์ด๋‹ค. ์ œ์•ˆ๋œ ๋„คํŠธ์›Œํฌ๋Š” ๊ธฐ์กด์˜ ๋‘ ๊ฐ€์ง€ ์—ฐ๊ตฌ์ธ optical flow ๊ธฐ๋ฐ˜ ํ”„๋ ˆ์ž„ ํ•ฉ์„ฑ๊ณผ CNN ๊ธฐ๋ฐ˜ ํ•ฉ์„ฑ ํ”„๋ ˆ์ž„ ํ•ฉ์„ฑ๋ฒ•์„ ์ฒ˜์Œ ๊ฒฐํ•ฉ์‹œํ‚จ ๋ฐฉ์‹์ด๋‹ค. ์‹คํ—˜์€ ๋‹ค์–‘ํ•˜๊ณ  ๋ณต์žกํ•œ ๋ฐ์ดํ„ฐ ์…‹์œผ๋กœ ์ด๋ฃจ์–ด์กŒ์œผ๋ฉฐ, ๋ณด๊ฐ„ ํ”„๋ ˆ์ž„ quality ์™€ optical flow ๊ณ„์‚ฐ ์ •ํ™•๋„ ์ธก๋ฉด์—์„œ ๊ธฐ์กด์˜ state-of-art ๋ฐฉ์‹์— ๋น„ํ•ด ์›”๋“ฑํžˆ ๋†’์€ ์„ฑ๋Šฅ์„ ๋ณด์˜€๋‹ค. ๋ณธ ๋…ผ๋ฌธ์˜ ํ›„ ์ฒ˜๋ฆฌ๋ฅผ ์œ„ํ•œ ์‹ฌ์ธต ๋น„๋””์˜ค ํ”„๋ ˆ์ž„ ๋ณด๊ฐ„ ๋„คํŠธ์›Œํฌ๋Š” ์ฝ”๋”ฉ ํšจ์œจ ํ–ฅ์ƒ์„ ์œ„ํ•ด ์ตœ์‹  ๋น„๋””์˜ค ์••์ถ• ํ‘œ์ค€์ธ HEVC/H.265์— ์ ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์‹คํ—˜ ๊ฒฐ๊ณผ๋Š” ์ œ์•ˆ ๋„คํŠธ์›Œํฌ์˜ ํšจ์œจ์„ฑ์„ ์ž…์ฆํ•œ๋‹ค.Abstract i Table of Contents iv List of Tables vii List of Figures viii Chapter 1. Introduction 1 1.1. Hierarchical Motion Estimation of Small Objects 2 1.2. Motion Estimation of a Repetition Pattern Region 4 1.3. Motion-Compensated Frame Interpolation 5 1.4. Video Frame Interpolation with Deep CNN 6 1.5. Outline of the Thesis 7 Chapter 2. Previous Works 9 2.1. Previous Works on Hierarchical Block-Based Motion Estimation 9 2.1.1.โ€‚Maximum a Posterior (MAP) Framework 10 2.1.2.Hierarchical Motion Estimation 12 2.2. Previous Works on Motion Estimation for a Repetition Pattern Region 13 2.3. Previous Works on Motion Compensation 14 2.4. Previous Works on Video Frame Interpolation with Deep CNN 16 Chapter 3. Hierarchical Motion Estimation for Small Objects 19 3.1. Problem Statement 19 3.2. The Alternative Motion Vector of High Cost Pixels 20 3.3. Modified Hierarchical Motion Estimation 23 3.4. Framework of the Proposed Algorithm 24 3.5. Experimental Results 25 3.5.1. Performance Analysis 26 3.5.2. Performance Evaluation 29 Chapter 4. Semi-Global Accurate Motion Estimation for a Repetition Pattern Region 32 4.1. Problem Statement 32 4.2. Objective Function and Constrains 33 4.3. Elector based Voting System 34 4.4. Voter based Voting System 36 4.5. Experimental Results 40 Chapter 5. Multiple Motion Vectors based Motion Compensation 44 5.1. Problem Statement 44 5.2. Adaptive Weighted Multiple Motion Vectors based Motion Compensation 45 5.2.1. One-to-Multiple Motion Vector Projection 45 5.2.2. A Comprehensive Metric as the Extension of Distance 48 5.3. Handling Hole Blocks 49 5.4. Framework of the Proposed Motion Compensated Frame Interpolation 50 5.5. Experimental Results 51 Chapter 6. Video Frame Interpolation with a Stack of Deep CNN 56 6.1. Problem Statement 56 6.2. The Proposed Network for Video Frame Interpolation 57 6.2.1. A Stack of Synthesis Networks 57 6.2.2. Intermediate Optical Flow Derivation Module 60 6.2.3. Warping Operations 62 6.2.4. Training and Loss Function 63 6.2.5. Network Architecture 64 6.2.6. Experimental Results 64 6.2.6.1. Frame Interpolation Evaluation 64 6.2.6.2. Ablation Experiments 77 6.3. Extension for Quality Enhancement for Compressed Videos Task 83 6.4. Extension for Improving the Coding Efficiency of HEVC based Low Bitrate Encoder 88 Chapter 7. Conclusion 94 References 97Docto

    A comprehensive video codec comparison

    Get PDF
    In this paper, we compare the video codecs AV1 (version 1.0.0-2242 from August 2019), HEVC (HM and x265), AVC (x264), the exploration software JEM which is based on HEVC, and the VVC (successor of HEVC) test model VTM (version 4.0 from February 2019) under two fair and balanced configurations: All Intra for the assessment of intra coding and Maximum Coding Efficiency with all codecs being tuned for their best coding efficiency settings. VTM achieves the highest coding efficiency in both configurations, followed by JEM and AV1. The worst coding efficiency is achieved by x264 and x265, even in the placebo preset for highest coding efficiency. AV1 gained a lot in terms of coding efficiency compared to previous versions and now outperforms HM by 24% BD-Rate gains. VTM gains 5% over AV1 in terms of BD-Rates. By reporting separate numbers for JVET and AOM test sequences, it is ensured that no bias in the test sequences exists. When comparing only intra coding tools, it is observed that the complexity increases exponentially for linearly increasing coding efficiency
    • โ€ฆ
    corecore