비디오 프레임 보간을 위한 다중 벡터 기반의 MEMC 및 심층 CNN

학위논문 (박사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 이혁재.Block-based hierarchical motion estimations are widely used and are successful in generating high-quality interpolation. However, it still fails in the motion estimation of small objects when a background region moves in a different direction. This is because the motion of small objects is neglected by the down-sampling and over-smoothing operations at the top level of image pyramids in the maximum a posterior (MAP) method. Consequently, the motion vector of small objects cannot be detected at the bottom level, and therefore, the small objects often appear deformed in an interpolated frame. This thesis proposes a novel algorithm that preserves the motion vector of the small objects by adding a secondary motion vector candidate that represents the movement of the small objects. This additional candidate is always propagated from the top to the bottom layers of the image pyramid. Experimental results demonstrate that the intermediate frame interpolated by the proposed algorithm significantly improves the visual quality when compared with conventional MAP-based frame interpolation. In motion compensated frame interpolation, a repetition pattern in an image makes it difficult to derive an accurate motion vector because multiple similar local minima exist in the search space of the matching cost for motion estimation. In order to improve the accuracy of motion estimation in a repetition region, this thesis attempts a semi-global approach that exploits both local and global characteristics of a repetition region. A histogram of the motion vector candidates is built by using a voter based voting system that is more reliable than an elector based voting system. Experimental results demonstrate that the proposed method significantly outperforms the previous local approach in term of both objective peak signal-to-noise ratio (PSNR) and subjective visual quality. In video frame interpolation or motion-compensated frame rate up-conversion (MC-FRUC), motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this research presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates bidirectional motion vectors from two unidirectional motion vector fields (forward and backward) obtained from the unidirectional motion estimations. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in the case when the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly. Video frame interpolation with a deep convolutional neural network (CNN) is also investigated in this thesis. Optical flow and video frame interpolation are considered as a chicken-egg problem such that one problem affects the other and vice versa. This thesis presents a stack of networks that are trained to estimate intermediate optical flows from the very first intermediate synthesized frame and later the very end interpolated frame is generated by the second synthesis network that is fed by stacking the very first one and two learned intermediate optical flows based warped frames. The primary benefit is that it glues two problems into one comprehensive framework that learns altogether by using both an analysis-by-synthesis technique for optical flow estimation and vice versa, CNN kernels based synthesis-by-analysis. The proposed network is the first attempt to bridge two branches of previous approaches, optical flow based synthesis and CNN kernels based synthesis into a comprehensive network. Experiments are carried out with various challenging datasets, all showing that the proposed network outperforms the state-of-the-art methods with significant margins for video frame interpolation and the estimated optical flows are accurate for challenging movements. The proposed deep video frame interpolation network to post-processing is applied to the improvement of the coding efficiency of the state-of-art video compress standard, HEVC/H.265 and experimental results prove the efficiency of the proposed network.블록 기반 계층적 움직임 추정은 고화질의 보간 이미지를 생성할 수 있어 폭넓게 사용되고 있다. 하지만, 배경 영역이 움직일 때, 작은 물체에 대한 움직임 추정 성능은 여전히 좋지 않다. 이는 maximum a posterior (MAP) 방식으로 이미지 피라미드의 최상위 레벨에서 down-sampling과 over-smoothing으로 인해 작은 물체의 움직임이 무시되기 때문이다. 결과적으로 이미지 피라미드의 최하위 레벨에서 작은 물체의 움직임 벡터는 검출될 수 없어 보간 이미지에서 작은 물체는 종종 변형된 것처럼 보인다. 본 논문에서는 작은 물체의 움직임을 나타내는 2차 움직임 벡터 후보를 추가하여 작은 물체의 움직임 벡터를 보존하는 새로운 알고리즘을 제안한다. 추가된 움직임 벡터 후보는 항상 이미지 피라미드의 최상위에서 최하위 레벨로 전파된다. 실험 결과는 제안된 알고리즘의 보간 생성 프레임이 기존 MAP 기반 보간 방식으로 생성된 프레임보다 이미지 화질이 상당히 향상됨을 보여준다. 움직임 보상 프레임 보간에서, 이미지 내의 반복 패턴은 움직임 추정을 위한 정합 오차 탐색 시 다수의 유사 local minima가 존재하기 때문에 정확한 움직임 벡터 유도를 어렵게 한다. 본 논문은 반복 패턴에서의 움직임 추정의 정확도를 향상시키기 위해 반복 영역의 local한 특성과 global한 특성을 동시에 활용하는 semi-global한 접근을 시도한다. 움직임 벡터 후보의 히스토그램은 선거 기반 투표 시스템보다 신뢰할 수 있는 유권자 기반 투표 시스템 기반으로 형성된다. 실험 결과는 제안된 방법이 이전의 local한 접근법보다 peak signal-to-noise ratio (PSNR)와 주관적 화질 판단 관점에서 상당히 우수함을 보여준다. 비디오 프레임 보간 또는 움직임 보상 프레임율 상향 변환 (MC-FRUC)에서, 단방향 움직임 궤적에 따른 움직임 보상은 overlap과 hole 문제를 일으킨다. 본 연구에서 이러한 문제를 해결하기 위해 양방향 움직임 보상 프레임 보간을 위한 새로운 알고리즘을 제시한다. 먼저, 제안된 방법은 단방향 움직임 추정으로부터 얻어진 두 개의 단방향 움직임 영역(전방 및 후방)으로부터 양방향 움직임 벡터를 생성한다. 이는 전방 및 후방 움직임 벡터를 보간 프레임에 투영함으로써 수행된다. 보간된 블록에 여러 개의 투영된 블록이 있는 경우, 투영된 블록과 보간된 블록 사이의 거리를 확장하는 기준이 가중 계수를 계산하기 위해 제안된다. Hole은 hole이 아닌 이웃 블록의 vector median filter를 기반으로 처리된다. 제안 방법은 기존의 MC-FRUC보다 성능이 우수하며, 블록 열화를 상당히 제거한다. 본 논문에서는 CNN을 이용한 비디오 프레임 보간에 대해서도 다룬다. Optical flow 및 비디오 프레임 보간은 한 가지 문제가 다른 문제에 영향을 미치는 chicken-egg 문제로 간주된다. 본 논문에서는 중간 optical flow 를 계산하는 네트워크와 보간 프레임을 합성 하는 두 가지 네트워크로 이루어진 하나의 네트워크 스택을 구조를 제안한다. The final 보간 프레임을 생성하는 네트워크의 경우 첫 번째 네트워크의 출력인 보간 프레임 와 중간 optical flow based warped frames을 입력으로 받아서 프레임을 생성한다. 제안된 구조의 가장 큰 특징은 optical flow 계산을 위한 합성에 의한 분석법과 CNN 기반의 분석에 의한 합성법을 모두 이용하여 하나의 종합적인 framework로 결합하였다는 것이다. 제안된 네트워크는 기존의 두 가지 연구인 optical flow 기반 프레임 합성과 CNN 기반 합성 프레임 합성법을 처음 결합시킨 방식이다. 실험은 다양하고 복잡한 데이터 셋으로 이루어졌으며, 보간 프레임 quality 와 optical flow 계산 정확도 측면에서 기존의 state-of-art 방식에 비해 월등히 높은 성능을 보였다. 본 논문의 후 처리를 위한 심층 비디오 프레임 보간 네트워크는 코딩 효율 향상을 위해 최신 비디오 압축 표준인 HEVC/H.265에 적용할 수 있으며, 실험 결과는 제안 네트워크의 효율성을 입증한다.Abstract i Table of Contents iv List of Tables vii List of Figures viii Chapter 1. Introduction 1 1.1. Hierarchical Motion Estimation of Small Objects 2 1.2. Motion Estimation of a Repetition Pattern Region 4 1.3. Motion-Compensated Frame Interpolation 5 1.4. Video Frame Interpolation with Deep CNN 6 1.5. Outline of the Thesis 7 Chapter 2. Previous Works 9 2.1. Previous Works on Hierarchical Block-Based Motion Estimation 9 2.1.1. Maximum a Posterior (MAP) Framework 10 2.1.2.Hierarchical Motion Estimation 12 2.2. Previous Works on Motion Estimation for a Repetition Pattern Region 13 2.3. Previous Works on Motion Compensation 14 2.4. Previous Works on Video Frame Interpolation with Deep CNN 16 Chapter 3. Hierarchical Motion Estimation for Small Objects 19 3.1. Problem Statement 19 3.2. The Alternative Motion Vector of High Cost Pixels 20 3.3. Modified Hierarchical Motion Estimation 23 3.4. Framework of the Proposed Algorithm 24 3.5. Experimental Results 25 3.5.1. Performance Analysis 26 3.5.2. Performance Evaluation 29 Chapter 4. Semi-Global Accurate Motion Estimation for a Repetition Pattern Region 32 4.1. Problem Statement 32 4.2. Objective Function and Constrains 33 4.3. Elector based Voting System 34 4.4. Voter based Voting System 36 4.5. Experimental Results 40 Chapter 5. Multiple Motion Vectors based Motion Compensation 44 5.1. Problem Statement 44 5.2. Adaptive Weighted Multiple Motion Vectors based Motion Compensation 45 5.2.1. One-to-Multiple Motion Vector Projection 45 5.2.2. A Comprehensive Metric as the Extension of Distance 48 5.3. Handling Hole Blocks 49 5.4. Framework of the Proposed Motion Compensated Frame Interpolation 50 5.5. Experimental Results 51 Chapter 6. Video Frame Interpolation with a Stack of Deep CNN 56 6.1. Problem Statement 56 6.2. The Proposed Network for Video Frame Interpolation 57 6.2.1. A Stack of Synthesis Networks 57 6.2.2. Intermediate Optical Flow Derivation Module 60 6.2.3. Warping Operations 62 6.2.4. Training and Loss Function 63 6.2.5. Network Architecture 64 6.2.6. Experimental Results 64 6.2.6.1. Frame Interpolation Evaluation 64 6.2.6.2. Ablation Experiments 77 6.3. Extension for Quality Enhancement for Compressed Videos Task 83 6.4. Extension for Improving the Coding Efficiency of HEVC based Low Bitrate Encoder 88 Chapter 7. Conclusion 94 References 97Docto

Grid and rotor sides of doubly-fed induction generator-based wind energy conversion system using sliding mode control approach

ix, 78 leaves : illustrations (chiefly coloured) ; 29 cmIncludes abstract and appendix.Includes bibliographical references (leaves 65-78).This thesis deals with the analysis, modeling, and control of a Doubly-Fed Induction machine used as a wind turbine generator. A sliding mode control scheme is applied to control the power and Dc-link voltage. Two back-to-back converters are used at rotor and stator sides. At the rotor side, power control is achieved by controlling the rotor current, while at the grid side, an active power transfer is used to model the dc-link voltage. Overall robustness and tracking performance are enhanced to deal with uncertainties due to the structure of the sliding mode control law, compensation combinations, sliding and integral terms. An experimental 2-kw DFIG wind turbine system was used to validate the proposed control system. Based on the results obtained, the proposed system showed good capabilities in tracking and control under various operating conditions as well as robustness to uncertainties

Radar Technology

In this book “Radar Technology”, the chapters are divided into four main topic areas: Topic area 1: “Radar Systems” consists of chapters which treat whole radar systems, environment and target functional chain. Topic area 2: “Radar Applications” shows various applications of radar systems, including meteorological radars, ground penetrating radars and glaciology. Topic area 3: “Radar Functional Chain and Signal Processing” describes several aspects of the radar signal processing. From parameter extraction, target detection over tracking and classification technologies. Topic area 4: “Radar Subsystems and Components” consists of design technology of radar subsystem components like antenna design or waveform design

Power Electronics Applications in Renewable Energy Systems

The renewable generation system is currently experiencing rapid growth in various power grids. The stability and dynamic response issues of power grids are receiving attention due to the increase in power electronics-based renewable energy. The main focus of this Special Issue is to provide solutions for power system planning and operation. Power electronics-based devices can offer new ancillary services to several industrial sectors. In order to fully include the capability of power conversion systems in the network integration of renewable generators, several studies should be carried out, including detailed studies of switching circuits, and comprehensive operating strategies for numerous devices, consisting of large-scale renewable generation clusters

DC & Hybrid Micro-Grids

This book is a printed version of the papers published in the Special Issue “DC & Hybrid Microgrids” of Applied Sciences. This Special Issue, co-organized by the University of Pisa, Italy and Østfold University College in Norway, has collected nine papers and the editorial, from 28 submitted, with authors from Asia, North America and Europe. The published articles provide an overview of the most recent research advances in direct current (DC) and hybrid microgrids, exploiting the opportunities offered by the use of renewable energy sources, battery energy storage systems, power converters, innovative control and energy management strategies

Fusion of sensor information to measure the total energy of an aircraft and provide information about flight performance and local microclimate

The application of using Unmanned Aerial Vehicles (UAVs) to locate thermal updraft currents is a relatively new topic. It was first proposed in 1998 by John Wharington, and, subsequently, several researchers have developed algorithms to search and exploit thermals. However, few people have physically implemented a system and performed field testing. The aim of this project was to develop a low cost system to be carried on a glider to detect thermals effectively. A system was developed from the ground up and consisted of custom hardware and software that was developed specifically for aircraft. Data fusion was performed to estimate the attitude of the aircraft; this was done using a direction cosine (DCM) based method. Altitude and airspeed data were fused by estimating potential and kinetic energy respectively; thus determining the aircraft's total energy. This data was then interpreted to locate thermal activity. The system comprised an Inertial Measurement Unit (IMU), airspeed sensor, barometric altitude sensor, Global Positioning System (GPS), temperature sensor, SD card and a realtime telemetry link. These features allowed the system to determine aircraft position, height, airspeed and air temperature in realtime. A custom-designed radio controlled (RC) glider was constructed from composite materials in addition to a second 3.6 m production glider that was used during flight testing. Sensor calibration was done using a wind tunnel with custom designed apparatus that allowed a complete wing with its pitot tube to be tested in one operation. Flight testing was conducted in the field at several different locations over the course of six months. A total of 25 recorded flights were made during this period. Both thermal soaring and ridge soaring were performed to test the system under varying weather conditions. A telemetry link was developed to transfer data in realtime from the aircraft to a custom ground station. The recorded results were post-processed using Matlab and showed that the system was able to detect thermal updrafts. The sensors used in the system were shown to provide acceptable performance once some calibration had been performed. Sensor noise proved to be problematic, and time was spent alleviating its effects. Results showed that the system was able to measure airspeed to within ± 1 km/h. The standard deviation of the altitude estimate was determined to be 0.94 m. This was deemed to be satisfactory. The system was highly reliable and no faults occurred during operation. In conclusion, the project showed that inexpensive sensors and low power microcontrollers could be used very effectively for the application of detecting thermals

An aircraft and provide information about flight performance and local microclimate

Includes abstract.Includes bibliographical referencesThe application of using Unmanned Aerial Vehicles (UAVs) to locate thermal updraft currentsis a relatively new topic. It was first proposed in 1998 by John Wharington, and, subsequently, several researchers have developed algorithms to search and exploit thermals. However, few people have physically implemented a system and performed field testing. The aim of this project was to develop a low cost system to be carried on a glider to detect thermals effectively. A system was developed from the ground up and consisted of custom hardware and software that was developed specifically for aircraft. Data fusion was performed to estimate the attitude of the aircraft; this was done using a direction cosine (DCM) based method. Altitude and airspeed data were fused by estimating potential and kinetic energy respectively; thus determining the aircraft’s total energy. This data was then interpreted to locate thermal activity. The system comprised an Inertial Measurement Unit (IMU), airspeed sensor, barometric altitude sensor, Global Positioning System (GPS), temperature sensor, SD card and a realtime telemetry link. These features allowed the system to determine aircraft position, height, airspeed and air temperature in realtime. A custom-designed radio controlled (RC) glider was constructed from composite materials in addition to a second 3.6 m production glider that was used during flight testing. Sensor calibration was done using a wind tunnel with custom designed apparatus that allowed a complete wing with its pitot tube to be tested in one operation. Flight testing was conducted in the field at several different locations over the course of six months. A total of 25 recorded flights were made during this period. Both thermal soaring and ridge soaring were performed to test the system under varying weather conditions. A telemetry link was developed to transfer data in realtime from the aircraft to a custom ground station. The recorded results were post-processed using Matlab and showed that the system was able to detect thermal updrafts. The sensors used in the system were shown to provide acceptable performance once some calibration had been performed. Sensor noise proved to be problematic, and time was spent alleviating its effects

NASA Tech Briefs, December 1989

Topics include: Electronic Components and Circuits. Electronic Systems, Physical Sciences, Materials, Computer Programs, Mechanics, Machinery, Fabrication Technology, Mathematics and Information Sciences, and Life Sciences