메모리 대역폭이 감소된 다중 프레임 레이트 옵티칼 플로우

학위논문 (박사)-- 서울대학교 대학원 : 전기·정보공학부, 2015. 2. 김수환.최근 high frame rate camera의 비약적인 발전으로 이미 4K 1000FPS camera가 출시되었고 휴대폰에서도 1080P 240FPS를 지원하고 있다. Camera의 Frame rate 증가는 optical flow의 구현에 시사하는 바가 큰데, 그 이유는 frame rate이 올라갈수록 frame 간의 움직임 크기가 줄어들기 때문이다. 그 동안 큰 움직임에 대한 부정확한 optical flow 문제를 해결하기 위해서 다양한 알고리즘이 사용되어 왔지만, 이로 인한 computation의 증가 또는 알고리즘 dependency로 인해 늘어난 연산 시간은 real-time operation에 제약으로 작용한다. 하지만 camera의 frame rate이 올라가면 모든 움직임들은 이에 반비례해서 작아지므로, 결국 high frame rate camera는 간단한 알고리즘으로 정확한 optical flow를 얻을 수 있는 길을 열고 있다. 본 논문은 accurate real-time optical flow의 구현을 위해서 multi-frame rate and multi-scale optical flow 알고리즘을 제안한다. High frame rate camera를 이용한 multi-frame rate and multi-scale optical flow 알고리즘은 real-time optical flow의 hardware 구현에 적합하도록 iterative calculation없는 알고리즘이다. Multi-frame rate 알고리즘은 다양한 frame rate의 optical flow를 연산하고 서로간의 연관관계를 이용하여 slow motion 뿐만 아니라 high motion에 관해서도 optical flow 결과를 얻게 함으로써 측정 가능한 움직임을 확장시킨 알고리즘이다. 이 알고리즘은 frame rate 증가에 따른 시스템 연산량 증가를 기존 연구의 O(n)에서 O(log n) 수준으로 감소시킴으로써 system performance에 의한 제약을 크게 줄인다. Multi scale 알고리즘은 high frame rate system을 위한 full density 지원 알고리즘이다. 또한 본 논문에서는 frame rate의 증가에 따른 external memory access bandwidth 증가 문제를 풀기 위해서 spatial & temporal bandwidth reduction 알고리즘을 제안한다. 이 방법은 기존 LK optical flow알고리즘의 연산 순서를 바꾸고, iterative sub-sampling scheme, temporal Gaussian tail cut 그리고 frame reuse 등 다양한 방식의 알고리즘들을 제안함으로써 high frame rate system의 external memory access bandwidth를 감소시킨다. 마지막으로 Multi-Frame rate and multi-scale optical flow 알고리즘의 Multi-scale 구조의 hardware 의 구현 시 multiplier의 개수를 mxm크기의 윈도우처리를 위해 m개의 multiplier를 이용해서 convolution방식으로 구현하던 기존의 방법을 윈도우의 크기에 상관없이 2개의 multiplier로 mxm multiplication을 구현하는 방식을 제안한다. 이 방식을 기반으로 multi frame rate과 multi-scale의 hardware architecture를 제안하고 single level LK optical flow의 fpga구현을 통해서 제안한 architecture의 hardware 동작을 검증한다. 이상의 과정들을 통해서 accurate real-time optical flow system을 위한 multi-frame rate and multi-scale optical flow의 알고리즘 제안부터 architecture 검증까지의 연구를 진행한다.차 례 초 록 i 차 례 iii 그림 목차 vii 표 목 차 x 제1장 서 론 1 1.1 연구 배경 1 1.2 연구 내용 4 1.3 논문 구성 6 제2장 이전연구 7 2.1 LK Optical Flow 7 2.2 Large Displacement Problem 10 2.2.1 Pyramidal LK Optical Flow 10 2.2.2 High Frame Rate Optical Flow 11 2.2.3 Oversampled Optical Flow System 11 2.2.4 High Frame Rate Optical Flow System 14 2.3 Problems in High Frame Rate System 16 2.3.1 Test Sequence for High Frame Rate System 16 2.3.2 Saturated Accuracy for High frame rate 17 2.3.3 Accurate displacement for LK optical flow 21 2.3.1 Accurate frame rate of High frame rate system 23 제3장 Multi-Frame Rate Optical Flow 28 3.1 Ideal and Real Optical Flow System 29 3.2 Multi-Frame Rate Optical Flow 31 3.3 Accurate Frame Rate Selection 33 3.3.1 Magnitude Selection Algorithm 33 3.3.2 Magnitude Algorithm Validation 36 3.3.3 SSD(Sum of Squared Difference) 45 3.3.4 Magnitude with NR Selection Algorithm 48 3.3.5 Temporal Aliasing 51 3.4 Multi Frame Rate Optical Flow Test Result 52 3.5 Comparisons with previous works 57 제4장 Multi-Scale Optical Flow 59 4.1 Pyramidal Level Selection 61 4.2 Level Selection Algorithm 62 4.3 Pyramidal Image Generation 63 4.4 Proposed Algorithm Verification 66 4.4.1 Accuracy comparison 66 4.4.2 연산량 및 알고리즘 특성 comparisons 67 4.4.3 Graphical Result with various test sequences 69 제5장 Memory Bandwidth Reduction 76 5.1 Single Level LK Optical Flow System 76 5.1.1 Bandwidth Problem 76 5.1.2 Matrix multiplication 77 5.1.3 FPGA Implementation 77 5.2 Spatial Bandwidth Reduction 79 5.2.1 LK Optical Flow System Architecture 79 5.2.2 External Memory Bandwidth Requirement 80 5.2.1 제안하는 알고리즘 83 5.2.2 Simulation Result 86 5.3 Temporal Bandwidth Reduction 93 5.3.1 Tail Cut 93 5.3.2 Frame Reuse 97 5.3.3 Experimental Result 101 5.4 Matrix Generation 107 5.4.1 Matrix Multiplication 107 5.4.2 Proposed Matrix Multiplication 108 5.5 FPGA Implementation 111 제6장 결론 117 참 고 문 헌 118 Abstract 122Docto

Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application

Optical flow computation with frame-based cameras provides high accuracy but the speed is limited either by the model size of the algorithm or by the frame rate of the camera. This makes it inadequate for high-speed applications. Event cameras provide continuous asynchronous event streams overcoming the frame-rate limitation. However, the algorithms for processing the data either borrow frame like setup limiting the speed or suffer from lower accuracy. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow while maintaining a low error rate. Our bio-mimetic network is validated with the MVSEC dataset showing 19% error degradation at 4x speed up. We then demonstrate the system with a high-speed drone flight scenario where a high-speed event camera computes the flow even before the optical camera sees the drone making it suited for applications like tracking and segmentation. This work shows the fundamental trade-offs in frame-based processing may be overcome by fusing data from other modalities

Increasing the field-of-view of dynamic cardiac OCT via post-acquisition mosaicing without affecting frame-rate or spatial resolution

Optical coherence tomography (OCT) allows imaging dynamic structures and fluid flow within scattering tissue, such as the beating heart and blood flow in murine embryos. For any given system, the frame rate, spatial resolution, field-of-view (FOV), and signal-to-noise ratio (SNR) are interconnected: favoring one aspect limits at least one of the others due to optical, instrumentation, and software constraints. Here we describe a spatio-temporal mosaicing technique to reconstruct high-speed, high spatial-resolution, and large-field-of-view OCT sequences. The technique is applicable to imaging any cyclically moving structure and operates on multiple, spatially overlapping tiled image sequences (each sequence acquired sequentially at a given spatial location) and effectively decouples the (rigid) spatial alignment and (non-rigid) temporal registration problems. Using this approach we reconstructed full-frame OCT sequences of the beating embryonic rat heart (11.5 days post coitus) and compared it to direct imaging on the same system, demonstrating a six-fold improvement of the frame rate without compromising spatial resolution, FOV, or SNR

Cost-effective approaches for high-resolution bioimaging by time-stretched confocal microscopy at 1um

Session: Optics Imaging Algorithms and Analysis IIOptical imaging based on time-stretch process has recently been proven as a powerful tool for delivering ultra-high frame rate (< 1MHz) which is not achievable by the conventional image sensors. Together with the capability of optical image amplification for overcoming the trade-off between detection sensitivity and speed, this new imaging modality is particularly valuable in high-throughput biomedical diagnostic practice, e.g. imaging flow cytometry. The ultra-high frame rate in time-stretch imaging is attained by two key enabling elements: dispersive fiber providing the time-stretch process via group-velocity-dispersion (GVD), and electronic digitizer. It is well-known that many biophotonic applications favor the spectral window of 1μm. However, reasonably high GVD (< 0.1 ns/nm) in this range can only be achieved by using specialty single-mode fiber (SMF) at 1μm. Moreover, the ultrafast detection has to rely on the state-of- the-art digitizer with significantly wide-bandwidth and high sampling rate (e.g. <10 GHz, <40 GS/s). These stringent requirements imply the prohibitively high-cost of the system and hinder its practical use in biomedical diagnostics. We here demonstrate two cost-effective approaches for realizing time-stretch confocal microscopy at 1μm: (i) using the standard telecommunication SMF (e.g. SMF28) to act as a few-mode fiber (FMF) at 1μm for the time-stretch process, and (ii) implementing the pixel super-resolution (SR) algorithm to restore the high-resolution (HR) image when using a lower-bandwidth digitizer. By using a FMF (with a GVD of 0.15ns/nm) and a modified pixel-SR algorithm, we can achieve time-stretch confocal microscopy at 1μm with cellular resolution ( 3μm) at a frame rate 1 MHz.© (2012) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.published_or_final_versio

Optical Flow Background Estimation for Real-time Pan/tilt Camera Object Tracking

As Computer Vision (CV) techniques develop, pan/tilt camera systems are able to enhance data capture capabilities over static camera systems. In order for these systems to be effective for metrology purposes, they will need to respond to the test article in real-time with a minimum of additional uncertainty. A methodology is presented here for obtaining high-resolution, high frame-rate images, of objects traveling at speeds ⩾1.2 m/s at 1 m from the camera by tracking the moving texture of an object. Strong corners are determined and used as flow points using implementations on a graphic processing unit (GPU), resulting in significant speed-up over central processing units (CPU). Based on directed pan/tilt motion, a pixel-to-pixel relationship is used to estimate whether optical flow points fit background motion, dynamic motion or noise. To smooth variation, a two-dimensional position and velocity vector is used with a Kalman filter to predict the next required position of the camera so the object stays centered in the image. High resolution images can be stored by a parallel process resulting in a high frame rate procession of images for post-processing. The results provide real-time tracking on a portable system using a pan/tilt unit for generic moving targets where no training is required and camera motion is observed from high accuracy encoders opposed to image correlation

HSTR-Net: Reference Based Video Super-resolution for Aerial Surveillance with Dual Cameras

Aerial surveillance requires high spatio-temporal resolution (HSTR) video for more accurate detection and tracking of objects. This is especially true for wide-area surveillance (WAS), where the surveyed region is large and the objects of interest are small. This paper proposes a dual camera system for the generation of HSTR video using reference-based super-resolution (RefSR). One camera captures high spatial resolution low frame rate (HSLF) video while the other captures low spatial resolution high frame rate (LSHF) video simultaneously for the same scene. A novel deep learning architecture is proposed to fuse HSLF and LSHF video feeds and synthesize HSTR video frames at the output. The proposed model combines optical flow estimation and (channel-wise and spatial) attention mechanisms to capture the fine motion and intricate dependencies between frames of the two video feeds. Simulations show that the proposed model provides significant improvement over existing reference-based SR techniques in terms of PSNR and SSIM metrics. The method also exhibits sufficient frames per second (FPS) for WAS when deployed on a power-constrained drone equipped with dual cameras.Comment: 15 pages, 8 figures, 8 table

GPU accelerated real-time multi-functional spectral-domain optical coherence tomography system at 1300 nm.

We present a GPU accelerated multi-functional spectral domain optical coherence tomography system at 1300 nm. The system is capable of real-time processing and display of every intensity image, comprised of 512 pixels by 2048 A-lines acquired at 20 frames per second. The update rate for all four images with size of 512 pixels by 2048 A-lines simultaneously (intensity, phase retardation, flow and en face view) is approximately 10 frames per second. Additionally, we report for the first time the characterization of phase retardation and diattenuation by a sample comprised of a stacked set of polarizing film and wave plate. The calculated optic axis orientation, phase retardation and diattenuation match well with expected values. The speed of each facet of the multi-functional OCT CPU-GPU hybrid acquisition system, intensity, phase retardation, and flow, were separately demonstrated by imaging a horseshoe crab lateral compound eye, a non-uniformly heated chicken muscle, and a microfluidic device. A mouse brain with thin skull preparation was imaged in vivo and demonstrated the capability of the system for live multi-functional OCT visualization