New image processing tools for structural dynamic monitoring

This paper presents an introduction to structural damage assessment using image processing on real data (non ideal conditions). Our contribution is much more a groundwork than a classical experimental validation. After measuring the bridge dynamic parameter on a small resolution video, we conjointly present advantages and limitations of our method. Finally we introduce several "computer vision" based rules and focus on the technical ability to detect damage using camera and video motion estimation

Mapping the spatiotemporal dynamics of calcium signaling in cellular neural networks using optical flow

An optical flow gradient algorithm was applied to spontaneously forming net- works of neurons and glia in culture imaged by fluorescence optical microscopy in order to map functional calcium signaling with single pixel resolution. Optical flow estimates the direction and speed of motion of objects in an image between subsequent frames in a recorded digital sequence of images (i.e. a movie). Computed vector field outputs by the algorithm were able to track the spatiotemporal dynamics of calcium signaling pat- terns. We begin by briefly reviewing the mathematics of the optical flow algorithm, and then describe how to solve for the displacement vectors and how to measure their reliability. We then compare computed flow vectors with manually estimated vectors for the progression of a calcium signal recorded from representative astrocyte cultures. Finally, we applied the algorithm to preparations of primary astrocytes and hippocampal neurons and to the rMC-1 Muller glial cell line in order to illustrate the capability of the algorithm for capturing different types of spatiotemporal calcium activity. We discuss the imaging requirements, parameter selection and threshold selection for reliable measurements, and offer perspectives on uses of the vector data.Comment: 23 pages, 5 figures. Peer reviewed accepted version in press in Annals of Biomedical Engineerin

A pedagogical image processing tool to understand structural dynamics

This paper presents a framework and one pedagogical application of motion tracking algorithms applied to structural dynamics. The aim of this work is to show the ability of high speed camera to study the dynamic characteristics of simple mechanical systems using a marker less and simultaneous Single Input Multiple Output (SIMO) broadband analysis. KLT (Kanade-Lucas-Tomasi) trackers are used as virtual sensors on mechanical systems video. First we introduce the paradigm of virtual sensors in the field of modal analysis using video processing. Then we present a pedagogical example of flexible beam (Fishing rod) video. From KLT tracking we extracted displacements data (virtual sensors) which are then enhanced using filtering and smoothing and then we can identify natural frequency and damping ratio from classical modal analysis. The experimental results (mode shapes) are compared to an analytical flexible beam model showing high correlation but also showing the limitation of linear analysis. The main interest of this paper is that displacements are simply measured using only video at FPS (Frame Per Second) that respects the Nyquist frequency. There is no target needed on the structure only few critical pixels that are good features to track and which become virtual sensors

Virtual vibration measurement using KLT motion tracking algorithm

This paper presents a practical framework and its applications of motion tracking algorithms applied to structural dynamics. Tracking points (“features”) across multiple images is a fundamental operation in many computer vision applications. The aim of this work is to show the capability of computer vision (CV) for estimating the dynamic characteristics of two mechanical systems using a non contact, marker less and simultaneous Single Input Multiple Output (SIMO) analysis. KLT (Kanade-Lucas-Tomasi) trackers are used as virtual sensors on mechanical systems video from high speed camera. First we introduce the paradigm of virtual sensors in the field of modal analysis using video processing. To validate our method, a simple experiment is proposed: an Oberst beam test with harmonic excitation (mode 1). Then with the example of helicopter blade, Frequency Response Functions (FRFs) reconstruction is carried out by introducing several signal processing enhancements (filtering, smoothing). The CV experimental results (frequencies, mode shapes) are compared with classical modal approach and FEM model showing high correlation. The main interest of this method is that displacements are simply measured using only video at FPS (Frame Per Second) respecting the Nyquist frequency

메모리 대역폭이 감소된 다중 프레임 레이트 옵티칼 플로우

학위논문 (박사)-- 서울대학교 대학원 : 전기·정보공학부, 2015. 2. 김수환.최근 high frame rate camera의 비약적인 발전으로 이미 4K 1000FPS camera가 출시되었고 휴대폰에서도 1080P 240FPS를 지원하고 있다. Camera의 Frame rate 증가는 optical flow의 구현에 시사하는 바가 큰데, 그 이유는 frame rate이 올라갈수록 frame 간의 움직임 크기가 줄어들기 때문이다. 그 동안 큰 움직임에 대한 부정확한 optical flow 문제를 해결하기 위해서 다양한 알고리즘이 사용되어 왔지만, 이로 인한 computation의 증가 또는 알고리즘 dependency로 인해 늘어난 연산 시간은 real-time operation에 제약으로 작용한다. 하지만 camera의 frame rate이 올라가면 모든 움직임들은 이에 반비례해서 작아지므로, 결국 high frame rate camera는 간단한 알고리즘으로 정확한 optical flow를 얻을 수 있는 길을 열고 있다. 본 논문은 accurate real-time optical flow의 구현을 위해서 multi-frame rate and multi-scale optical flow 알고리즘을 제안한다. High frame rate camera를 이용한 multi-frame rate and multi-scale optical flow 알고리즘은 real-time optical flow의 hardware 구현에 적합하도록 iterative calculation없는 알고리즘이다. Multi-frame rate 알고리즘은 다양한 frame rate의 optical flow를 연산하고 서로간의 연관관계를 이용하여 slow motion 뿐만 아니라 high motion에 관해서도 optical flow 결과를 얻게 함으로써 측정 가능한 움직임을 확장시킨 알고리즘이다. 이 알고리즘은 frame rate 증가에 따른 시스템 연산량 증가를 기존 연구의 O(n)에서 O(log n) 수준으로 감소시킴으로써 system performance에 의한 제약을 크게 줄인다. Multi scale 알고리즘은 high frame rate system을 위한 full density 지원 알고리즘이다. 또한 본 논문에서는 frame rate의 증가에 따른 external memory access bandwidth 증가 문제를 풀기 위해서 spatial & temporal bandwidth reduction 알고리즘을 제안한다. 이 방법은 기존 LK optical flow알고리즘의 연산 순서를 바꾸고, iterative sub-sampling scheme, temporal Gaussian tail cut 그리고 frame reuse 등 다양한 방식의 알고리즘들을 제안함으로써 high frame rate system의 external memory access bandwidth를 감소시킨다. 마지막으로 Multi-Frame rate and multi-scale optical flow 알고리즘의 Multi-scale 구조의 hardware 의 구현 시 multiplier의 개수를 mxm크기의 윈도우처리를 위해 m개의 multiplier를 이용해서 convolution방식으로 구현하던 기존의 방법을 윈도우의 크기에 상관없이 2개의 multiplier로 mxm multiplication을 구현하는 방식을 제안한다. 이 방식을 기반으로 multi frame rate과 multi-scale의 hardware architecture를 제안하고 single level LK optical flow의 fpga구현을 통해서 제안한 architecture의 hardware 동작을 검증한다. 이상의 과정들을 통해서 accurate real-time optical flow system을 위한 multi-frame rate and multi-scale optical flow의 알고리즘 제안부터 architecture 검증까지의 연구를 진행한다.차 례 초 록 i 차 례 iii 그림 목차 vii 표 목 차 x 제1장 서 론 1 1.1 연구 배경 1 1.2 연구 내용 4 1.3 논문 구성 6 제2장 이전연구 7 2.1 LK Optical Flow 7 2.2 Large Displacement Problem 10 2.2.1 Pyramidal LK Optical Flow 10 2.2.2 High Frame Rate Optical Flow 11 2.2.3 Oversampled Optical Flow System 11 2.2.4 High Frame Rate Optical Flow System 14 2.3 Problems in High Frame Rate System 16 2.3.1 Test Sequence for High Frame Rate System 16 2.3.2 Saturated Accuracy for High frame rate 17 2.3.3 Accurate displacement for LK optical flow 21 2.3.1 Accurate frame rate of High frame rate system 23 제3장 Multi-Frame Rate Optical Flow 28 3.1 Ideal and Real Optical Flow System 29 3.2 Multi-Frame Rate Optical Flow 31 3.3 Accurate Frame Rate Selection 33 3.3.1 Magnitude Selection Algorithm 33 3.3.2 Magnitude Algorithm Validation 36 3.3.3 SSD(Sum of Squared Difference) 45 3.3.4 Magnitude with NR Selection Algorithm 48 3.3.5 Temporal Aliasing 51 3.4 Multi Frame Rate Optical Flow Test Result 52 3.5 Comparisons with previous works 57 제4장 Multi-Scale Optical Flow 59 4.1 Pyramidal Level Selection 61 4.2 Level Selection Algorithm 62 4.3 Pyramidal Image Generation 63 4.4 Proposed Algorithm Verification 66 4.4.1 Accuracy comparison 66 4.4.2 연산량 및 알고리즘 특성 comparisons 67 4.4.3 Graphical Result with various test sequences 69 제5장 Memory Bandwidth Reduction 76 5.1 Single Level LK Optical Flow System 76 5.1.1 Bandwidth Problem 76 5.1.2 Matrix multiplication 77 5.1.3 FPGA Implementation 77 5.2 Spatial Bandwidth Reduction 79 5.2.1 LK Optical Flow System Architecture 79 5.2.2 External Memory Bandwidth Requirement 80 5.2.1 제안하는 알고리즘 83 5.2.2 Simulation Result 86 5.3 Temporal Bandwidth Reduction 93 5.3.1 Tail Cut 93 5.3.2 Frame Reuse 97 5.3.3 Experimental Result 101 5.4 Matrix Generation 107 5.4.1 Matrix Multiplication 107 5.4.2 Proposed Matrix Multiplication 108 5.5 FPGA Implementation 111 제6장 결론 117 참 고 문 헌 118 Abstract 122Docto

IEEE TRANSACTIONS ON IMAGE PROCESSING NO. XX, XXX 200X 1 Optical Flow Estimation Using Temporally Oversampled Video

Abstract — Recent advances in imaging sensor technology make high frame rate video capture practical. As demonstrated in previous work, this capability can be used to enhance the performance of many image and video processing applications. The idea is to use the high frame rate capability to temporally oversample the scene and thus to obtain more accurate information about scene motion and illumination. This information is then used to improve the performance of image and standard frame-rate video applications. The paper investigates the use of temporal oversampling to improve the accuracy of optical flow estimation (OFE). A method for obtaining high accuracy optical flow estimates at a conventional standard frame rate, e.g. 30 frames/s, by first capturing and processing a high frame rate version of the video is presented. The method uses the Lucas-Kanade algorithm to obtain optical flow estimates at a high frame rate, which are then accumulated and refined to estimate the optical flow at the desired standard frame rate. The method demonstrates significant improvements in optical flow estimation accuracy both on synthetically generated video sequences and on a real video sequence captured using an experimental highspeed imaging system. It is then shown that a key benefit of using temporal oversampling to estimate optical flow is the reduction in motion aliasing. Using sinusoidal input sequences, the reduction in motion aliasing is identified and the desired minimum sampling rate as a function of the velocity and spatial bandwidth of the scene is determined. Using both synthetic and real video sequences it is shown that temporal oversampling improves OFE accuracy by reducing motion aliasing not only for areas with large displacements but also for areas with small displacements and high spatial frequencies. The use of other OFE algorithms with temporally oversampled video is then discussed. In particular the Haussecker algorithm is extended to work with high frame rate sequences. This extension demonstrates yet another important benefit of temporal oversampling, which is improving OFE accuracy when brightness varies with time