Video Upright Adjustment and Stabilization

Upright adjustment, Video stabilization, Camera pathWe propose a novel video upright adjustment method that can reliably correct slanted video contents that are often found in casual videos. Our approach combines deep learning and Bayesian inference to estimate accurate rotation angles from video frames. We train a convolutional neural network to obtain initial estimates of the rotation angles of input video frames. The initial estimates from the network are temporally inconsistent and inaccurate. To resolve this, we use Bayesian inference. We analyze estimation errors of the network, and derive an error model. We then use the error model to formulate video upright adjustment as a maximum a posteriori problem where we estimate consistent rotation angles from the initial estimates, while respecting relative rotations between consecutive frames. Finally, we propose a joint approach to video stabilization and upright adjustment, which minimizes information loss caused by separately handling stabilization and upright adjustment. Experimental results show that our video upright adjustment method can effectively correct slanted video contents, and its combination with video stabilization can achieve visually pleasing results from shaky and slanted videos.openI. INTRODUCTION 1.1. Related work II. ROTATION ESTIMATION NETWORK III. ERROR ANALYSIS IV. VIDEO UPRIGHT ADJUSTMENT 4.1. Initial angle estimation 4.2. Robust angle estimation 4.3. Optimization 4.4. Warping V. JOINT UPRIGHT ADJUSTMENT AND STABILIZATION 5.1. Bundled camera paths for video stabilization 5.2. Joint approach VI. EXPERIMENTS VII. CONCLUSION ReferencesCNN)을 훈련시킨다. 신경망의 초기 추정치는 완전히 정확하지 않으며 시간적으로도 일관되지 않는다. 이를 해결하기 위해 베이지안 인퍼런스를 사용한다. 본 논문은 신경망의 추정 오류를 분석하고 오류 모델을 도출한다. 그런 다음 오류 모델을 사용하여 연속 프레임 간의 상대 회전 각도(Relative rotation angle)를 반영하면서 초기 추정치로부터 시간적으로 일관된 회전 각도를 추정하는 최대 사후 문제(Maximum a posteriori problem)로 동영상 수평 보정을 공식화한다. 마지막으로, 동영상 수평 보정 및 동영상 안정화(Video stabilization)에 대한 동시 접근 방법을 제안하여 수평 보정과 안정화를 별도로 수행할 때 발생하는 공간 정보 손실과 연산량을 최소화하며 안정화의 성능을 최대화한다. 실험 결과에 따르면 동영상 수평 보정으로 기울어진 동영상을 효과적으로 보정할 수 있으며 동영상 안정화 방법과 결합하여 흔들리고 기울어진 동영상으로부터 시각적으로 만족스러운 새로운 동영상을 획득할 수 있다.본 논문은 일반인들이 촬영한 동영상에서 흔히 발생하는 문제인 기울어짐을 제거하여 수평이 올바른 동영상을 획득할 수 있게 하는 동영상 수평 보정(Video upright adjustment) 방법을 제안한다. 본 논문의 접근 방식은 딥 러닝(Deep learning)과 베이지안 인퍼런스(Bayesian inference)를 결합하여 동영상 프레임(Frame)에서 정확한 각도를 추정한다. 먼저 입력 동영상 프레임의 회전 각도의 초기 추정치를 얻기 위해 회선 신경망(Convolutional neural networkMasterdCollectio

Video Acceleration Magnification

The ability to amplify or reduce subtle image changes over time is useful in contexts such as video editing, medical video analysis, product quality control and sports. In these contexts there is often large motion present which severely distorts current video amplification methods that magnify change linearly. In this work we propose a method to cope with large motions while still magnifying small changes. We make the following two observations: i) large motions are linear on the temporal scale of the small changes; ii) small changes deviate from this linearity. We ignore linear motion and propose to magnify acceleration. Our method is pure Eulerian and does not require any optical flow, temporal alignment or region annotations. We link temporal second-order derivative filtering to spatial acceleration magnification. We apply our method to moving objects where we show motion magnification and color magnification. We provide quantitative as well as qualitative evidence for our method while comparing to the state-of-the-art.Comment: Accepted paper at CVPR 2017. Project webpage: http://acceleration-magnification.github.io

Fast Full-frame Video Stabilization with Iterative Optimization

Video stabilization refers to the problem of transforming a shaky video into a visually pleasing one. The question of how to strike a good trade-off between visual quality and computational speed has remained one of the open challenges in video stabilization. Inspired by the analogy between wobbly frames and jigsaw puzzles, we propose an iterative optimization-based learning approach using synthetic datasets for video stabilization, which consists of two interacting submodules: motion trajectory smoothing and full-frame outpainting. First, we develop a two-level (coarse-to-fine) stabilizing algorithm based on the probabilistic flow field. The confidence map associated with the estimated optical flow is exploited to guide the search for shared regions through backpropagation. Second, we take a divide-and-conquer approach and propose a novel multiframe fusion strategy to render full-frame stabilized views. An important new insight brought about by our iterative optimization approach is that the target video can be interpreted as the fixed point of nonlinear mapping for video stabilization. We formulate video stabilization as a problem of minimizing the amount of jerkiness in motion trajectories, which guarantees convergence with the help of fixed-point theory. Extensive experimental results are reported to demonstrate the superiority of the proposed approach in terms of computational speed and visual quality. The code will be available on GitHub.Comment: Accepted by ICCV202

A Primer for Folklore Videographers

This is an orientation for novices beginning to shoot interviews and events. The cost of making and distributing videos on the Internet continues to drop and now the primary barrier to using video is acquiring the storytelling and technical skills to make movies that people will watch

Exploring Hollywood Cinema: A Look at Cinematic Techniques and the Classical Hollywood Ideology

The focus of this thesis is to explore the ideas behind Hollywood Cinema and techniques used to craft camera shots of today’s cinema. The movie industry is one of the United States’ biggest export profits; because of this, American movies standout above the rest. I will be exploring and demonstrating the researched techniques of American cinema

Autonomous Execution of Cinematographic Shots with Multiple Drones

This paper presents a system for the execution of autonomous cinematography missions with a team of drones. The system allows media directors to design missions involving different types of shots with one or multiple cameras, running sequentially or concurrently. We introduce the complete architecture, which includes components for mission design, planning and execution. Then, we focus on the components related to autonomous mission execution. First, we propose a novel parametric description for shots, considering different types of camera motion and tracked targets; and we use it to implement a set of canonical shots. Second, for multi-drone shot execution, we propose distributed schedulers that activate different shot controllers on board the drones. Moreover, an event-based mechanism is used to synchronize shot execution among the drones and to account for inaccuracies during shot planning. Finally, we showcase the system with field experiments filming sport activities, including a real regatta event. We report on system integration and lessons learnt during our experimental campaigns

Little Trees

I believe in the power of film as a medium of storytelling. My desire to head a year-long production as well as create a moving story that underlined my interest in sibling relationships motivated me to create a short film as my honors thesis. My film, Little Trees, emphasizes themes such as alienation and growth within a broken household. My ultimate goal is to become an artist who creates meaningful cinema by telling deeply personal stories, and creating Little Trees was a successful first step in that process

Style Talk: A collection of Interviews on Personal Style

My concept for this creative Capstone project is to see how much of a correlation there is between culture and fashion. I wanted to speak to people from a variety of backgrounds to see how much a person’s family and culture impacts the way one regards fashion and affects the way one dresses. In order to have a diverse set of interviews, I chose to interview three American female students and three non-American female students who attend Syracuse University. In the past, I have noticed that many style shows focus on the current trends and do profiles on celebrities and fashion designers. Those stories are filled with entertainment news and are intended to feed us information about what’s up and coming in the fashion and beauty world. There are television pieces I have seen online, in which interview centers on the designer and a couple models before a runway show. I appreciate interviews that take the time to discover the designer’s inspiration for a collection and the model’s experience preparing for the show, because it seems more intimate than a typical promotional interview. Jessica Simpson’s The Price of Beauty, which aired on VH1 in March 2010, was a show aimed at discussing culture with fashion and beauty with local citizens around the world. It seemed coincidental that her show came out around the time I developed an idea for my Capstone. I knew that different cultures have varying opinions on fashion and beauty, but I wanted to localize that idea, since it’s out of my budget to fly around the world. Syracuse University has a diverse body of students and I wanted to focus on the individual who happens to be part of a specific culture. That way I can get to know that female’s personal story and not stereotype her culture’s perception of fashion and beauty. It will solely be the views of someone who comes from a particular culture. As a Television-Radio-Film major, I used video as my medium, but also incorporated still images and graphics into the videos, so it would be a multiplatform project. I did short interviews instead of one short twenty-minute movie, so the interviews would be easy to upload online and people can watch one of the interviews during a quick five-minute break. Through a handheld flipcamera, I recorded my interviews. The advantage of the flipcamera was the ability to easily upload my footage to any computer and edit in the specialized labs that contain the video editing programs. After putting all the interviews together, I branded the project as Style Talk. It’s a simple, self-explanatory title that fits the project perfectly. My interviews focused on individuals to talk about their personal style. It’s a direct process where these young women tell their story to the camera and talk about their style, hence Style Talk. I noticed throughout the interviews that all the girls had a varied amount of interest in regards to fashion. Even the girl who would be classified as the “least fashionable” still puts effort into her look knowing that she doesn’t follow the high-end fashion magazines and works with pieces that she can afford and tailors them to suit her personal needs. Overall, family and culture has had some effect on the way these girls view fashion. What they’ve observed with fashion among their family, friends, and the media, has given them a foundation of how they perceive fashion, in which they have developed their personal style over the years, so now they can make independent decisions in regards to what they wear on a daily basis

高速ビジョンを用いたリアルタイムビデオモザイキングと安定化に関する研究

広島大学(Hiroshima University)博士(工学)Doctor of Engineeringdoctora