Detecting and removing visual distractors for video aesthetic enhancement

Personal videos often contain visual distractors, which are objects that are accidentally captured that can distract viewers from focusing on the main subjects. We propose a method to automatically detect and localize these distractors through learning from a manually labeled dataset. To achieve spatially and temporally coherent detection, we propose extracting features at the Temporal-Superpixel (TSP) level using a traditional SVM-based learning framework. We also experiment with end-to-end learning using Convolutional Neural Networks (CNNs), which achieves slightly higher performance than other methods. The classification result is further refined in a post-processing step based on graph-cut optimization. Experimental results show that our method achieves an accuracy of 81% and a recall of 86%. We demonstrate several ways of removing the detected distractors to improve the video quality, including video hole filling; video frame replacement; and camera path re-planning. The user study results show that our method can significantly improve the aesthetic quality of videos

Video Upright Adjustment and Stabilization

Upright adjustment, Video stabilization, Camera pathWe propose a novel video upright adjustment method that can reliably correct slanted video contents that are often found in casual videos. Our approach combines deep learning and Bayesian inference to estimate accurate rotation angles from video frames. We train a convolutional neural network to obtain initial estimates of the rotation angles of input video frames. The initial estimates from the network are temporally inconsistent and inaccurate. To resolve this, we use Bayesian inference. We analyze estimation errors of the network, and derive an error model. We then use the error model to formulate video upright adjustment as a maximum a posteriori problem where we estimate consistent rotation angles from the initial estimates, while respecting relative rotations between consecutive frames. Finally, we propose a joint approach to video stabilization and upright adjustment, which minimizes information loss caused by separately handling stabilization and upright adjustment. Experimental results show that our video upright adjustment method can effectively correct slanted video contents, and its combination with video stabilization can achieve visually pleasing results from shaky and slanted videos.openI. INTRODUCTION 1.1. Related work II. ROTATION ESTIMATION NETWORK III. ERROR ANALYSIS IV. VIDEO UPRIGHT ADJUSTMENT 4.1. Initial angle estimation 4.2. Robust angle estimation 4.3. Optimization 4.4. Warping V. JOINT UPRIGHT ADJUSTMENT AND STABILIZATION 5.1. Bundled camera paths for video stabilization 5.2. Joint approach VI. EXPERIMENTS VII. CONCLUSION ReferencesCNN)을 훈련시킨다. 신경망의 초기 추정치는 완전히 정확하지 않으며 시간적으로도 일관되지 않는다. 이를 해결하기 위해 베이지안 인퍼런스를 사용한다. 본 논문은 신경망의 추정 오류를 분석하고 오류 모델을 도출한다. 그런 다음 오류 모델을 사용하여 연속 프레임 간의 상대 회전 각도(Relative rotation angle)를 반영하면서 초기 추정치로부터 시간적으로 일관된 회전 각도를 추정하는 최대 사후 문제(Maximum a posteriori problem)로 동영상 수평 보정을 공식화한다. 마지막으로, 동영상 수평 보정 및 동영상 안정화(Video stabilization)에 대한 동시 접근 방법을 제안하여 수평 보정과 안정화를 별도로 수행할 때 발생하는 공간 정보 손실과 연산량을 최소화하며 안정화의 성능을 최대화한다. 실험 결과에 따르면 동영상 수평 보정으로 기울어진 동영상을 효과적으로 보정할 수 있으며 동영상 안정화 방법과 결합하여 흔들리고 기울어진 동영상으로부터 시각적으로 만족스러운 새로운 동영상을 획득할 수 있다.본 논문은 일반인들이 촬영한 동영상에서 흔히 발생하는 문제인 기울어짐을 제거하여 수평이 올바른 동영상을 획득할 수 있게 하는 동영상 수평 보정(Video upright adjustment) 방법을 제안한다. 본 논문의 접근 방식은 딥 러닝(Deep learning)과 베이지안 인퍼런스(Bayesian inference)를 결합하여 동영상 프레임(Frame)에서 정확한 각도를 추정한다. 먼저 입력 동영상 프레임의 회전 각도의 초기 추정치를 얻기 위해 회선 신경망(Convolutional neural networkMasterdCollectio

Lessons in looking : the digital audiovisual essay

This thesis examines the contemporary practice of the digital audiovisual essay, which is defined as a material form of thinking at the crossroads of academic textual analysis, personal cinephilia, and popular online fandom practices, to suggest that it allows rich epistemological discoveries not only about individual films and viewing experiences, but also about how cinema is perceived in the context of digitally mediated audiovisual culture. Chapter one advances five key defining tensions of the digital audiovisual essay: its object is the investigation of specific films and cinephiliac experiences; it uses a performative research methodology based on the affordances of digital viewing and editing technologies; it exists primarily in Web 2.0 and takes advantage of its collaborative and dialogical modes of production; it is a “rich text object” that continuously tests the different contributions of both verbal and audiovisual forms of communication to the production of knowledge about the cinema; and finally, the digital audiovisual essay has an important pedagogical potential, not only for those who watch it, but especially for those who practice it. Chapter two presents the theoretical framework of the dissertation, challenges the ‘newness’ of the digital audiovisual essay, and suggests that any investigation of this cultural practice must address its ideological implications and its role in the context of contemporary audiovisual culture. Accordingly, it relates the editing and compositional techniques of the digital audiovisual essay with modernist montage and suggests that the audiovisual essay has not only inherited, but has also updated and enhanced the dialectical interdependency between critical and consumerism drives that shaped modernism’s ambiguous relation to mass culture. The final chapter examines four case studies (David Bordwell, Catherine Grant, ::kogonada, and Kevin B. Lee) that showcase the contradictory tensions of this cultural practice and broad our understanding of the politics of the digital audiovisual essay

Content Seeking Students: Site-and-Sound Bites as Participants in Ubiquitous Social Computing

Discussion of digital, collaborative environments for architectural work often focuses on the structure of discourse, rather than upon its substance. An implied assumption is that the various means of electronic-based communication are suitable for any kind of subject matter, whether visual, sound-based, or text. Our project team has chosen to challenge this assumption by example: We have created new media artifacts for collaborative architectural education. Our project is an attempt to leverage on-going research concerning the efficacy of "ubiquitous social computing” (USC) for design-studio teaching. With a pilot project already put in place by one of our team's leaders, we have supplemented graphic and verbal communication among participants with purpose-crafted video for their use and exchange. Smart-screens, placed strategically within students' "social enclaves,” provide access to curated content. Our approach challenges traditional educational emphasis upon explicit types of architectural knowledge. The construction of tacit knowledge, usually derived from first-hand architectural experiences, is here effected by mediated, digital-based experiences. Nevertheless, the social dimension of the USC framework may be significant in negotiating the interface between immediate and mediated experiences

Bootlegger : Turning Fans into Film Crew

Abstract Bootlegger is a system for creating multi-camera films of live music events using mobile devices. Using readily available technology and a synthesis of film-making conventions, the system coordinates music fans at live shows into an improvised film crew, suggesting shots, collating footage and generating rich metadata in real time. Bootlegger is part of a research project exploring adapting professional media workflows to amateur contexts in order to lower the bar to entry for media production. By enabling concert-goers ..