Object-based 2D-to-3D video conversion for effective stereoscopic content generation in 3D-TV applications

Three-dimensional television (3D-TV) has gained increasing popularity in the broadcasting domain, as it enables enhanced viewing experiences in comparison to conventional two-dimensional (2D) TV. However, its application has been constrained due to the lack of essential contents, i.e., stereoscopic videos. To alleviate such content shortage, an economical and practical solution is to reuse the huge media resources that are available in monoscopic 2D and convert them to stereoscopic 3D. Although stereoscopic video can be generated from monoscopic sequences using depth measurements extracted from cues like focus blur, motion and size, the quality of the resulting video may be poor as such measurements are usually arbitrarily defined and appear inconsistent with the real scenes. To help solve this problem, a novel method for object-based stereoscopic video generation is proposed which features i) optical-flow based occlusion reasoning in determining depth ordinal, ii) object segmentation using improved region-growing from masks of determined depth layers, and iii) a hybrid depth estimation scheme using content-based matching (inside a small library of true stereo image pairs) and depth-ordinal based regularization. Comprehensive experiments have validated the effectiveness of our proposed 2D-to-3D conversion method in generating stereoscopic videos of consistent depth measurements for 3D-TV applications

Stereoscopic theatre: the impact of gestalt perceptual organization in the stereoscopic theatre environment

This paper argues that it is essential for live theatre that incorporates stereoscopic imagery to reconceptualise the performance space to facilitate a successful audience experience. While 3D technology greatly increases artistic possibilities, the risks of perceptual confusion exist in live theatre just as in stereoscopic cinema, indeed more so given the co-existence of live performers. This paper argues that Gestalt perceptual organization theory can be valuable in informing how best to employ stereoscopic imagery within a live theater environment, with reference to the artistic works of one of the authors

CycleISP: Real Image Restoration via Improved Data Synthesis

The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However, for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synthetic data that is usually generated with a widespread assumption of additive white Gaussian noise (AWGN). While the CNNs achieve impressive results on these synthetic datasets, they do not perform well when applied on real camera images, as reported in recent benchmark datasets. This is mainly because the AWGN is not adequate for modeling the real camera noise which is signal-dependent and heavily transformed by the camera imaging pipeline. In this paper, we present a framework that models camera imaging pipeline in forward and reverse directions. It allows us to produce any number of realistic image pairs for denoising both in RAW and sRGB spaces. By training a new image denoising network on realistic synthetic data, we achieve the state-of-the-art performance on real camera benchmark datasets. The parameters in our model are ~5 times lesser than the previous best method for RAW denoising. Furthermore, we demonstrate that the proposed framework generalizes beyond image denoising problem e.g., for color matching in stereoscopic cinema. The source code and pre-trained models are available at https://github.com/swz30/CycleISP.Comment: CVPR 2020 (Oral

Gradient-based 2D-to-3D Conversion for Soccer Videos

A wide spread adoption of 3D videos and technologies is hindered by the lack of high-quality 3D content. One promising solution to address this problem is to use automated 2D-to-3D conversion. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We address this problem by showing how to construct a high-quality, domain-specific conversion method for soccer videos. We propose a novel, data-driven method that generates stereoscopic frames by transferring depth information from similar frames in a database of 3D stereoscopic videos. Creating a database of 3D stereoscopic videos with accurate depth is, however, very difficult. One of the key findings in this paper is showing that computer generated content in current sports computer games can be used to generate high-quality 3D video reference database for 2D-to-3D conversion methods. Once we retrieve similar 3D video frames, our technique transfers depth gradients to the target frame while respecting object boundaries. It then computes depth maps from the gradients, and generates the output stereoscopic video. We implement our method and validate it by conducting user-studies that evaluate depth perception and visual comfort of the converted 3D videos. We show that our method produces high-quality 3D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-the-art method. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from Good to Excellent.Qatar Computing Research Institute-CSAIL PartnershipNational Science Foundation (U.S.) (Grant IIS-1111415

FROM CELLULOID REALITIES TO BINARY DREAMSCAPES: CINEMA AND PERCEPTUAL EXPERIENCE IN THE AGE OF DIGITAL IMMERSION

Technologies in digital cinema are quickly changing the way contemporary filmmakers create films and how audiences currently perceive them. As we move onward into the digital turn, it becomes ever more apparent that the medium of film has been emancipated from its dependence on the photograph. Directors are no longer required to capture the objectively real as it sits before the photographic lens, but can essentially construct it via groundbreaking advancements in computer-generated imagery, motion capture technology, and digital 3D camera systems and display technologies. Since the origins of film, spectators and filmmakers have assumed an existing relationship between reality and the photographic image. Yet digital film technologies now provide us with hyper-facsimiles of reality that are perceived as photographic, but are often created by way of computer processes. Digital cinema currently allows the viewer to inhabit and interact with cinematic realities in unprecedented ways, and it is this contemporary paradigmatic shift from the analog to the digital that has catalyzed fundamentally new ways of looking at the filmic image. In this paper, I will examine the perceptual complexities of contemporary digital film through the lens of these cinematic technologies by examining their impact on the viewer’s experience

Interactive Visual Analytics for Large-scale Particle Simulations

Particle based model simulations are widely used in scientific visualization. In cosmology, particles are used to simulate the evolution of dark matter in the universe. Clusters of particles (that have special statistical properties) are called halos. From a visualization point of view, halos are clusters of particles, each having a position, mass and velocity in three dimensional space, and they can be represented as point clouds that contain various structures of geometric interest such as filaments, membranes, satellite of points, clusters, and cluster of clusters. The thesis investigates methods for interacting with large scale data-sets represented as point clouds. The work mostly aims at the interactive visualization of cosmological simulation based on large particle systems. The study consists of three components: a) two human factors experiments into the perceptual factors that make it possible to see features in point clouds; b) the design and implementation of a user interface making it possible to rapidly navigate through and visualize features in the point cloud, c) software development and integration to support visualization