Optimization techniques for computationally expensive rendering algorithms

Realistic rendering in computer graphics simulates the interactions of light and surfaces. While many accurate models for surface reflection and lighting, including solid surfaces and participating media have been described; most of them rely on intensive computation. Common practices such as adding constraints and assumptions can increase performance. However, they may compromise the quality of the resulting images or the variety of phenomena that can be accurately represented. In this thesis, we will focus on rendering methods that require high amounts of computational resources. Our intention is to consider several conceptually different approaches capable of reducing these requirements with only limited implications in the quality of the results. The first part of this work will study rendering of time-­¿varying participating media. Examples of this type of matter are smoke, optically thick gases and any material that, unlike the vacuum, scatters and absorbs the light that travels through it. We will focus on a subset of algorithms that approximate realistic illumination using images of real world scenes. Starting from the traditional ray marching algorithm, we will suggest and implement different optimizations that will allow performing the computation at interactive frame rates. This thesis will also analyze two different aspects of the generation of anti-­¿aliased images. One targeted to the rendering of screen-­¿space anti-­¿aliased images and the reduction of the artifacts generated in rasterized lines and edges. We expect to describe an implementation that, working as a post process, it is efficient enough to be added to existing rendering pipelines with reduced performance impact. A third method will take advantage of the limitations of the human visual system (HVS) to reduce the resources required to render temporally antialiased images. While film and digital cameras naturally produce motion blur, rendering pipelines need to explicitly simulate it. This process is known to be one of the most important burdens for every rendering pipeline. Motivated by this, we plan to run a series of psychophysical experiments targeted at identifying groups of motion-­¿blurred images that are perceptually equivalent. A possible outcome is the proposal of criteria that may lead to reductions of the rendering budgets

Egocentric Spatial Representation in Action and Perception

Neuropsychological findings used to motivate the “two visual systems” hypothesis have been taken to endanger a pair of widely accepted claims about spatial representation in visual experience. The first is the claim that visual experience represents 3-D space around the perceiver using an egocentric frame of reference. The second is the claim that there is a constitutive link between the spatial contents of visual experience and the perceiver’s bodily actions. In this paper, I carefully assess three main sources of evidence for the two visual systems hypothesis and argue that the best interpretation of the evidence is in fact consistent with both claims. I conclude with some brief remarks on the relation between visual consciousness and rational agency

The Hobbit: The Desolation of Smaug - A New Era of Realism?

Peter Jackson’s Hobbit trilogy is a familiar blockbuster franchise, adapting a well-known piece of literature and designed to appeal to global audiences. This trilogy, however, is also experimental, as the premium release of each film utilised higher frame rate (HFR) technologies together with computer generated imaging (CGI) and 3D in ways that were intended to extent the apparatus of cinema itself. These technological processes are part of a long line of developments aimed at creating a more compelling cinematic experience (Michelle, Davis, Hight, and Hardy, 2015). 3D film is thought to be the culmination of technological advances in film as the format’s ‘implicit mission was to conquer the entire sensorial complex, to represent reality in its totality’ (Asselin and Gosselin, 2013, p.132). Thus, this thesis focuses on the reception of global audiences to the technological aspects of the second film of The Hobbit franchise, The Desolation of Smaug (2013), focusing on whether 3D HFR quantifiably alters viewers’ viewing experience in terms of improving perceptions of realism and immersion. The research draws from a mixture of qualitative and quantitative methods, including an online survey of 650 respondents across multiple countries and 39 Skype and email follow-up interviews. The responses to and interpretations of a self-selected audience formed the basis of understanding whether these technological advancements have created a more perceptually realistic and immersive cinematic experience. The findings from this research indicate that these new technologies were a challenge to many of the expectations of Hobbit viewers. Despite general approval of the nature of these technologies and their possibilities for enhancing the aesthetic experience of cinema, key segments of the audience were clearly disenchanted with these innovations, especially in comparison with their experience of Jackson’s earlier Lord of the Rings trilogy (2001-2003) and other CGI-based and 3D cinema. Respondents outlined problems in the interplay between the different imaging techniques, which generated jarring visual artefacts. They critiqued scenes where the filmmakers failed to seamlessly meld the technologies effectively, and many reported being frustrated at interruptions to their efforts to immerse themselves in the film’s narrative. Furthermore, my findings suggest that ultimately, the 3D HFR technologies and the aesthetic presented were subsidiary issues to the narrative surrounding the Middle-earth world that emotionally resonates with the majority of respondents. This does not mean that these interviewees found 3D HFR technology to have clashed with the narrative, but that the film ultimately stood as a return to their Middle-earth world. These 3 responses are consistent with those noted by Michelle et al. (2015), who found that existing fan communities of Tolkien and the Lord of the Rings trilogy had complex reactions to the use of 3D HFR technologies and their impact on The Hobbit films. These findings suggest a mixed future for similar efforts to advance cinematic aesthetics through new technologies

Depth, shading, and stylization in stereoscopic cinematography

Due to the constantly increasing focus of the entertainment industry on stereoscopic imaging, techniques and tools that enable precise control over the depth impression and help to overcome limitations of the current stereoscopic hardware are gaining in importance. In this dissertation, we address selected problems encountered during stereoscopic content production, with a particular focus on stereoscopic cinema. First, we consider abrupt changes of depth, such as those induced by cuts in films. We derive a model predicting the time the visual system needs to adapt to such changes and propose how to employ this model for film cut optimization. Second, we tackle the issue of discrepancies between the two views of a stereoscopic image due to view-dependent shading of glossy materials. The suggested solution eliminates discomfort caused by non-matching specular highlights while preserving the perception of gloss. Last, we deal with the problem of filmgrainmanagement in stereoscopic productions and propose a new method for film grain application that reconciles visual comfort with the idea of medium-scene separation.Aufgrund der ständig steigenden Beachtung der stereoskopische Abbildung durch die Unterhaltungsindustrie, gewinnen Techniken und Werkzeuge an Bedeutung, die eine präzise Steuerung der Tiefenwahrnehmung ermöglichen und Einschränkungen der gegenwärtigen stereoskopischen Geräte überwinden. In dieser Dissertation adressieren wir ausgewählte Probleme, die während der Erzeugung von stereoskopischen Inhalten auftreten, mit besonderem Schwerpunkt auf der stereoskopischen Kinematographie. Zuerst betrachten wir abrupte Tiefenänderungen, wie sie durch Filmschnitte hervergerufen werden. Wir leiten ein Modell her, das die Zeit vorhersagt, die für das menschliche Sehsystem notwendig ist, um sich an solche Änderungen der Tiefe zu adaptieren, und schlagen vor wie dieses Modell für Schnittoptimierung angewendet werden kann. Danach gehen wir das Problem der Unstimmigkeiten zwischen den zwei Ansichten eines stereoskopischen Bildes, infolge der sichtabhängigen Schattierung von glänzenden Materialien, an. Die vorgeschlagene Lösung eliminiert das visuelle Unbehagen, welches von nicht zusammenpassenden Glanzlichtern verursacht wird, indessen bewahrt sie die Glanzwahrnehmung. Zuletzt behandeln wir das Problem des Filmkornsmanagements in stereoskopischen Produktionen und schlagen eine neue Methode für das Hinzufügen vom Filmkorn vor, die die visuelle Behaglichkeit mit der Idee der Medium-Szenen-Trennung in Einklang bringt

Foveation for 3D visualization and stereo imaging

Even though computer vision and digital photogrammetry share a number of goals, techniques, and methods, the potential for cooperation between these fields is not fully exploited. In attempt to help bridging the two, this work brings a well-known computer vision and image processing technique called foveation and introduces it to photogrammetry, creating a hybrid application. The results may be beneficial for both fields, plus the general stereo imaging community, and virtual reality applications. Foveation is a biologically motivated image compression method that is often used for transmitting videos and images over networks. It is possible to view foveation as an area of interest management method as well as a compression technique. While the most common foveation applications are in 2D there are a number of binocular approaches as well. For this research, the current state of the art in the literature on level of detail, human visual system, stereoscopic perception, stereoscopic displays, 2D and 3D foveation, and digital photogrammetry were reviewed. After the review, a stereo-foveation model was constructed and an implementation was realized to demonstrate a proof of concept. The conceptual approach is treated as generic, while the implementation was conducted under certain limitations, which are documented in the relevant context. A stand-alone program called Foveaglyph is created in the implementation process. Foveaglyph takes a stereo pair as input and uses an image matching algorithm to find the parallax values. It then calculates the 3D coordinates for each pixel from the geometric relationships between the object and the camera configuration or via a parallax function. Once 3D coordinates are obtained, a 3D image pyramid is created. Then, using a distance dependent level of detail function, spherical volume rings with varying resolutions throughout the 3D space are created. The user determines the area of interest. The result of the application is a user controlled, highly compressed non-uniform 3D anaglyph image. 2D foveation is also provided as an option. This type of development in a photogrammetric visualization unit is beneficial for system performance. The research is particularly relevant for large displays and head mounted displays. Although, the implementation, because it is done for a single user, would possibly be best suited to a head mounted display (HMD) application. The resulting stereo-foveated image can be loaded moderately faster than the uniform original. Therefore, the program can potentially be adapted to an active vision system and manage the scene as the user glances around, given that an eye tracker determines where exactly the eyes accommodate. This exploration may also be extended to robotics and other robot vision applications. Additionally, it can also be used for attention management and the viewer can be directed to the object(s) of interest the demonstrator would like to present (e.g. in 3D cinema). Based on the literature, we also believe this approach should help resolve several problems associated with stereoscopic displays such as the accommodation convergence problem and diplopia. While the available literature provides some empirical evidence to support the usability and benefits of stereo foveation, further tests are needed. User surveys related to the human factors in using stereo foveated images, such as its possible contribution to prevent user discomfort and virtual simulator sickness (VSS) in virtual environments, are left as future work.reviewe

Credulous Spectatorship from Zeuxis to Barthes

This dissertation explores intersections between trompe l\u27oeil painting and photography. It began as an interest in contemporary photographers, such as Thomas Demand, whose photographs of constructed paper models encourage viewers to discover the nature of his interventions. His strategy resonates with a centuries-old strategy in trompe l\u27oeil painting, but now in the terms of photographic, rather than pictorial presence. That is, most of Demand\u27s photographs do not compel the viewer\u27s belief in the tangible presence of the object represented; instead, they exploit photography\u27s indexical promise of delivering the world as it once appeared, in order to temporarily trick viewers about the terms of that indexical delivery. Beyond intersections in artistic strategies, I track reception accounts of trompe l\u27oeil painting and photography for their reliance on a credulous spectator. Pliny\u27s Zeuxis, who is tricked by Parrhasius\u27s painting of a curtain, remains the model for this errant credulity. In their efforts to reveal the manipulation of photographs, historians and theorists assume that the natural attitude for viewing photographs is wholly credulous and recast postmodern viewers as contemporary Zeuxises. Instead of admonishing spectators for such credulity, I argue that trompe l\u27oeil facilitates a pleasurable experience of oscillation between belief and disbelief. I also suggest that these trompe l’oeil deployments of oscillation tend to coincide with historical moments of perceived change in visual technologies—changes due to digitalization, as well as mechanical or other forms of reproduction. Trompe l\u27oeil artists play upon our supposed willingness to accept reproductions for the objects they represent. The inclusion of photographs and/or engravings in these trompe l’oeil paintings simultaneously stages and reprimands our desire for the aura of the actual object. Finally, I suggest that a contemporary renewal of trompe l\u27oeil in the medium of photography reveals an interest in recuperating belief in photographs—a belief not unlike that which Roland Barthes narrates in Camera Lucida. Just as Barthes can discover something of photography\u27s indexical promise, even after decades of his own scholarly efforts to unveil photography\u27s rhetoric of construction, so might we, even while heeding the postmodernist lessons of disbelief, recuperate a moment of belief in a skeptical age

Credulous Spectatorship from Zeuxis to Barthes

This dissertation explores intersections between trompe l\u27oeil painting and photography. It began as an interest in contemporary photographers, such as Thomas Demand, whose photographs of constructed paper models encourage viewers to discover the nature of his interventions. His strategy resonates with a centuries-old strategy in trompe l\u27oeil painting, but now in the terms of photographic, rather than pictorial presence. That is, most of Demand\u27s photographs do not compel the viewer\u27s belief in the tangible presence of the object represented; instead, they exploit photography\u27s indexical promise of delivering the world as it once appeared, in order to temporarily trick viewers about the terms of that indexical delivery. Beyond intersections in artistic strategies, I track reception accounts of trompe l\u27oeil painting and photography for their reliance on a credulous spectator. Pliny\u27s Zeuxis, who is tricked by Parrhasius\u27s painting of a curtain, remains the model for this errant credulity. In their efforts to reveal the manipulation of photographs, historians and theorists assume that the natural attitude for viewing photographs is wholly credulous and recast postmodern viewers as contemporary Zeuxises. Instead of admonishing spectators for such credulity, I argue that trompe l\u27oeil facilitates a pleasurable experience of oscillation between belief and disbelief. I also suggest that these trompe l’oeil deployments of oscillation tend to coincide with historical moments of perceived change in visual technologies—changes due to digitalization, as well as mechanical or other forms of reproduction. Trompe l\u27oeil artists play upon our supposed willingness to accept reproductions for the objects they represent. The inclusion of photographs and/or engravings in these trompe l’oeil paintings simultaneously stages and reprimands our desire for the aura of the actual object. Finally, I suggest that a contemporary renewal of trompe l\u27oeil in the medium of photography reveals an interest in recuperating belief in photographs—a belief not unlike that which Roland Barthes narrates in Camera Lucida. Just as Barthes can discover something of photography\u27s indexical promise, even after decades of his own scholarly efforts to unveil photography\u27s rhetoric of construction, so might we, even while heeding the postmodernist lessons of disbelief, recuperate a moment of belief in a skeptical age

Never Seen Before--Again: Aesthetic Adaptation and Transmedia Storytelling Networks in the Age of Digital Cinema

The primary purpose of this study is to re-evaluate the theorization of the narrative/non-narrative divide in cinema because there is still far too much rich information that is being lost in the chasm left between these two positions, particularly when it comes to the study of spectacular imagery. Working beyond the binary construction of a narrative/non-narrative divide allows for a more nuanced and dynamic analysis of the information that is conveyed by the cinematic image and, more specifically, by way of its aesthetic design. By no means is this study championing image over story or any other such formulation; rather it aims to free both image and story from the constraints of narratological binary models and the binary language of narratology, because in practice cinema does not adhere to narrative/non-narrative narratological models. It is through an exploration of contemporary theories of visual effects that I launch my investigation. I argue that there are many agents (and agencies) that convey information in the cinema and that not all information will come in a form that is easily described by narratological terminology. Visual effects have become a focal point of the narrative/non-narrative divide and thus provide a constructive territory in which to interrogate: (1) how visual effects theory influences the theorization of contemporary film aesthetics; (2) how as film theorists we have arrived at this moment in which narrative disruptions occupy such an integral part of digital film theory; and (3), what we are missing if we maintain our current binaristic narrative/non-narrative rhetoric. Ultimately, the purpose of this study is to propose a dynamic, non-binary model to address the post-narrative/non-narrative nature of cinema: the theory of aesthetic adaptation

Screen Space Reconfigured

Screen Space Reconfigured is the first edited volume that critically and theoretically examines the many novel renderings of space brought to us by 21st century screens. Exploring key cases such as post-perspectival space, 3D, vertical framing, haptics, and layering, this volume takes stock of emerging forms of screen space and spatialities as they move from the margins to the centre of contemporary media practice.Recent years have seen a marked scholarly interest in spatial dimensions and conceptions of moving image culture, with some theorists claiming that a 'spatial turn' has taken place in media studies and screen practices alike. Yet this is the first book-length study dedicated to on-screen spatiality as such.Spanning mainstream cinema, experimental film, video art, mobile screens, and stadium entertainment, the volume includes contributions from such acclaimed authors as Giuliana Bruno and Tom Gunning as well as a younger generation of scholars

Colour videos with depth : acquisition, processing and evaluation

The human visual system lets us perceive the world around us in three dimensions by integrating evidence from depth cues into a coherent visual model of the world. The equivalent in computer vision and computer graphics are geometric models, which provide a wealth of information about represented objects, such as depth and surface normals. Videos do not contain this information, but only provide per-pixel colour information. In this dissertation, I hence investigate a combination of videos and geometric models: videos with per-pixel depth (also known as RGBZ videos). I consider the full life cycle of these videos: from their acquisition, via filtering and processing, to stereoscopic display. I propose two approaches to capture videos with depth. The first is a spatiotemporal stereo matching approach based on the dual-cross-bilateral grid – a novel real-time technique derived by accelerating a reformulation of an existing stereo matching approach. This is the basis for an extension which incorporates temporal evidence in real time, resulting in increased temporal coherence of disparity maps – particularly in the presence of image noise. The second acquisition approach is a sensor fusion system which combines data from a noisy, low-resolution time-of-flight camera and a high-resolution colour video camera into a coherent, noise-free video with depth. The system consists of a three-step pipeline that aligns the video streams, efficiently removes and fills invalid and noisy geometry, and finally uses a spatiotemporal filter to increase the spatial resolution of the depth data and strongly reduce depth measurement noise. I show that these videos with depth empower a range of video processing effects that are not achievable using colour video alone. These effects critically rely on the geometric information, like a proposed video relighting technique which requires high-quality surface normals to produce plausible results. In addition, I demonstrate enhanced non-photorealistic rendering techniques and the ability to synthesise stereoscopic videos, which allows these effects to be applied stereoscopically. These stereoscopic renderings inspired me to study stereoscopic viewing discomfort. The result of this is a surprisingly simple computational model that predicts the visual comfort of stereoscopic images. I validated this model using a perceptual study, which showed that it correlates strongly with human comfort ratings. This makes it ideal for automatic comfort assessment, without the need for costly and lengthy perceptual studies