Real-Time Stylized Rendering for Large-Scale 3D Scenes

While modern digital entertainment has seen a major shift toward photorealism in animation, there is still significant demand for stylized rendering tools. Stylized, or non-photorealistic rendering (NPR), applications generally sacrifice physical accuracy for artistic or functional visual output. Oftentimes, NPR applications focus on extracting specific features from a 3D environment and highlighting them in a unique manner. One application of interest involves recreating 2D hand-drawn art styles in a 3D-modeled environment. This task poses challenges in the form of spatial coherence, feature extraction, and stroke line rendering. Previous research on this topic has also struggled to overcome specific performance bottlenecks, which have limited use of this technology in real-time applications. Specifically, many stylized rendering techniques have difficulty operating on large-scale scenes, such as open-world terrain environments. In this paper, we describe various novel rendering techniques for mimicking hand-drawn art styles in a large-scale 3D environment, including modifications to existing methods for stroke rendering and hatch-line texturing. Our system focuses on providing various complex styles while maintaining real-time performance, to maximize user-interactability. Our results demonstrate improved performance over existing real-time methods, and offer a few unique style options for users, though the system still suffers from some visual inconsistencies

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.ioComment: Project page: https://textual-inversion.github.i

Computer Aided Design of Handwriting Trajectories with the Kinematic Theory of Rapid Human Movements

We present the Sigma Lognormal model as a potential tool for curve generation in computer graphics related applications. We discuss its extension and parameterisations for the interactive definition of handwriting, drawing and calligraphic trajectories. This results in an effcient trajectory synthesis method, that has a user interface similar to the ones commonly used with Bezier curves or splines, but with the added benefit of capturing the kinematics of human drawing or writing movements. Such kinematics produced by the model can then be exploited to generate realistic stroke animations or to facilitate expressive rendering methods

Revealing the Invisible: On the Extraction of Latent Information from Generalized Image Data

The desire to reveal the invisible in order to explain the world around us has been a source of impetus for technological and scientific progress throughout human history. Many of the phenomena that directly affect us cannot be sufficiently explained based on the observations using our primary senses alone. Often this is because their originating cause is either too small, too far away, or in other ways obstructed. To put it in other words: it is invisible to us. Without careful observation and experimentation, our models of the world remain inaccurate and research has to be conducted in order to improve our understanding of even the most basic effects. In this thesis, we1 are going to present our solutions to three challenging problems in visual computing, where a surprising amount of information is hidden in generalized image data and cannot easily be extracted by human observation or existing methods. We are able to extract the latent information using non-linear and discrete optimization methods based on physically motivated models and computer graphics methodology, such as ray tracing, real-time transient rendering, and image-based rendering

Gaming and Photography: Investigating the Elision of Illusion and the Actual

The application of aesthetics and techniques from photography into computer generated images leads audiences to read images of a virtual space similarly to images of a physical space. This phenomenon has allowed for a continuation in the cultural fascination with photorealism, and cases of audiences mistaking images from the virtual space as ‘real’. This thesis looks in detail at how the boundaries between the virtual and physical space shift when approaching the virtual space of a video game from the perspective of a photographer, rather than a player. It looks in detail at how audiences interpret images of the virtual space of video games when displayed in a form reminiscent of art photography. Photographs of the virtual and physical spaces were produced for online surveys and an exhibition to test audience perceptions of image origin. Participants were also asked to try and distinguish photorealism in the landscape form, urban form, and material form. Technical analysis of audience responses, combined with textual analysis of the images themselves, helped in determining the types of content, as well as styles of photography that were used by the audience as indexes to reality in the virtual space. In some cases, the technical theory could explain the thought process of the participants, however in other cases there were dominant factors that more significantly impacted participant interpretations, despite what theory suggested. This highlighted the blur that is emerging between the physical and virtual spaces. There were difficulties in designing tests that could identify and isolate the elements that influenced perceptions of photorealism, due to the complex, and sometimes unexpected, ways in which people made judgments about the images. A variety of factors and areas for future research arose from the tests, including using the medium of photography to document the ever-changing landscape of the virtual space

Pathway to Future Symbiotic Creativity

This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation. We propose a classification of the creative system with a hierarchy of 5 classes, showing the pathway of creativity evolving from a mimic-human artist (Turing Artists) to a Machine artist in its own right. We begin with an overview of the limitations of the Turing Artists then focus on the top two-level systems, Machine Artists, emphasizing machine-human communication in art creation. In art creation, it is necessary for machines to understand humans' mental states, including desires, appreciation, and emotions, humans also need to understand machines' creative capabilities and limitations. The rapid development of immersive environment and further evolution into the new concept of metaverse enable symbiotic art creation through unprecedented flexibility of bi-directional communication between artists and art manifestation environments. By examining the latest sensor and XR technologies, we illustrate the novel way for art data collection to constitute the base of a new form of human-machine bidirectional communication and understanding in art creation. Based on such communication and understanding mechanisms, we propose a novel framework for building future Machine artists, which comes with the philosophy that a human-compatible AI system should be based on the "human-in-the-loop" principle rather than the traditional "end-to-end" dogma. By proposing a new form of inverse reinforcement learning model, we outline the platform design of machine artists, demonstrate its functions and showcase some examples of technologies we have developed. We also provide a systematic exposition of the ecosystem for AI-based symbiotic art form and community with an economic model built on NFT technology. Ethical issues for the development of machine artists are also discussed

Using Precisionism Within American Modern Art as Stylistic Inspiration for 3D Digital Works

This thesis presents the analysis of artistic techniques of paintings from the Precisionist movement and the implementation of the results of the analysis in the creation of three new works of art using digital media. Artists working in digital media express features of pre-digital artistic movements with varying degrees of adherence to principles, intentions, and awareness. This thesis seeks to create a bridge between the recognition of common features of Precisionist works and the expression of those elements in new works through the use of a system of analysis, interpretation, and translation. One outcome of this thesis is the description of a methodology for interpretation and translation that can be applied to other art movements. The Precisionist period within the Modern Art movement has both a historical importance in the world of art and a thematic relevance to popular uses of digital media ? specifically the representation of meaning and mood derived from industrial settings. Its influences can be traced from cubist, futurist, and constructivist art, as well as influencing the development of surrealism. It is considered the first solely American movement within Modern Art. Charles Sheeler's work plays a key role in the visual analysis portion of this research. Sheeler's work offers examples for applying 2D precisionist artistic style as aesthetic inspiration in creating a three-part production of 3D digital and video work. Work from precisionist artists Charles Demuth and Edmund Lewandowski also contribute some unique artistic characteristics considered during the analytical portion of this study. The new artistic works proposed include: (1) a linear, live-action short video with post-production manipulation; (2) a linear, 3D animated work; and (3) a non-linear, interactive 3D game environment