Natural User Interfaces for Virtual Character Full Body and Facial Animation in Immersive Virtual Worlds

In recent years, networked virtual environments have steadily grown to become a frontier in social computing. Such virtual cyberspaces are usually accessed by multiple users through their 3D avatars. Recent scientific activity has resulted in the release of both hardware and software components that enable users at home to interact with their virtual persona through natural body and facial activity performance. Based on 3D computer graphics methods and vision-based motion tracking algorithms, these techniques aspire to reinforce the sense of autonomy and telepresence within the virtual world. In this paper we present two distinct frameworks for avatar animation through user natural motion input. We specifically target the full body avatar control case using a Kinect sensor via a simple, networked skeletal joint retargeting pipeline, as well as an intuitive user facial animation 3D reconstruction pipeline for rendering highly realistic user facial puppets. Furthermore, we present a common networked architecture to enable multiple remote clients to capture and render any number of 3D animated characters within a shared virtual environment

Computational design of steady 3D dissection puzzles

Dissection puzzles require assembling a common set of pieces into multiple distinct forms. Existing works focus on creating 2D dissection puzzles that form primitive or naturalistic shapes. Unlike 2D dissection puzzles that could be supported on a tabletop surface, 3D dissection puzzles are preferable to be steady by themselves for each assembly form. In this work, we aim at computationally designing steady 3D dissection puzzles. We address this challenging problem with three key contributions. First, we take two voxelized shapes as inputs and dissect them into a common set of puzzle pieces, during which we allow slightly modifying the input shapes, preferably on their internal volume, to preserve the external appearance. Second, we formulate a formal model of generalized interlocking for connecting pieces into a steady assembly using both their geometric arrangements and friction. Third, we modify the geometry of each dissected puzzle piece based on the formal model such that each assembly form is steady accordingly. We demonstrate the effectiveness of our approach on a wide variety of shapes, compare it with the state-of-the-art on 2D and 3D examples, and fabricate some of our designed puzzles to validate their steadiness

Defining a Novel Meaning of the New Organic Architecture

Starting an overall investigation by categorizing current bio-inspired architectural design developments into “Material”, “Morphological”, and “Behavioral” to explore a novel definition of the “New Generation Organic Architecture”. At present, people are confronting the unprecedented unification of machine and biology which has been revealed by the means of advancing industrial processes towards the organic model. In his remarkable publication, “Out of Control: The New Biology of Machines, Social Systems, and the Economic World” (Kelly, 1995), Kevin Kelly makes an interesting observation that “Machines are becoming biological and the biological is becoming engineered”. In other words, the clear boundary of machine vs biology is blurring through current technological developments. In “Out of Control”, Kevin Kelly has further made several explicit points to support his views, that Industry will inevitably adopt bio-inspired methods: It takes less material to do the same job better. The complexity of built things now reaches biological complexity. Nature will not move, so it must be accommodated. The natural world itself—genes and life forms—can be engineered (and patented) just like industrial systems. All the crucial points described above can be easily observed in the architectural industry as well. Each statement corresponds with material optimization, multidisciplinary technologies, evolutionary processes, and genetic engineering which are all involved in current digital architectural design developments. After years of evolution, the developments of “Organic Architecture” have been now separated into various research focuses which are distant from the original idea coined by the well-known American architect, Frank Lloyd Wright. A group of followers still insist on maintaining Wright’s original idea to develop buildings which are green and sustainable, they fit or even blend into the surrounding environment as a whole. But since the power of personal computers and sophisticated modeling software has become relatively easy to access and is employed in all aspects of architectural design, various experiments have been conducted in the last decade, which try to outline a number of new definitions pertaining to “what are the essential ideas/principles of ‘Organic Architecture’?”. Nature has undoubtedly always been the greatest inspiration for the manmade industry, technology, and architecture. This development has only escalated with the assistance from computational technology over the last few decades. The thesis will preview the pros and cons of current design developments under the big umbrella of digital organic/bio-inspired architecture. This discussion will be categorized into three major divisions: “Morphological”, “Material”, and “Behavioral” owing to the different focus of computational applications within each one of them

Object-oriented motion abstraction

Ankara : Department of Computer Engineering and Information Science and Institute of Engineering and Science, Bilkent Univ., 1993.Thesis (Master's) -- Bilkent University, 1993.Includes bibliographical references leaves 74-77Erkan, BilgeM.S

Animation of a speaking chatbot: The Deep Speaking Avatar project

This thesis is a mixture of scientific research related to pre-developed technologies for animating virtual avatars and the actual development of an own application for the purpose. The topic was restricted by compatibility with other components of a chatbot pipeline, available techniques for computer graphics, and the required user experience for a Deep Speaking Avatar application. Other components in the project’s pipeline were implemented by four other students as their own bachelor’s theses. Two other components of the pipeline are relevant to our application, one of which is face recognition guiding the rotation of the avatar’s head. The other component dependency was the output of the text-to-speech module that would time the lip movements of the avatar. Available techniques were first discovered for facial expression modeling and face-audio synchronization. Two of them were tested, but both proved to be too restricted to be animated and rotated in real-time. Both techniques were competent in their intended aspects but fulfilling all the requirements for the Deep Speaking Avatar required lowering the standards of the avatar’s realism and making an own implementation using general-purpose tools. A suitable amount of implementational freedom and technical abilities was offered by the popular 3D-rendering and development tool Unity. Another popular tool, Blender, was used to craft the 3D-mesh model that is animated and rendered in Unity. The main desired functionalities of the avatar were achieved successfully in the specific technical conditions used to test the implementation. The built avatar can turn its head approximately in the direction of the user’s face, and the lips of the avatar move at a monotonic pace when the avatar produces speech. Relative locations of the camera, user, and screen should not differ much from the used test setup. However, ideas to improve the application’s adaptivity to different physical setups were discussed. A Linux environment with the other Avatar-pipeline components is required to run the Deep Speaking Avatar, and an integration script, written by another student can be used for setting up the pipeline. A good amount of computational power is required to run the whole system because many of the chatbot modules utilize heavy neural networks

AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control

Neural implicit fields are powerful for representing 3D scenes and generating high-quality novel views, but it remains challenging to use such implicit representations for creating a 3D human avatar with a specific identity and artistic style that can be easily animated. Our proposed method, AvatarCraft, addresses this challenge by using diffusion models to guide the learning of geometry and texture for a neural avatar based on a single text prompt. We carefully design the optimization framework of neural implicit fields, including a coarse-to-fine multi-bounding box training strategy, shape regularization, and diffusion-based constraints, to produce high-quality geometry and texture. Additionally, we make the human avatar animatable by deforming the neural implicit field with an explicit warping field that maps the target human mesh to a template human mesh, both represented using parametric human models. This simplifies animation and reshaping of the generated avatar by controlling pose and shape parameters. Extensive experiments on various text descriptions show that AvatarCraft is effective and robust in creating human avatars and rendering novel views, poses, and shapes. Our project page is: https://avatar-craft.github.io/.Comment: ICCV 2023 Camera Read

KINE[SIS]TEM'17 From Nature to Architectural Matter

Kine[SiS]tem – From Kinesis + System. Kinesis is a non-linear movement or activity of an organism in response to a stimulus. A system is a set of interacting and interdependent agents forming a complex whole, delineated by its spatial and temporal boundaries, influenced by its environment. How can architectural systems moderate the external environment to enhance comfort conditions in a simple, sustainable and smart way? This is the starting question for the Kine[SiS]tem’17 – From Nature to Architectural Matter International Conference. For decades, architectural design was developed despite (and not with) the climate, based on mechanical heating and cooling. Today, the argument for net zero energy buildings needs very effective strategies to reduce energy requirements. The challenge ahead requires design processes that are built upon consolidated knowledge, make use of advanced technologies and are inspired by nature. These design processes should lead to responsive smart systems that deliver the best performance in each specific design scenario. To control solar radiation is one key factor in low-energy thermal comfort. Computational-controlled sensor-based kinetic surfaces are one of the possible answers to control solar energy in an effective way, within the scope of contradictory objectives throughout the year.FC

A tool for generating three dimensional animation on computers

Ankara : The Department of Graphic Design and the Institute of Fine Arts of Bilkent Univ. , 1991.Thesis (Master's) -- Bilkent University, 1991.Includes bibliographical references leaves 31-32.In this work, a three dimensional computer animation system has been designed to be employed in schools, for the training of art students on basic three dimensional animation techniques. Puppet Theater, as we have called the system, utilizes the flexibility and effectiveness of the low-end hardware, namely IBM PC™ computers supported with Targa 16™ graphics board and gives special emphasis to user friendliness. It Is basically a software to design three dimensional objects and choreograph the object data in the computer's memory, before rendering the resulting scenery with shading methods. The system is the result of reflecting the recent advances in the field of computer graphics and pushing the potentials of the existing platform. Software is Implemented in C language, thus the code is transportable. A custom designed object oriented windowing system called WODNTW is used as the user Interface. This open windowing system supports pull-down menus, interactive buttons, scalable windows and other popular user interface elements.Türün, Cemil ŞinasiM.S