130,373 research outputs found

    MoFaNeRF: Morphable Facial Neural Radiance Field

    Full text link
    We propose a parametric model that maps free-view images into a vector space of coded facial shape, expression and appearance with a neural radiance field, namely Morphable Facial NeRF. Specifically, MoFaNeRF takes the coded facial shape, expression and appearance along with space coordinate and view direction as input to an MLP, and outputs the radiance of the space point for photo-realistic image synthesis. Compared with conventional 3D morphable models (3DMM), MoFaNeRF shows superiority in directly synthesizing photo-realistic facial details even for eyes, mouths, and beards. Also, continuous face morphing can be easily achieved by interpolating the input shape, expression and appearance codes. By introducing identity-specific modulation and texture encoder, our model synthesizes accurate photometric details and shows strong representation ability. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view synthesis. Experiments show that our method achieves higher representation ability than previous parametric models, and achieves competitive performance in several applications. To the best of our knowledge, our work is the first facial parametric model built upon a neural radiance field that can be used in fitting, generation and manipulation. The code and data is available at https://github.com/zhuhao-nju/mofanerf.Comment: accepted to ECCV2022; code available at http://github.com/zhuhao-nju/mofaner

    Synthesis of Parametric Programs using Genetic Programming and Model Checking

    Get PDF
    Formal methods apply algorithms based on mathematical principles to enhance the reliability of systems. It would only be natural to try to progress from verification, model checking or testing a system against its formal specification into constructing it automatically. Classical algorithmic synthesis theory provides interesting algorithms but also alarming high complexity and undecidability results. The use of genetic programming, in combination with model checking and testing, provides a powerful heuristic to synthesize programs. The method is not completely automatic, as it is fine tuned by a user that sets up the specification and parameters. It also does not guarantee to always succeed and converge towards a solution that satisfies all the required properties. However, we applied it successfully on quite nontrivial examples and managed to find solutions to hard programming challenges, as well as to improve and to correct code. We describe here several versions of our method for synthesizing sequential and concurrent systems.Comment: In Proceedings INFINITY 2013, arXiv:1402.661

    HeadOn: Real-time Reenactment of Human Portrait Videos

    Get PDF
    We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.Comment: Video: https://www.youtube.com/watch?v=7Dg49wv2c_g Presented at Siggraph'1

    A Generic Synthesis Algorithm for Well-Defined Parametric Design

    Get PDF
    This paper aims to improve the way synthesis tools can be built by formalizing: 1) the design artefact, 2) related knowledge and 3) an algorithm to generate solutions. This paper focuses on well-defined parametric engineering design, ranging from machine elements to industrial products. A design artefact is formalized in terms of parameters and topology elements. The knowledge is classified in three types: resolving rules to determine parameter values, constraining rules to restrict parameter values and expansion rules to add elements to the topology. A synthesis algorithm, based on an opportunistic design strategy, is described and tested for three design cases

    Barry Truax Riverrun (1986/2004), a case study from the TaCEM project, exploring new approaches to techniques of analysis and re-synthesis in the study of concert electroacoustic works

    Get PDF
    At last year’s EMS in Lisbon we introduced the TaCEM project (Technology and Creativity in Electroacoustic Music), a 30-month project funded by the UK’s Arts and Humanities Research Council, and demonstrated the generic TIAALS software being produced as part of this project. This year we present an update on the project, focusing particularly on the first of our case studies, Barry Truax’s Riverrun. Eight works have been selected for the project, taking into account criteria such as historical context, the nature of the synthesis techniques employed, and the aesthetics that have underpinned their realisation. Key considerations have included the accessibility of the technical resources and composing materials used in their production, and opportunities to pursue particular lines of enquiry with the composer concerned. In selecting the eight works for detailed study, a further consideration has been the extent to which the composers explored techniques that were already available at the time in ways that are unique and distinctive, or alternatively developed entirely new methods of synthesis in pursuit of their creative goals. The pioneering work of Barry Truax in terms of developing techniques of granular synthesis assign his achievements almost exclusively to the latter classification, and the composition of Riverrun (1986/2004) is a landmark achievement in this regard. Truax’s composing environment evolved from the early study of interactive real-time synthesis techniques at the Institute of Sonology, Utrecht 1971-73, exploring the possibilities of using Poisson-ordered distributions in the generation of microsound, to the emergence of entirely granular techniques at Simon Fraser University, British Columbia a decade later, culminating in the development of his program GSX designed specifically for waveform-based synthesis and first used to compose Riverrun, and its later extension, GSAMX, that extended these granular techniques to include the manipulation of previously sampled sound material. At the time of composition conventional minicomputers still lacked the capacity to generate multiple voices of granulated sound material in real time, but for Truax the acquisition in 1982 of a high speed bit slice array processor, the DMX-1000, provided the enhancedprocessing power necessary for achieving such a goal. The unique characteristics of its special hardware and associated programming environment, managed in turn via a host PDP 11/23 computer, both empowered his creative objectives and also materially shaped and influenced the ways in which they could be practically achieved. The significance of such causal relationships in the evolution of the electroacoustic music repertory has yet to be widely understood, and this study of Riverrun corroborates the importance of such a line of investigation. In this case it has been possible to carry out a detailed study of the original system, still maintained in working order by Truax, leading to a reconstruction of key elements of Riverrun using a Max-based simulation of GSX, the authenticity of the results being assessed both subjectively by means of a direct aural comparison and also measured objectively using software. Our presentation at this year’s EMS in Berlin included a demonstration of examples of the software we have developed to enable readers to engage with Riverrun interactively, both by analysing the original recordings and by using our emulation of the GSX system to be able to recreate passages of the work and manipulate the techniques employed in order to learn more about them. We also gave examples of other materials we have collected in relation to this case study, including videos of the composer himself working with the GSX system and discussing the composition of Riverrun

    Mage - Reactive articulatory feature control of HMM-based parametric speech synthesis

    Get PDF
    In this paper, we present the integration of articulatory control into MAGE, a framework for realtime and interactive (reactive) parametric speech synthesis using hidden Markov models (HMMs). MAGE is based on the speech synthesis engine from HTS and uses acoustic features (spectrum and f0) to model and synthesize speech. In this work, we replace the standard acoustic models with models combining acoustic and articulatory features, such as tongue, lips and jaw positions. We then use feature-space-switched articulatory-to-acoustic regression matrices to enable us to control the spectral acoustic features by manipulating the articulatory features. Combining this synthesis model with MAGE allows us to interactively and intuitively modify phones synthesized in real time, for example transforming one phone into another, by controlling the configuration of the articulators in a visual display. Index Terms: speech synthesis, reactive, articulators 1
    • 

    corecore