71 research outputs found
LiveHand: Real-time and Photorealistic Neural Hand Rendering
The human hand is the main medium through which we interact with our
surroundings. Hence, its digitization is of uttermost importance, with direct
applications in VR/AR, gaming, and media production amongst other areas. While
there are several works for modeling the geometry and articulations of hands,
little attention has been dedicated to capturing photo-realistic appearance. In
addition, for applications in extended reality and gaming, real-time rendering
is critical. In this work, we present the first neural-implicit approach to
photo-realistically render hands in real-time. This is a challenging problem as
hands are textured and undergo strong articulations with various pose-dependent
effects. However, we show that this can be achieved through our carefully
designed method. This includes training on a low-resolution rendering of a
neural radiance field, together with a 3D-consistent super-resolution module
and mesh-guided space canonicalization and sampling. In addition, we show the
novel application of a perceptual loss on the image space is critical for
achieving photorealism. We show rendering results for several identities, and
demonstrate that our method captures pose- and view-dependent appearance
effects. We also show a live demo of our method where we photo-realistically
render the human hand in real-time for the first time in literature. We ablate
all our design choices and show that our design optimizes for both photorealism
and rendering speed. Our code will be released to encourage further research in
this area.Comment: 11 pages, 8 figure
- …