Human motion synthesis is a long-standing problem with various applications
in digital twins and the Metaverse. However, modern deep learning based motion
synthesis approaches barely consider the physical plausibility of synthesized
motions and consequently they usually produce unrealistic human motions. In
order to solve this problem, we propose a system ``Skeleton2Humanoid'' which
performs physics-oriented motion correction at test time by regularizing
synthesized skeleton motions in a physics simulator. Concretely, our system
consists of three sequential stages: (I) test time motion synthesis network
adaptation, (II) skeleton to humanoid matching and (III) motion imitation based
on reinforcement learning (RL). Stage I introduces a test time adaptation
strategy, which improves the physical plausibility of synthesized human
skeleton motions by optimizing skeleton joint locations. Stage II performs an
analytical inverse kinematics strategy, which converts the optimized human
skeleton motions to humanoid robot motions in a physics simulator, then the
converted humanoid robot motions can be served as reference motions for the RL
policy to imitate. Stage III introduces a curriculum residual force control
policy, which drives the humanoid robot to mimic complex converted reference
motions in accordance with the physical law. We verify our system on a typical
human motion synthesis task, motion-in-betweening. Experiments on the
challenging LaFAN1 dataset show our system can outperform prior methods
significantly in terms of both physical plausibility and accuracy. Code will be
released for research purposes at:
https://github.com/michaelliyunhao/Skeleton2HumanoidComment: Accepted by ACMMM202