Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior

Creating believable motions for various characters has long been a goal in computer graphics. Current learning-based motion synthesis methods depend on extensive motion datasets, which are often challenging, if not impossible, to obtain. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we utilize this alternative data source and introduce a neural motion synthesis approach through retargeting. Our method generates plausible motions for characters that have only pose data by transferring motion from an existing motion capture dataset of another character, which can have drastically different skeletons. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist-created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Project page: https://cyanzhao42.github.io/pose2motionComment: Project page: https://cyanzhao42.github.io/pose2motio

Multicontact Motion Retargeting Using Whole-Body Optimization of Full Kinematics and Sequential Force Equilibrium

This article presents a multicontact motion adaptation framework that enables teleoperation of high degree-of-freedom robots, such as quadrupeds and humanoids, for loco-manipulation tasks in multicontact settings. Our proposed algorithms optimize whole-body configurations and formulate the retargeting of multicontact motions as sequential quadratic programming, which is robust and stable near the edges of feasibility constraints. Our framework allows real-time operation of the robot and reduces cognitive load for the operator because infeasible commands are automatically adapted into physically stable and viable motions on the robot. The results in simulations with full dynamics demonstrated the effectiveness of teleoperating different legged robots interactively and generating rich multicontact movements. We evaluated the computational efficiency of the proposed algorithms, and further validated and analyzed multicontact loco-manipulation tasks on humanoid and quadruped robots by reaching, active pushing, and various traversal on uneven terrains

심층 강화학습을 이용한 사람의 모션을 통한 이형적 캐릭터 제어기 개발

학위논문(석사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2022. 8. 서진욱.사람의 모션을 이용한 로봇 컨트롤 인터페이스는 사용자의 직관과 로봇의 모터 능력을 합하여 위험한 환경에서 로봇의 유연한 작동을 만들어낸다. 하지만, 휴머노이드 외의 사족보행 로봇이나 육족보행 로봇을 위한 모션 인터페이스를 디자인 하는 것은 쉬운일이 아니다. 이것은 사람과 로봇 사이의 형태 차이로 오는 다이나믹스 차이와 제어 전략이 크게 차이나기 때문이다. 우리는 사람 사용자가 움직임을 통하여 사족보행 로봇에서 부드럽게 여러 과제를 수행할 수 있게끔 하는 새로운 모션 제어 시스템을 제안한다. 우리는 우선 캡쳐한 사람의 모션을 상응하는 로봇의 모션으로 리타겟 시킨다. 이때 상응하는 로봇의 모션은 유저가 의도한 의미를 내포하게 되며, 우리는 이를 지도학습 방법과 후처리 기술을 이용하여 가능케 하였다. 그 뒤 우리는 모션을 모사하는 학습을 커리큘럼 학습과 병행하여 주어진 리타겟된 참조 모션을 따라가는 제어 정책을 생성하였다. 우리는 "전문가 집단"을 학습함으로 모션 리타게팅 모듈과 모션 모사 모듈의 성능을 크게 증가시켰다. 결과에서 볼 수 있듯, 우리의 시스템을 이용하여 사용자가 사족보행 로봇의 서있기, 앉기, 기울이기, 팔 뻗기, 걷기, 돌기와 같은 다양한 모터 과제들을 시뮬레이션 환경과 현실에서 둘 다 수행할 수 있었다. 우리는 연구의 성능을 평가하기 위하여 다양한 분석을 하였으며, 특히 우리 시스템의 각각의 요소들의 중요성을 보여줄 수 있는 실험들을 진행하였다.A human motion-based interface fuses operator intuitions with the motor capabilities of robots, enabling adaptable robot operations in dangerous environments. However, the challenge of designing a motion interface for non-humanoid robots, such as quadrupeds or hexapods, is emerged from the different morphology and dynamics of a human controller, leading to an ambiguity of control strategy. We propose a novel control framework that allows human operators to execute various motor skills on a quadrupedal robot by their motion. Our system first retargets the captured human motion into the corresponding robot motion with the operator's intended semantics. The supervised learning and post-processing techniques allow this retargeting skill which is ambiguity-free and suitable for control policy training. To enable a robot to track a given retargeted motion, we then obtain the control policy from reinforcement learning that imitates the given reference motion with designed curriculums. We additionally enhance the system's performance by introducing a set of experts. Finally, we randomize the domain parameters to adapt the physically simulated motor skills to real-world tasks. We demonstrate that a human operator can perform various motor tasks using our system including standing, tilting, manipulating, sitting, walking, and steering on both physically simulated and real quadruped robots. We also analyze the performance of each system component ablation study.1 Introduction 1 2 Related Work 5 2.1 Legged Robot Control 5 2.2 Motion Imitation 6 2.3 Motion-based Control 7 3 Overview 9 4 Motion Retargeting Module 11 4.1 Motion Retargeting Network 12 4.2 Post-processing for Consistency 14 4.3 A Set of Experts for Multi-task Support 15 5 Motion Imitation Module 17 5.1 Background: Reinforcement Learning 18 5.2 Formulation of Motion Imitation 18 5.3 Curriculum Learning over Tasks and Difficulties 21 5.4 Hierarchical Control with States 21 5.5 Domain Randomization 22 6 Results and Analysis 23 6.1 Experimental Setup 23 6.2 Motion Performance 24 6.3 Analysis 28 6.4 Comparison to Other Methods 31 7 Conclusion And Future Work 32 Bibliography 34 Abstract (In Korean) 44 감사의 글 45석

Sketch-based skeleton-driven 2D animation and motion capture.

This research is concerned with the development of a set of novel sketch-based skeleton-driven 2D animation techniques, which allow the user to produce realistic 2D character animation efficiently. The technique consists of three parts: sketch-based skeleton-driven 2D animation production, 2D motion capture and a cartoon animation filter. For 2D animation production, the traditional way is drawing the key-frames by experienced animators manually. It is a laborious and time-consuming process. With the proposed techniques, the user only inputs one image ofa character and sketches a skeleton for each subsequent key-frame. The system then deforms the character according to the sketches and produces animation automatically. To perform 2D shape deformation, a variable-length needle model is developed, which divides the deformation into two stages: skeleton driven deformation and nonlinear deformation in joint areas. This approach preserves the local geometric features and global area during animation. Compared with existing 2D shape deformation algorithms, it reduces the computation complexity while still yielding plausible deformation results. To capture the motion of a character from exiting 2D image sequences, a 2D motion capture technique is presented. Since this technique is skeleton-driven, the motion of a 2D character is captured by tracking the joint positions. Using both geometric and visual features, this problem can be solved by ptimization, which prevents self-occlusion and feature disappearance. After tracking, the motion data are retargeted to a new character using the deformation algorithm proposed in the first part. This facilitates the reuse of the characteristics of motion contained in existing moving images, making the process of cartoon generation easy for artists and novices alike. Subsequent to the 2D animation production and motion capture,"Cartoon Animation Filter" is implemented and applied. Following the animation principles, this filter processes two types of cartoon input: a single frame of a cartoon character and motion capture data from an image sequence. It adds anticipation and follow-through to the motion with related squash and stretch effect

Collaborative Bimanual Manipulation Using Optimal Motion Adaptation and Interaction Control Retargetting Human Commands to Feasible Robot Control References

This article presents a robust and reliable human–robot collaboration (HRC) framework for bimanual manipulation. We propose an optimal motion adaptation method to retarget arbitrary human commands to feasible robot pose references while maintaining payload stability. The framework comprises three modules: 1) a task-space sequential equilibrium and inverse kinematics optimization ( task-space SEIKO ) for retargeting human commands and enforcing feasibility constraints, 2) an admittance controller to facilitate compliant human–robot physical interactions, and 3) a low-level controller improving stability during physical interactions. Experimental results show that the proposed framework successfully adapted infeasible and dangerous human commands into continuous motions within safe boundaries and achieved stable grasping and maneuvering of large and heavy objects on a real dual-arm robot via teleoperation and physical interaction. Furthermore, the framework demonstrated the capability in the assembly task of building blocks and the insertion task of industrial power connectors

A simple footskate removal method for virtual reality applications

Footskate is a common problem encountered in interactive applications dealing with virtual character animations. It has proven difficult to fix without the use of complex numerical methods, which require expert skills for their implementations, along with a fair amount of user interaction to correct a motion. On the other hand, deformable bodies are being increasingly used in virtual reality (VR) applications, allowing users to customize their avatar as they wish. This introduces the need of adapting motions without any help from a designer, as a random user seldom has the skills required to drive the existing algorithms towards the right solution. In this paper, we present a simple method to remove footskate artifacts in VR applications. Unlike previous algorithms, our approach does not rely on the skeletal animation to perform the correction but rather on the skin. This ensures that the final foot planting really matches the virtual character's motion. The changes are applied to the root joint of the skeleton only so that the resulting animation is as close as possible to the original one. Eventually, thanks to the simplicity of its formulation, it can be quickly and easily added to existing framework