SLoMo: A General System for Legged Robot Motion Imitation from Casual Videos

We present SLoMo: a first-of-its-kind framework for transferring skilled motions from casually captured "in the wild" video footage of humans and animals to legged robots. SLoMo works in three stages: 1) synthesize a physically plausible reconstructed key-point trajectory from monocular videos; 2) optimize a dynamically feasible reference trajectory for the robot offline that includes body and foot motion, as well as contact sequences that closely tracks the key points; 3) track the reference trajectory online using a general-purpose model-predictive controller on robot hardware. Traditional motion imitation for legged motor skills often requires expert animators, collaborative demonstrations, and/or expensive motion capture equipment, all of which limits scalability. Instead, SLoMo only relies on easy-to-obtain monocular video footage, readily available in online repositories such as YouTube. It converts videos into motion primitives that can be executed reliably by real-world robots. We demonstrate our approach by transferring the motions of cats, dogs, and humans to example robots including a quadruped (on hardware) and a humanoid (in simulation). To the best knowledge of the authors, this is the first attempt at a general-purpose motion transfer framework that imitates animal and human motions on legged robots directly from casual videos without artificial markers or labels.Comment: accepted at RA-L 2023, with ICRA 2024 optio

Imitating individualized facial expressions in a human-like avatar through a hybrid particle swarm optimization - tabu search algorithm

This thesis describes a machine learning method for automatically imitating a particular person\u27s facial expressions in a human-like avatar through a hybrid Particle Swarm Optimization - Tabu Search algorithm. The muscular structures of the facial expressions are measured by Ekman and Friesen\u27s Facial Action Coding System (FACS). Using a neutral face as a reference, the minute movements of the Action Units, used in FACS, are automatically tracked and mapped onto the avatar using a hybrid method. The hybrid algorithm is composed of Kennedy and Eberhart\u27s Particle Swarm Optimization algorithm (PSO) and Glover\u27s Tabu Search (TS). Distinguishable features portrayed on the avatar ensure a personalized, realistic imitation of the facial expressions. To evaluate the feasibility of using PSO-TS in this approach, a fundamental proof-of-concept test is employed on the system using the OGRE avatar. This method is analyzed in-depth to ensure its proper functionality and evaluate its performance compared to previous work