7 research outputs found
A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis
The synthesis of human motion has traditionally been addressed through
task-dependent models that focus on specific challenges, such as predicting
future motions or filling in intermediate poses conditioned on known key-poses.
In this paper, we present a novel task-independent model called UNIMASK-M,
which can effectively address these challenges using a unified architecture.
Our model obtains comparable or better performance than the state-of-the-art in
each field. Inspired by Vision Transformers (ViTs), our UNIMASK-M model
decomposes a human pose into body parts to leverage the spatio-temporal
relationships existing in human motion. Moreover, we reformulate various
pose-conditioned motion synthesis tasks as a reconstruction problem with
different masking patterns given as input. By explicitly informing our model
about the masked joints, our UNIMASK-M becomes more robust to occlusions.
Experimental results show that our model successfully forecasts human motion on
the Human3.6M dataset. Moreover, it achieves state-of-the-art results in motion
inbetweening on the LaFAN1 dataset, particularly in long transition periods.
More information can be found on the project website
https://sites.google.com/view/estevevallsmascaro/publications/unimask-m
Unsupervised human-to-robot motion retargeting via expressive latent space
This paper introduces a novel approach for human-to-robot motion retargeting,
enabling robots to mimic human motion with precision while preserving the
semantics of the motion. For that, we propose a deep learning method for direct
translation from human to robot motion. Our method does not require annotated
paired human-to-robot motion data, which reduces the effort when adopting new
robots. To this end, we first propose a cross-domain similarity metric to
compare the poses from different domains (i.e., human and robot). Then, our
method achieves the construction of a shared latent space via contrastive
learning and decodes latent representations to robot motion control commands.
The learned latent space exhibits expressiveness as it captures the motions
precisely and allows direct motion control in the latent space. We showcase how
to generate in-between motion through simple linear interpolation in the latent
space between two projected human poses. Additionally, we conducted a
comprehensive evaluation of robot control using diverse modality inputs, such
as texts, RGB videos, and key-poses, which enhances the ease of robot control
to users of all backgrounds. Finally, we compare our model with existing works
and quantitatively and qualitatively demonstrate the effectiveness of our
approach, enhancing natural human-robot communication and fostering trust in
integrating robots into daily life
HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs
Robots are becoming increasingly integrated into our lives, assisting us in
various tasks. To ensure effective collaboration between humans and robots, it
is essential that they understand our intentions and anticipate our actions. In
this paper, we propose a Human-Object Interaction (HOI) anticipation framework
for collaborative robots. We propose an efficient and robust transformer-based
model to detect and anticipate HOIs from videos. This enhanced anticipation
empowers robots to proactively assist humans, resulting in more efficient and
intuitive collaborations. Our model outperforms state-of-the-art results in HOI
detection and anticipation in VidHOI dataset with an increase of 1.76% and
1.04% in mAP respectively while being 15.4 times faster. We showcase the
effectiveness of our approach through experimental results in a real robot,
demonstrating that the robot's ability to anticipate HOIs is key for better
Human-Robot Interaction. More information can be found on our project webpage:
https://evm7.github.io/HOI4ABOT_page/Comment: Proceedings in Conference on Robot Learning 202
Robust Human Motion Forecasting using Transformer-based Model
Comprehending human motion is a fundamental challenge for developing
Human-Robot Collaborative applications. Computer vision researchers have
addressed this field by only focusing on reducing error in predictions, but not
taking into account the requirements to facilitate its implementation in
robots. In this paper, we propose a new model based on Transformer that
simultaneously deals with the real time 3D human motion forecasting in the
short and long term. Our 2-Channel Transformer (2CH-TR) is able to efficiently
exploit the spatio-temporal information of a shortly observed sequence (400ms)
and generates a competitive accuracy against the current state-of-the-art.
2CH-TR stands out for the efficient performance of the Transformer, being
lighter and faster than its competitors. In addition, our model is tested in
conditions where the human motion is severely occluded, demonstrating its
robustness in reconstructing and predicting 3D human motion in a highly noisy
environment. Our experiment results show that the proposed 2CH-TR outperforms
the ST-Transformer, which is another state-of-the-art model based on the
Transformer, in terms of reconstruction and prediction under the same
conditions of input prefix. Our model reduces in 8.89% the mean squared error
of ST-Transformer in short-term prediction, and 2.57% in long-term prediction
in Human3.6M dataset with 400ms input prefix.Comment: This paper has been already accepted to the 2022 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS 2022
Can We Use Diffusion Probabilistic Models for 3D Motion Prediction?
After many researchers observed fruitfulness from the recent diffusion probabilistic model, its effectiveness in image generation is actively studied these days. In this paper, our objective is to evaluate the potential of diffusion probabilistic models for 3D human motion-related tasks. To this end, this pa-per presents a study of employing diffusion probabilistic models to predict future 3D human motion(s) from the previously observed motion. Based on the Human 3.6M and HumanEva-I datasets, our results show that diffusion probabilistic models are competitive for both single (deterministic) and multiple (stochastic) 3D motion prediction tasks, after finishing a single training process. In addition, we find out that diffusion probabilistic models can offer an attractive compromise, since they can strike the right balance between the likelihood and diversity of the predicted future motions. Our code is publicly available on the project website: https://sites.google.com/view/diffusion-motion-prediction
Intention-Conditioned Long-Term Human Egocentric Action Anticipation
To anticipate how a person would act in the future, it is essential to understand the human intention since it guides the subject towards a certain action. In this paper, we propose a hierarchical architecture which assumes a sequence of human action (low-level) can be driven from the human intention (high-level). Based on this, we deal with long-term action anticipation task in egocentric videos. Our framework first extracts this low- and high-level human information over the observed human actions in a video through a Hierarchical Multi-task Multi-Layer Perceptrons Mixer (H3M). Then, we constrain the uncertainty of the future through an Intention-Conditioned Variational Auto-Encoder (I-CVAE) that generates multiple stable predictions of the next actions that the observed human might perform. By leveraging human intention as high-level information, we claim that our model is able to anticipate more time-consistent actions in the long-term, thus improving the results over the baseline in Ego4D dataset. This work results in the state-of-the-art for Long-Term Anticipation (LTA) task in Ego4D by providing more plausible anticipated sequences, improving the anticipation scores of nouns and actions. Our work ranked first in both CVPR@2022 and ECCV@2022 Ego4D LTA Challenge
ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space
This paper introduces a novel deep-learning approach for human-to-robot motion retargeting, enabling robots to mimic human poses accurately. Contrary to prior deep-learning-based works, our method does not require paired human-to-robot data, which facilitates its translation to new robots. First, we construct a shared latent space between humans and robots via adaptive contrastive learning that takes advantage of a proposed cross-domain similarity metric between the human and robot poses. Additionally, we propose a consistency term to build a common latent space that captures the similarity of the poses with precision while allowing direct robot motion control from the latent space. For instance, we can generate in-between motion through simple linear interpolation between two projected human poses. We conduct a comprehensive evaluation of robot control from diverse modalities (i.e., texts, RGB videos, and key poses), which facilitates robot control for non-expert users. Our model outperforms existing works regarding human-to-robot retargeting in terms of efficiency and precision. Finally, we implemented our method in a real robot with self-collision avoidance through a whole-body controller to showcase the effectiveness of our approach