18 research outputs found
Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression
We design a new approach that allows robot learning of new activities from
unlabeled human example videos. Given videos of humans executing the same
activity from a human's viewpoint (i.e., first-person videos), our objective is
to make the robot learn the temporal structure of the activity as its future
regression network, and learn to transfer such model for its own motor
execution. We present a new deep learning model: We extend the state-of-the-art
convolutional object detection network for the representation/estimation of
human hands in training videos, and newly introduce the concept of using a
fully convolutional network to regress (i.e., predict) the intermediate scene
representation corresponding to the future frame (e.g., 1-2 seconds later).
Combining these allows direct prediction of future locations of human hands and
objects, which enables the robot to infer the motor control plan using our
manipulation network. We experimentally confirm that our approach makes
learning of robot activities from unlabeled human interaction videos possible,
and demonstrate that our robot is able to execute the learned collaborative
activities in real-time directly based on its camera input
One-Shot Observation Learning
Observation learning is the process of learning a task by observing an expert
demonstrator. We present a robust observation learning method for robotic
systems. Our principle contributions are in introducing a one shot learning
method where only a single demonstration is needed for learning and in
proposing a novel feature extraction method for extracting unique activity
features from the demonstration. Reward values are then generated from these
demonstrations. We use a learning algorithm with these rewards to learn the
controls for a robotic manipulator to perform the demonstrated task. With
simulation and real robot experiments, we show that the proposed method can be
used to learn tasks from a single demonstration under varying conditions of
viewpoints, object properties, morphology of manipulators and scene
backgrounds