1,008 research outputs found
Game theory of mind
This paper introduces a model of ‘theory of mind’, namely, how we represent the intentions and goals of others to optimise our mutual interactions. We draw on ideas from optimum control and game theory to provide a ‘game theory of mind’. First, we consider the representations of goals in terms of value functions that are prescribed by utility or rewards. Critically, the joint value functions and ensuing behaviour are optimised recursively, under the assumption that I represent your value function, your representation of mine, your representation of my representation of yours, and so on ad infinitum. However, if we assume that the degree of recursion is bounded, then players need to estimate the opponent's degree of recursion (i.e., sophistication) to respond optimally. This induces a problem of inferring the opponent's sophistication, given behavioural exchanges. We show it is possible to deduce whether players make inferences about each other and quantify their sophistication on the basis of choices in sequential games. This rests on comparing generative models of choices with, and without, inference. Model comparison is demonstrated using simulated and real data from a ‘stag-hunt’. Finally, we note that exactly the same sophisticated behaviour can be achieved by optimising the utility function itself (through prosocial utility), producing unsophisticated but apparently altruistic agents. This may be relevant ethologically in hierarchal game theory and coevolution
A survey on policy search algorithms for learning robot controllers in a handful of trials
Most policy search algorithms require thousands of training episodes to find
an effective policy, which is often infeasible with a physical robot. This
survey article focuses on the extreme other end of the spectrum: how can a
robot adapt with only a handful of trials (a dozen) and a few minutes? By
analogy with the word "big-data", we refer to this challenge as "micro-data
reinforcement learning". We show that a first strategy is to leverage prior
knowledge on the policy structure (e.g., dynamic movement primitives), on the
policy parameters (e.g., demonstrations), or on the dynamics (e.g.,
simulators). A second strategy is to create data-driven surrogate models of the
expected reward (e.g., Bayesian optimization) or the dynamical model (e.g.,
model-based policy search), so that the policy optimizer queries the model
instead of the real system. Overall, all successful micro-data algorithms
combine these two strategies by varying the kind of model and prior knowledge.
The current scientific challenges essentially revolve around scaling up to
complex robots (e.g., humanoids), designing generic priors, and optimizing the
computing time.Comment: 21 pages, 3 figures, 4 algorithms, accepted at IEEE Transactions on
Robotic
Backwards is the way forward: feedback in the cortical hierarchy predicts the expected future
Clark offers a powerful description of the brain as a prediction machine, which offers progress on two distinct levels. First, on an abstract conceptual level, it provides a unifying framework for perception, action, and cognition (including subdivisions such as attention, expectation, and imagination). Second, hierarchical prediction offers progress on a concrete descriptive level for testing and constraining conceptual elements and mechanisms of predictive coding models (estimation of predictions, prediction errors, and internal models)
Neural Categorical Priors for Physics-Based Character Control
Recent advances in learning reusable motion priors have demonstrated their
effectiveness in generating naturalistic behaviors. In this paper, we propose a
new learning framework in this paradigm for controlling physics-based
characters with significantly improved motion quality and diversity over
existing state-of-the-art methods. The proposed method uses reinforcement
learning (RL) to initially track and imitate life-like movements from
unstructured motion clips using the discrete information bottleneck, as adopted
in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure
compresses the most relevant information from the motion clips into a compact
yet informative latent space, i.e., a discrete space over vector quantized
codes. By sampling codes in the space from a trained categorical prior
distribution, high-quality life-like behaviors can be generated, similar to the
usage of VQ-VAE in computer vision. Although this prior distribution can be
trained with the supervision of the encoder's output, it follows the original
motion clip distribution in the dataset and could lead to imbalanced behaviors
in our setting. To address the issue, we further propose a technique named
prior shifting to adjust the prior distribution using curiosity-driven RL. The
outcome distribution is demonstrated to offer sufficient behavioral diversity
and significantly facilitates upper-level policy learning for downstream tasks.
We conduct comprehensive experiments using humanoid characters on two
challenging downstream tasks, sword-shield striking and two-player boxing game.
Our results demonstrate that the proposed framework is capable of controlling
the character to perform considerably high-quality movements in terms of
behavioral strategies, diversity, and realism. Videos, codes, and data are
available at https://tencent-roboticsx.github.io/NCP/
Lifelike Agility and Play on Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
Summarizing knowledge from animals and human beings inspires robotic
innovations. In this work, we propose a framework for driving legged robots act
like real animals with lifelike agility and strategy in complex environments.
Inspired by large pre-trained models witnessed with impressive performance in
language and image understanding, we introduce the power of advanced deep
generative models to produce motor control signals stimulating legged robots to
act like real animals. Unlike conventional controllers and end-to-end RL
methods that are task-specific, we propose to pre-train generative models over
animal motion datasets to preserve expressive knowledge of animal behavior. The
pre-trained model holds sufficient primitive-level knowledge yet is
environment-agnostic. It is then reused for a successive stage of learning to
align with the environments by traversing a number of challenging obstacles
that are rarely considered in previous approaches, including creeping through
narrow spaces, jumping over hurdles, freerunning over scattered blocks, etc.
Finally, a task-specific controller is trained to solve complex downstream
tasks by reusing the knowledge from previous stages. Enriching the knowledge
regarding each stage does not affect the usage of other levels of knowledge.
This flexible framework offers the possibility of continual knowledge
accumulation at different levels. We successfully apply the trained multi-level
controllers to the MAX robot, a quadrupedal robot developed in-house, to mimic
animals, traverse complex obstacles, and play in a designed challenging
multi-agent Chase Tag Game, where lifelike agility and strategy emerge on the
robots. The present research pushes the frontier of robot control with new
insights on reusing multi-level pre-trained knowledge and solving highly
complex downstream tasks in the real world
- …