59 research outputs found
Efficient Deep Reinforcement Learning via Adaptive Policy Transfer
Transfer Learning (TL) has shown great potential to accelerate Reinforcement
Learning (RL) by leveraging prior knowledge from past learned policies of
relevant tasks. Existing transfer approaches either explicitly computes the
similarity between tasks or select appropriate source policies to provide
guided explorations for the target task. However, how to directly optimize the
target policy by alternatively utilizing knowledge from appropriate source
policies without explicitly measuring the similarity is currently missing. In
this paper, we propose a novel Policy Transfer Framework (PTF) to accelerate RL
by taking advantage of this idea. Our framework learns when and which source
policy is the best to reuse for the target policy and when to terminate it by
modeling multi-policy transfer as the option learning problem. PTF can be
easily combined with existing deep RL approaches. Experimental results show it
significantly accelerates the learning process and surpasses state-of-the-art
policy transfer methods in terms of learning efficiency and final performance
in both discrete and continuous action spaces.Comment: Accepted by IJCAI'202
Transfer Reinforcement Learning Based Negotiating Agent Framework
While achieving tremendous success, there is still a major issue standing out in the domain of automated negotiation: it is inefficient for a negotiating agent to learn a strategy from scratch when being faced with an unknown opponent. Transfer learning can alleviate this problem by utilizing the knowledge of previously learned policies to accelerate the current task learning. This work presents a novel Transfer Learning based Negotiating Agent (TLNAgent) framework that allows a negotiating agent to transfer previous knowledge from source strategies optimized by deep reinforcement learning, to boost its performance in new tasks. TLNAgent comprises three key components: the negotiation module, the adaptation module and the transfer module. To be specific, the negotiation module is responsible for interacting with the other agent during negotiation. The adaptation module measures the helpfulness of each source policy based on a fusion of two selection mechanisms. The transfer module is based on lateral connections between source and target networks and accelerates the agent’s training by transferring knowledge from the selected source strategy. Our comprehensive experiments clearly demonstrate that TL is effective in the context of automated negotiation, and TLNAgent outperforms state-of-the-art Automated Negotiating Agents Competition (ANAC) negotiating agents in various domains
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
A lot of efforts have been devoted to investigating how agents can learn
effectively and achieve coordination in multiagent systems. However, it is
still challenging in large-scale multiagent settings due to the complex
dynamics between the environment and agents and the explosion of state-action
space. In this paper, we design a novel Dynamic Multiagent Curriculum Learning
(DyMA-CL) to solve large-scale problems by starting from learning on a
multiagent scenario with a small size and progressively increasing the number
of agents. We propose three transfer mechanisms across curricula to accelerate
the learning process. Moreover, due to the fact that the state dimension varies
across curricula,, and existing network structures cannot be applied in such a
transfer setting since their network input sizes are fixed. Therefore, we
design a novel network structure called Dynamic Agent-number Network (DyAN) to
handle the dynamic size of the network input. Experimental results show that
DyMA-CL using DyAN greatly improves the performance of large-scale multiagent
learning compared with state-of-the-art deep reinforcement learning approaches.
We also investigate the influence of three transfer mechanisms across curricula
through extensive simulations.Comment: Accepted by AAAI202
Semi-Centralised Multi-Agent Reinforcement Learning with Policy-Embedded Training
Centralised training (CT) is the basis for many popular multi-agent
reinforcement learning (MARL) methods because it allows agents to quickly learn
high-performing policies. However, CT relies on agents learning from one-off
observations of other agents' actions at a given state. Because MARL agents
explore and update their policies during training, these observations often
provide poor predictions about other agents' behaviour and the expected return
for a given action. CT methods therefore suffer from high variance and
error-prone estimates, harming learning. CT methods also suffer from explosive
growth in complexity due to the reliance on global observations, unless strong
factorisation restrictions are imposed (e.g., monotonic reward functions for
QMIX). We address these challenges with a new semi-centralised MARL framework
that performs policy-embedded training and decentralised execution. Our method,
policy embedded reinforcement learning algorithm (PERLA), is an enhancement
tool for Actor-Critic MARL algorithms that leverages a novel parameter sharing
protocol and policy embedding method to maintain estimates that account for
other agents' behaviour. Our theory proves PERLA dramatically reduces the
variance in value estimates. Unlike various CT methods, PERLA, which seamlessly
adopts MARL algorithms, scales easily with the number of agents without the
need for restrictive factorisation assumptions. We demonstrate PERLA's superior
empirical performance and efficient scaling in benchmark environments including
StarCraft Micromanagement II and Multi-agent Mujoc
Synthetic engineering of a new biocatalyst encapsulating [NiFe]-hydrogenases for enhanced hydrogen production
Hydrogenases are microbial metalloenzymes capable of catalyzing the reversible interconversion between molecular hydrogen and protons with high efficiency, and have great potential in the development of new electrocatalysts for renewable...</jats:p
Reprogramming bacterial protein organelles as a nanoreactor for hydrogen production
Compartmentalization is a ubiquitous building principle in cells, which permits segregation of biological elements and reactions. The carboxysome is a specialized bacterial organelle that encapsulates enzymes into a virus-like protein shell and plays essential roles in photosynthetic carbon fixation. The naturally designed architecture, semi-permeability, and catalytic improvement of carboxysomes have inspired rational design and engineering of new nanomaterials to incorporate desired enzymes into the protein shell for enhanced catalytic performance. Here, we build large, intact carboxysome shells (over 90 nm in diameter) in the industrial microorganism Escherichia coli by expressing a set of carboxysome protein-encoding genes. We develop strategies for enzyme activation, shell self-assembly, and cargo encapsulation to construct a robust nanoreactor that incorporates catalytically active [FeFe]-hydrogenases and functional partners within the empty shell for the production of hydrogen. We show that shell encapsulation and the internal microenvironment of the new catalyst facilitate hydrogen production of the encapsulated oxygen-sensitive hydrogenases. The study provides insights into the assembly and formation of carboxysomes and paves the way for engineering carboxysome shell-based nanoreactors to recruit specific enzymes for diverse catalytic reactions
- …