26 research outputs found
Goal-oriented Dialogue Policy Learning from Failures
Reinforcement learning methods have been used for learning dialogue policies.
However, learning an effective dialogue policy frequently requires
prohibitively many conversations. This is partly because of the sparse rewards
in dialogues, and the very few successful dialogues in early learning phase.
Hindsight experience replay (HER) enables learning from failures, but the
vanilla HER is inapplicable to dialogue learning due to the implicit goals. In
this work, we develop two complex HER methods providing different trade-offs
between complexity and performance, and, for the first time, enabled HER-based
dialogue policy learning. Experiments using a realistic user simulator show
that our HER methods perform better than existing experience replay methods (as
applied to deep Q-networks) in learning rate
Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
Applying reinforcement learning in physical-world tasks is extremely
challenging. It is commonly infeasible to sample a large number of trials, as
required by current reinforcement learning methods, in a physical environment.
This paper reports our project on using reinforcement learning for better
commodity search in Taobao, one of the largest online retail platforms and
meanwhile a physical environment with a high sampling cost. Instead of training
reinforcement learning in Taobao directly, we present our approach: first we
build Virtual Taobao, a simulator learned from historical customer behavior
data through the proposed GAN-SD (GAN for Simulating Distributions) and MAIL
(multi-agent adversarial imitation learning), and then we train policies in
Virtual Taobao with no physical costs in which ANC (Action Norm Constraint)
strategy is proposed to reduce over-fitting. In experiments, Virtual Taobao is
trained from hundreds of millions of customers' records, and its properties are
compared with the real environment. The results disclose that Virtual Taobao
faithfully recovers important properties of the real environment. We also show
that the policies trained in Virtual Taobao can have significantly superior
online performance to the traditional supervised approaches. We hope our work
could shed some light on reinforcement learning applications in complex
physical environments