1,662 research outputs found
Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems
Recently, self-learning methods based on user satisfaction metrics and
contextual bandits have shown promising results to enable consistent
improvements in conversational AI systems. However, directly targeting such
metrics by off-policy bandit learning objectives often increases the risk of
making abrupt policy changes that break the current user experience. In this
study, we introduce a scalable framework for supporting fine-grained
exploration targets for individual domains via user-defined constraints. For
example, we may want to ensure fewer policy deviations in business-critical
domains such as shopping, while allocating more exploration budget to domains
such as music. Furthermore, we present a novel meta-gradient learning approach
that is scalable and practical to address this problem. The proposed method
adjusts constraint violation penalty terms adaptively through a meta objective
that encourages balanced constraint satisfaction across domains. We conduct
extensive experiments using data from a real-world conversational AI on a set
of realistic constraint benchmarks. Based on the experimental results, we
demonstrate that the proposed approach is capable of achieving the best balance
between the policy value and constraint satisfaction rate
- …