Artificial agents have traditionally been trained to maximize reward, which
may incentivize power-seeking and deception, analogous to how next-token
prediction in language models (LMs) may incentivize toxicity. So do agents
naturally learn to be Machiavellian? And how do we measure these behaviors in
general-purpose models such as GPT-4? Towards answering these questions, we
introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games
containing over half a million rich, diverse scenarios that center on social
decision-making. Scenario labeling is automated with LMs, which are more
performant than human annotators. We mathematize dozens of harmful behaviors
and use our annotations to evaluate agents' tendencies to be power-seeking,
cause disutility, and commit ethical violations. We observe some tension
between maximizing reward and behaving ethically. To improve this trade-off, we
investigate LM-based methods to steer agents' towards less harmful behaviors.
Our results show that agents can both act competently and morally, so concrete
progress can currently be made in machine ethics--designing agents that are
Pareto improvements in both safety and capabilities.Comment: ICML 2023 Oral; 31 pages, 5 figure