Safe Opponent Exploitation For Epsilon Equilibrium Strategies

In safe opponent exploitation players hope to exploit their opponents' potentially sub-optimal strategies while guaranteeing at least the value of the game in expectation for themselves. Safe opponent exploitation algorithms have been successfully applied to small instances of two-player zero-sum imperfect information games, where Nash equilibrium strategies are typically known in advance. Current methods available to compute these strategies are however not scalable to desirable large domains of imperfect information such as No-Limit Texas Hold 'em (NLHE) poker, where successful agents rely on game abstractions in order to compute an equilibrium strategy approximation. This paper will extend the concept of safe opponent exploitation by introducing prime-safe opponent exploitation, in which we redefine the value of the game of a player to be the worst-case payoff their strategy could be susceptible to. This allows weaker epsilon equilibrium strategies to benefit from utilising a form of opponent exploitation with our revised value of the game, still allowing for a practical game-theoretical guaranteed lower-bound. We demonstrate the empirical advantages of our generalisation when applied to the main safe opponent exploitation algorithms

Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence

An important challenge for safety in machine learning and artificial intelligence systems is a~set of related failures involving specification gaming, reward hacking, fragility to distributional shifts, and Goodhart's or Campbell's law. This paper presents additional failure modes for interactions within multi-agent systems that are closely related. These multi-agent failure modes are more complex, more problematic, and less well understood than the single-agent case, and are also already occurring, largely unnoticed. After motivating the discussion with examples from poker-playing artificial intelligence (AI), the paper explains why these failure modes are in some senses unavoidable. Following this, the paper categorizes failure modes, provides definitions, and cites examples for each of the modes: accidental steering, coordination failures, adversarial misalignment, input spoofing and filtering, and goal co-option or direct hacking. The paper then discusses how extant literature on multi-agent AI fails to address these failure modes, and identifies work which may be useful for the mitigation of these failure modes.Comment: 12 Pages, This version re-submitted to Big Data and Cognitive Computing, Special Issue "Artificial Superintelligence: Coordination & Strategy

Learning about Learning in Games through Experimental Control of Strategic Interdependence

We conduct experiments in which humans repeatedly play one of two games against a computer decision maker that follows either Roth and Erev's reinforcement learning algorithm or Camerer and Ho's EWA algorithm. The human/algorithm interaction provides results that can't be obtained from the analysis of pure human interactions or model simulations. The learning algorithms are more sensitive than humans in calculating exploitable opponent play. Learning algorithms respond to these calculated opportunities systematically; however, the magnitude of these responses are too weak to improve the algorithm's payoffs. Human play against various decision maker types does not significantly vary. These results demonstrate that humans and currently proposed models of their behavior differ in that humans do not adjust payoff assessments by smooth transition functions and that when humans detect exploitable play they are more likely to choose the best response to this belief.

Social dilemmas, time preferences and technology adoption in a commons problem

Agents interacting on a body of water choose between technologies to catch fish. One is harmless to the resource, as it allows full recovery; the other yields high immediate catches, but low(er) future catches. Strategic interaction in one 'objective'resource game may induce several 'subjective' games in the class of social dilemmas. Which unique 'subjective'game is actually played depends crucially on how the agents discount their future payo¤s. We examine equilibrium behavior and its consequences on sustainability of the common-pool resource system under exponential and hyperbolic discounting. A sufficient degree of patience on behalf of the agents may lead to equilibrium behavior averting exhaustion of the resource, though full restraint (both agents choosing the ecologically or environmentally sound technology) is not necessarily achieved. Furthermore, if the degree of patience between agents is sufficiently dissimilar, the more patient is exploited by the less patient one in equilibrium. We demonstrate the generalizability of our approach developed throughout the paper. We provide recommendations to reduce the enormous complexity surrounding the general cases