Generating Behaviorally Diverse Policies with Latent Diffusion Models

Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/hom

Learning to Walk Autonomously via Reset-Free Quality-Diversity

Quality-Diversity (QD) algorithms can discover large and complex behavioural repertoires consisting of both diverse and high-performing skills. However, the generation of behavioural repertoires has mainly been limited to simulation environments instead of real-world learning. This is because existing QD algorithms need large numbers of evaluations as well as episodic resets, which require manual human supervision and interventions. This paper proposes Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous learning for robotics in open-ended environments. We build on Dynamics-Aware Quality-Diversity (DA-QD) and introduce a behaviour selection policy that leverages the diversity of the imagined repertoire and environmental information to intelligently select of behaviours that can act as automatic resets. We demonstrate this through a task of learning to walk within defined training zones with obstacles. Our experiments show that we can learn full repertoires of legged locomotion controllers autonomously without manual resets with high sample efficiency in spite of harsh safety constraints. Finally, using an ablation of different target objectives, we show that it is important for RF-QD to have diverse types solutions available for the behaviour selection policy over solutions optimised with a specific objective. Videos and code available at https://sites.google.com/view/rf-qd

What is Cultural Science? (And what it is not.)

Hartley and Potts (2014) argue that cultural science represents a new theoretical and methodological approach to the study of cultural structure, dynamics and use. We explain how this differs from the extant analytic frameworks of cultural studies, both as a research program and as a policy platform. The central idea is to reconceptualize what culture is, through a reinterpretation of what culture does. We argue that the semiotic productivity of culture makes groups – which we call demes – and demes make knowledge (what we call the externalism hypothesis); and the interaction of demes makes newness – new knowledge. Cultural science, then, is a new model of the cultural processes involved in socio-economic evolution and innovation of knowledge-making demes. The paper is in three sections, the first on the exhaustion of cultural studies; the second on the emergence of cultural science; and the third on some implications for cultural policy – illustrated by reference to Matthew Arnold’s policy on language preservation