Plecto: Investigating the Musical Affordances of Continuous Time Recurrent Neural Networks

"Plecto: Investigating the musical affordances of Continuous Time Recurrent Neural Networks" is a practice-based research project that investigates how continuous time recurrent neural networks (CTRNNs) can be applied to the problem of achieving gestural control in improvised electronic music. One of the challenges of improvising using computers is manipulating different compositional layers during a performance while maintaining granular and expressive control. Artists turn to concepts such as artificial life to solve this problem and pursue software agents with complex, responsive and organic qualities that lead to the perception of lifelikeness. Guided by this theme, I propose a design for a low frequency oscillator (LFO), called Plecto, for use within existing composition workflows that harnesses the idiosyncratic behaviours of CTRNNs as a gestural agent within improvised electronic music performances. CTRNNs have been used in studies of biological modelling such as animal locomotion, and also of minimally cognitive behaviours such as basic object perception. Their ability to produce lifelike abstract forms makes them well suited as a source of gestural control. Oliver Bown and Sebastian Lexer have applied CTRNNs to musical event generation, using evolutionary algorithms (EA) to search for different CTRNN behaviours. I have extended this approach, using a novelty search (NS) variant for the open-ended discovery of CTRNN configurations, each exhibiting novel behaviours that can be applied to different musical problems. Through a series of computational studies, I have explored the lifelike qualities of CTRNNs best suited for gestural control and a novelty search algorithm design for their discovery. An iterative design process was also undertaken, establishing clear design principles adopted to build a usable representation of the CTRNN algorithm within an LFO device built for the Ableton Live environment. Evaluation of the tool was conducted through a user survey and practice-based case studies that incorporate the device into my own improvised electronic music workflow as a gestural agent. The primary outcomes of this research are a suite of software that can be adopted by the broader community of practitioners and a series of compositions reflecting the impacts of the CTRNN algorithm on my creative process

Deep Visual Instruments: Realtime Continuous, Meaningful Human Control over Deep Neural Networks for Creative Expression

In this thesis, we investigate Deep Learning models as an artistic medium for new modes of performative, creative expression. We call these Deep Visual Instruments: realtime interactive generative systems that exploit and leverage the capabilities of state-of-the-art Deep Neural Networks (DNN), while allowing Meaningful Human Control, in a Realtime Continuous manner. We characterise Meaningful Human Control in terms of intent, predictability, and accountability; and Realtime Continuous Control with regards to its capacity for performative interaction with immediate feedback, enhancing goal-less exploration. The capabilities of DNNs that we are looking to exploit and leverage in this manner, are their ability to learn hierarchical representations modelling highly complex, real-world data such as images. Thinking of DNNs as tools that extract useful information from massive amounts of Big Data, we investigate ways in which we can navigate and explore what useful information a DNN has learnt, and how we can meaningfully use such a model in the production of artistic and creative works, in a performative, expressive manner. We present five studies that approach this from different but complementary angles. These include: a collaborative, generative sketching application using MCTS and discriminative CNNs; a system to gesturally conduct the realtime generation of text in different styles using an ensemble of LSTM RNNs; a performative tool that allows for the manipulation of hyperparameters in realtime while a Convolutional VAE trains on a live camera feed; a live video feed processing software that allows for digital puppetry and augmented drawing; and a method that allows for long-form story telling within a generative model's latent space with meaningful control over the narrative. We frame our research with the realtime, performative expression provided by musical instruments as a metaphor, in which we think of these systems as not used by a user, but played by a performer

16th Sound and Music Computing Conference SMC 2019 (28–31 May 2019, Malaga, Spain)

The 16th Sound and Music Computing Conference (SMC 2019) took place in Malaga, Spain, 28-31 May 2019 and it was organized by the Application of Information and Communication Technologies Research group (ATIC) of the University of Malaga (UMA). The SMC 2019 associated Summer School took place 25-28 May 2019. The First International Day of Women in Inclusive Engineering, Sound and Music Computing Research (WiSMC 2019) took place on 28 May 2019. The SMC 2019 TOPICS OF INTEREST included a wide selection of topics related to acoustics, psychoacoustics, music, technology for music, audio analysis, musicology, sonification, music games, machine learning, serious games, immersive audio, sound synthesis, etc

Bodies in place : enactive cognition as development of ecological norms

Les partisans de l’approche énactive soutiennent que la cognition se constitue à travers l’histoire des différentes formes d'interaction (biologique, sensorimotrice, intercorporelle, linguistique, etc.) entre un vivant et son environnement. Ces interactions ne sont pas aléatoires, mais des activités obéissant à certaines normes que les énactivistes appellent sense-making. La cognition est, de ce point de vue, une forme de sense-making. Malgré les avantages indéniables que confère une telle perspective pour étudier la cognition, la présente thèse développe un point de vue critique par rapport à l’approche énactive et soutient qu'il est nécessaire d'approfondir notre compréhension de la dimension écologique du sense-making. Le but principal de la thèse est en conséquence de montrer que l'environnement joue un rôle encore plus important que l’approche énactive ne lui attribue habituellement. En m'engageant de manière critique dans le répertoire conceptuel de la cognition énactive, de la phénoménologie et des approches écologiques de la cognition, l’objectif de cette thèse consiste à poser les bases conceptuelles d'une approche énactive-écologique de la cognition. Pour ce faire, la thèse s’attèle à mettre de l’avant trois idées principales. La première consiste à redéfinir le concept du sense-making : contrairement à la conception qui s’est traditionnellement imposée dans le mouvement énactif, nous allons démontrer qu’il s’agit d’un phénomène de développement (et non de création) de normes. La rencontre du corps et du monde est toujours ancrée dans un champ normatif prédéfini, de sorte que nous devons réévaluer le rôle que joue l'environnement dans les processus de sense-making. En effet, si les agents se retrouvent toujours-déjà plongés dans un champ normatif (et non dans un environnement purement causal et physique), il faut alors reconnaître que l'environnement joue un rôle actif dans la constitution et l'auto-transformation des normes de sense-making. La deuxième idée poursuit dans cette veine et porte sur cette nouvelle conception de l'environnement, qui est ici défini comme un champ normatif actif, incarnant une tension entre le passé habituel du système agent-environnement et les contingences incessantes des événements du monde qui poussent le système vers leur auto-transformation et développement. La troisième idée principale de cette thèse consiste en une description holistique du champ d'action des agents (un lieu énactif) et des normes édictées (enacted) par des processus de sense-making sur le terrain (normes de lieu). Une esquisse générale du lieu énactif montre que les activités de sense-making sont liées à des processus écologiques qui enchevêtrent de multiples agents et localités matérielles dans un réseau écologique local. Ces réseaux écologiques forment une unité systémique et résiliente qui se déploie dans le temps avec les habitants du lieu, et fonctionne comme un champ normatif qui contraint et motive l'auto-transformation de chaque système agent-environnementSupporters of autonomist enactivism or the enactive approach claim that cognition is a phenomenon constituted by the historical development of different forms of interaction (biological, sensorimotor, intercorporeal, and linguistic) between living bodies and their environments. For autonomist enactivists, the nature of these interactions is not entirely predetermined by general laws of causation but by norms enacted in the historical path of the agent-environment system, and thanks to processes of sense-making. Cognition is, from the enactivist standpoint, a form of sense-making. While there are multiple advantages in holding such perspective to study mind and cognition, this thesis develops a critical point of view and argues that it is necessary to deepen our understanding of the ecological dimension of sense-making. Specifically, the thesis aims to show that the environment plays a more critical role than autonomist enactivism usually attributes to it. By drawing on and critically engaging with the conceptual repertoire of enactive cognition, phenomenology, and ecological approaches to cognition, my objective is to set the conceptual foundations for an enactive-ecological approach to cognition. For this task, I propose three interrelated ideas. The first redefines sense-making as a phenomenon of norm development. The most common descriptions of sense-making involve the emergence of meaning from raw physical matter thanks to the activity of living organisms. As norm development, by contrast, sense-making refers to a constant enactment and re-enactment of norms of interaction from other pregiven norms, previously enacted in the past of the agent-environment system. I argue that the encounter of the body and the world is permanently embedded in a pregiven normative field and never in an abstract void where raw physical interactions occur. From this standpoint, we need, however, to re-evaluate the role that the environment plays in sense-making processes. If agents find themselves immersed in normative fields and not in raw physical landscapes, then the environment has a more active role for the constitution and self-transformation of sense-making norms than autonomist enactivists have acknowledged. In this vein, the second main idea of this thesis concerns the environment as an active normative field that incarnates a tension between the habitual past of the agent-environment system and the ongoing contingencies of worldly events that push the system to their self-transformation and development. The third main idea of this thesis consists of a holistic description of the field of action of agents (enactive place) and the norms enacted by processes of sense-making in the field (place-norms). A general sketch of enactive place shows that sense-making is tied to processes that entangle multiple agents and material localities into a local ecological web. An enactive place constitutes a systemic and resilient unity that unfolds in time altogether with its inhabitants, working as a normative field that constrains and motivates the self-transformation of each agent-environment system. Bodies are therefore part of wider unities of historical development: places

Neuroevolution in Games: State of the Art and Open Challenges

This paper surveys research on applying neuroevolution (NE) to games. In neuroevolution, artificial neural networks are trained through evolutionary algorithms, taking inspiration from the way biological brains evolved. We analyse the application of NE in games along five different axes, which are the role NE is chosen to play in a game, the different types of neural networks used, the way these networks are evolved, how the fitness is determined and what type of input the network receives. The article also highlights important open research challenges in the field.Comment: - Added more references - Corrected typos - Added an overview table (Table 1

What does semantic tiling of the cortex tell us about semantics?

Recent use of voxel-wise modeling in cognitive neuroscience suggests that semantic maps tile the cortex. Although this impressive research establishes distributed cortical areas active during the conceptual processing that underlies semantics, it tells us little about the nature of this processing. While mapping concepts between Marr's computational and implementation levels to support neural encoding and decoding, this approach ignores Marr's algorithmic level, central for understanding the mechanisms that implement cognition, in general, and conceptual processing, in particular. Following decades of research in cognitive science and neuroscience, what do we know so far about the representation and processing mechanisms that implement conceptual abilities? Most basically, much is known about the mechanisms associated with: (1) features and frame representations, (2) grounded, abstract, and linguistic representations, (3) knowledge-based inference, (4) concept composition, and (5) conceptual flexibility. Rather than explaining these fundamental representation and processing mechanisms, semantic tiles simply provide a trace of their activity over a relatively short time period within a specific learning context. Establishing the mechanisms that implement conceptual processing in the brain will require more than mapping it to cortical (and sub-cortical) activity, with process models from cognitive science likely to play central roles in specifying the intervening mechanisms. More generally, neuroscience will not achieve its basic goals until it establishes algorithmic-level mechanisms that contribute essential explanations to how the brain works, going beyond simply establishing the brain areas that respond to various task conditions

Backwards is the way forward: feedback in the cortical hierarchy predicts the expected future

Clark offers a powerful description of the brain as a prediction machine, which offers progress on two distinct levels. First, on an abstract conceptual level, it provides a unifying framework for perception, action, and cognition (including subdivisions such as attention, expectation, and imagination). Second, hierarchical prediction offers progress on a concrete descriptive level for testing and constraining conceptual elements and mechanisms of predictive coding models (estimation of predictions, prediction errors, and internal models)

The perceptual and cognitive roles of the motor system

The motor system in the brain is crucial in allowing us to successfully move aroundin our environment, interact with people and objects, and execute finely controlled motorcommands. While most of the early neuroscience research on these regions tends to focuson these “main” functions, over the last few decades evidence has been surfacing thatpoints to a more broadly integrated role for the motor system. Many recent findingssuggest that it is also of importance in many other aspects of human cognition, fromlanguage and thought to social cognition and, as I discuss in depth in the followingsections, many perceptual processes. In the following chapters, I outline and compareexisting prediction-based and simulation-based theories for motor system involvement inperception. I also describe experiments I completed investigating motor systeminvolvement in written language perception, music perception, and action observation.Furthermore, I discuss how these processes relate to conceptual learning and recall. Insummary, a vast literature points to the motor system proper not being a neural networkthat is only good for controlling and planning our actions. As we develop the vocabularyof the field to use terms like “action-perception loops” and discuss these processes as lessseparable than previously considered, perhaps we should also reconsider the term “motorsystem” to reflect its diverse roles in sensorimotor prediction

Examining participatory sense-making frames: how autonomous patterns of being together emerge in recurrent social interaction

This thesis investigates how recurrent face-to-face social interactions engender relatively invariant patterns of being together that cause those who instantiate them to act in ways that support their reproduction. Existing accounts within both cognitive science and sociology offer important insights into the consideration of patterns of being together. However, given their explanatory strategies, they struggle to integrate both ‘social’ and ‘individual’ levels of explanation. Herein a compatibilist account is developed, intended as a ‘third way’ that obviates the limitations of existing accounts. This compatibilist account — by integrating insights from across disciplines and theoretical frameworks — develops a philosophical vocabulary with which to maintain explanatory consistency when articulating patterns of being together and moving between individual and social levels of explanation. It relies heavily on an extension of the enactive notion of autonomous habits to the social domain, redescribing patterns of being together as habituated participatory sense-making frames. Participatory sense-making frames result from processes of coenhabiting, i.e. processes of ongoing social habit making implicit in the dynamics of recurrent social interactions. Such processes are one primary means by which we produce and reproduce the relatively stable forms that characterise our shared worlds. These habituated frames embed much of the normativity of social life, and can serve our felicitous coordinations therein, allowing us to feel well situated, particularly in contexts within which we have some history of interacting. However, when they are not well aligned, they lead to tensions that result in either the production of novel frames or breakdowns in social interaction. The account developed has implications for many domains of human action, from psychotherapy to epistemology, and from critical studies to the development of political and ecological praxes