Sequential decision making in artificial musical intelligence

Over the past 60 years, artificial intelligence has grown from a largely academic field of research to a ubiquitous array of tools and approaches used in everyday technology. Despite its many recent successes and growing prevalence, certain meaningful facets of computational intelligence have not been as thoroughly explored. Such additional facets cover a wide array of complex mental tasks which humans carry out easily, yet are difficult for computers to mimic. A prime example of a domain in which human intelligence thrives, but machine understanding is still fairly limited, is music. Over the last decade, many researchers have applied computational tools to carry out tasks such as genre identification, music summarization, music database querying, and melodic segmentation. While these are all useful algorithmic solutions, we are still a long way from constructing complete music agents, able to mimic (at least partially) the complexity with which humans approach music. One key aspect which hasn't been sufficiently studied is that of sequential decision making in musical intelligence. This thesis strives to answer the following question: Can a sequential decision making perspective guide us in the creation of better music agents, and social agents in general? And if so, how? More specifically, this thesis focuses on two aspects of musical intelligence: music recommendation and human-agent (and more generally agent-agent) interaction in the context of music. The key contributions of this thesis are the design of better music playlist recommendation algorithms; the design of algorithms for tracking user preferences over time; new approaches for modeling people's behavior in situations that involve music; and the design of agents capable of meaningful interaction with humans and other agents in a setting where music plays a roll (either directly or indirectly). Though motivated primarily by music-related tasks, and focusing largely on people's musical preferences, this thesis also establishes that insights from music-specific case studies can also be applicable in other concrete social domains, such as different types of content recommendation. Showing the generality of insights from musical data in other contexts serves as evidence for the utility of music domains as testbeds for the development of general artificial intelligence techniques. Ultimately, this thesis demonstrates the overall usefulness of taking a sequential decision making approach in settings previously unexplored from this perspectiveComputer Science

The use of multilayer network analysis in animal behaviour

Network analysis has driven key developments in research on animal behaviour by providing quantitative methods to study the social structures of animal groups and populations. A recent formalism, known as \emph{multilayer network analysis}, has advanced the study of multifaceted networked systems in many disciplines. It offers novel ways to study and quantify animal behaviour as connected 'layers' of interactions. In this article, we review common questions in animal behaviour that can be studied using a multilayer approach, and we link these questions to specific analyses. We outline the types of behavioural data and questions that may be suitable to study using multilayer network analysis. We detail several multilayer methods, which can provide new insights into questions about animal sociality at individual, group, population, and evolutionary levels of organisation. We give examples for how to implement multilayer methods to demonstrate how taking a multilayer approach can alter inferences about social structure and the positions of individuals within such a structure. Finally, we discuss caveats to undertaking multilayer network analysis in the study of animal social networks, and we call attention to methodological challenges for the application of these approaches. Our aim is to instigate the study of new questions about animal sociality using the new toolbox of multilayer network analysis.Comment: Thoroughly revised; title changed slightl

Knowledge management for self-organised resource allocation

Many open systems, such as networks, distributed computing and socio-technical systems address a common problem of how to define knowledge management processes to structure and guide decision-making, coordination and learning. While participation is an essential and desirable feature of such systems, the amount of information produced by its individual agents can often be overwhelming and intractable. The challenge, thus, is how to organise and process such information, so it is transformed into productive knowledge used for the resolution of collective action problems. To address this problem, we consider a study of classical Athenian democracy which investigates how the governance model of the city-state flourished. The work suggests that exceptional knowledge management, i.e. making information available for socially productive purposes, played a crucial role in sustaining its democracy for nearly 200 years, by creating processes for aggregation, alignment and codification of knowledge. We therefore examine the proposition that some properties of this historical experience can be generalised and applied to computational systems, so we establish a set of design principles intended to make knowledge management processes open, inclusive, transparent and effective in self-governed social technical systems. We operationalise three of these principles in the context of a collective action situation, namely self-organised common-pool resource allocation, exploring four governance problems: (a) how fairness can be perceived; (b) how resources can be distributed; (c) how policies should be enforced and (d) how tyranny can be opposed. By applying this operationalisation of the design principles for knowledge management processes as a complement to institutional approaches to governance, we demonstrate empirically how it can guide solutions that satisfice shared values, distribute power fairly, apply "common sense" in dealing with rule violations, and protect agents against abuse of power. We conclude by arguing that this approach to the design of open systems can provide the foundations for sustainable and democratic self-governance in socio-technical systems.Open Acces

Digital ecosystems

We view Digital Ecosystems to be the digital counterparts of biological ecosystems, which are considered to be robust, self-organising and scalable architectures that can automatically solve complex, dynamic problems. So, this work is concerned with the creation, investigation, and optimisation of Digital Ecosystems, exploiting the self-organising properties of biological ecosystems. First, we created the Digital Ecosystem, a novel optimisation technique inspired by biological ecosystems, where the optimisation works at two levels: a first optimisation, migration of agents which are distributed in a decentralised peer-to-peer network, operating continuously in time; this process feeds a second optimisation based on evolutionary computing that operates locally on single peers and is aimed at finding solutions to satisfy locally relevant constraints. We then investigated its self-organising aspects, starting with an extension to the definition of Physical Complexity to include the evolving agent populations of our Digital Ecosystem. Next, we established stability of evolving agent populations over time, by extending the Chli-DeWilde definition of agent stability to include evolutionary dynamics. Further, we evaluated the diversity of the software agents within evolving agent populations, relative to the environment provided by the user base. To conclude, we considered alternative augmentations to optimise and accelerate our Digital Ecosystem, by studying the accelerating effect of a clustering catalyst on the evolutionary dynamics of our Digital Ecosystem, through the direct acceleration of the evolutionary processes. We also studied the optimising effect of targeted migration on the ecological dynamics of our Digital Ecosystem, through the indirect and emergent optimisation of the agent migration patterns. Overall, we have advanced the understanding of creating Digital Ecosystems, the self-organisation that occurs within them, and the optimisation of their Ecosystem-Oriented Architecture

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Responsible machine learning: supporting privacy preservation and normative alignment with multi-agent simulation

This dissertation aims to advance responsible machine learning through multi-agent simulation (MAS). I introduce and demonstrate an open source, multi-domain discrete event simulation framework and use it to: (1) improve state-of-the-art privacy-preserving federated learning and (2) construct a novel method for normatively-aligned learning from synthetic negative examples. Due to their complexity and capacity, the training of modern machine learning (ML) models can require vast user-collected data sets. The current formulation of federated learning arose in 2016 after repeated exposure of sensitive user information from centralized data stores where mobile and wearable training data was aggregated. Privacy-preserving federated learning (PPFL) soon added stochastic and cryptographic layers to protect against additional vectors of data exposure. Recent state of the art protocols have combined differential privacy (DP) and secure multiparty computation (MPC) to keep client training data set parameters private from an ``honest but curious'' server which is legitimately involved in the learning process, but attempting to infer information it should not have. Investigation of PPFL can be cost prohibitive if each iteration of a proposed experimental protocol is distributed to virtual computational nodes geolocated around the world. It can also be inaccurate when locally simulated without concern for client parallelism, accurate timekeeping, or computation and communication loads. In this work, a recent PPFL protocol is instantiated as a single-threaded MAS to show that its model accuracy, deployed parallel running time, and resistance to inference of client model parameters can be inexpensively evaluated. The protocol is then extended using oblivious distributed differential privacy to a new state of the art secure against attacks of collusion among all except one participant, with an empirical demonstration that the new protocol improves privacy with no loss of accuracy to the final model. State of the art reinforcement learning (RL) is also increasingly complex and hard to interpret, such that a sequence of individually innocuous actions may produce an unexpectedly harmful result. Safe RL seeks to avoid these results through techniques like reward variance reduction, error state prediction, or constrained exploration of the state-action space. Development of the field has been heavily influenced by robotics and finance, and thus it is primarily concerned with physical failures like a helicopter crash or a robot-human workplace collision, or monetary failures like the depletion of an investment account. The related field of Normative RL is concerned with obeying the behavioral expectations of a broad human population, like respecting personal space or not sneaking up behind people. Because normative behavior often implicates safety, for example the assumption that an autonomous navigation robot will not walk through a human to reach its goal more quickly, there is significant overlap between the two areas. There are problem domains not easily addressed by current approaches in safe or normative RL, where the undesired behavior is subtle, violates legal or ethical rather than physical or monetary constraints, and may be composed of individually-normative actions. In this work, I consider an intelligent stock trading agent that maximizes profit but may inadvertently learn ``spoofing'', a form of illegal market manipulation that can be difficult to detect. Using a financial market based on MAS, I safely coerce a variety of spoofing behaviors, learn to distinguish them from other profit-driven strategies, and carefully analyze the empirical results. I then demonstrate how this spoofing recognizer can be used as a normative guide to train an intelligent trading agent that will generate positive returns while avoiding spoofing behaviors, even if their adoption would increase short-term profits. I believe this contribution to normative RL, of deriving an method for normative alignment from synthetic non-normative action sequences, should generalize to many other problem domains.Ph.D

Distributed Load Testing by Modeling and Simulating User Behavior

Modern human-machine systems such as microservices rely upon agile engineering practices which require changes to be tested and released more frequently than classically engineered systems. A critical step in the testing of such systems is the generation of realistic workloads or load testing. Generated workload emulates the expected behaviors of users and machines within a system under test in order to find potentially unknown failure states. Typical testing tools rely on static testing artifacts to generate realistic workload conditions. Such artifacts can be cumbersome and costly to maintain; however, even model-based alternatives can prevent adaptation to changes in a system or its usage. Lack of adaptation can prevent the integration of load testing into system quality assurance, leading to an incomplete evaluation of system quality. The goal of this research is to improve the state of software engineering by addressing open challenges in load testing of human-machine systems with a novel process that a) models and classifies user behavior from streaming and aggregated log data, b) adapts to changes in system and user behavior, and c) generates distributed workload by realistically simulating user behavior. This research contributes a Learning, Online, Distributed Engine for Simulation and Testing based on the Operational Norms of Entities within a system (LODESTONE): a novel process to distributed load testing by modeling and simulating user behavior. We specify LODESTONE within the context of a human-machine system to illustrate distributed adaptation and execution in load testing processes. LODESTONE uses log data to generate and update user behavior models, cluster them into similar behavior profiles, and instantiate distributed workload on software systems. We analyze user behavioral data having differing characteristics to replicate human-machine interactions in a modern microservice environment. We discuss tools, algorithms, software design, and implementation in two different computational environments: client-server and cloud-based microservices. We illustrate the advantages of LODESTONE through a qualitative comparison of key feature parameters and experimentation based on shared data and models. LODESTONE continuously adapts to changes in the system to be tested which allows for the integration of load testing into the quality assurance process for cloud-based microservices

Stability of a time-homogeneous system of money and antimoney & kinetic microscale thermophoresis

Human history is pervaded with financial crises. Lately, the global financial crisis of 2008 highlighted the role of uncontrolled creation of money through lending as a relevant source of financial instability. Motivated by an analogy to particle physics, time-homogeneity is imposed on monetary systems to approach and even possibly solve the associated problems. This implies a full reserve banking with a two-currency system which discriminates with an exchange rate between non-bank assets (money) and bank assets (antimoney). Payments can be made by passing on money in exchange for a good or receiving antimoney along with the good at respective price levels. Liquidity can be provided by the simultaneous transfer of money and antimoney at a negotiated exchange rate between money and antimoney, also termed the liquidity price. In this system, interest rates and credit creation are replaced by a varying price for liquidity. Here, the economic stability of such a system with an agent-based random economy model is studied, in which households and firms are urged by random boundary conditions to apply stochastic exchanges of goods via a limit order book mechanism, implementing the trading scheme of stock markets. The comparison of the market simulations for equilibrium and external shock scenarios of the prevailing monetary system with the money-antimoney system highlights two core aspects: First, the need of debt-limiting boundary conditions in order to equilibrate markets and second the similarity of the price dynamics of the studied systems as an indicator of fundamental functionality of the money-antimoney system. The formation, stability and dissociation of biological ligand-binder systems play a fundamental role in nearly all aspects of living matter. Whereas the binding affinity is well described for many molecular ligand-binder interactions, their kinetic association and dissociation rates are far less well studied, due to lack of comprehensive experimental techniques. Here, Kinetic Microscale Thermophoresis KMST is established, which allows for a purely optical, immobilization-free and quantitative analysis of kinetic rates of biological ligand-binder processes. In a KMST measurement, the kinetic fingerprint is extracted from the fluorescence change back to equilibrium within a formerly IR laser-heated spot. Kinetic relaxation time constants between 0.01-0.5 /s can be measured, allowing for the determination of on-rates 10^4-10^6 /Ms and off-rates 10^-4 - 0.1/s in principle. For Cy5-labeled DNA strands, the expected exponential dependence of the off-rates on salt concentration, strand length and inverse temperature, respectively, was confirmed and measurements in crowded solutions were performed. The measured on-rates show linear dependence on salt concentration but weak dependence on strand length and inverse temperature. For biological reaction processes with sufficient enthalpic component, KMST offers a suitable immobilization-free determination of kinetic ratesFinanzkrisen treten in der Geschichte der Menschheit frequent auf. Die globale Finanzkrise machte 2008 im Besonderen die Rolle unkontrollierter Kreditschöpfung als relevante Quelle finanzieller Instabilität deutlich. Motiviert durch eine Analogie zur Teilchenphysik wird Zeithomogenität auf ein Geldsystem angewandt, um die mit Kreditschöpfung verbundenen Probleme zu mindern und möglicherweise auch zu lösen. Das führt zu einem Vollgeldsystem mit zwei distinkten Währungen: Nichtbankaktiva (Geld) und Bankaktiva (Antigeld). In diesem Geld-Antigeldsystem kö̈nnen Zahlungen durch Entgegennahme von Geld gegen eine Ware oder durch den Transfer von Antigeld zusammen mit der Ware zum jeweiligen Preis abgewickelt werden. Liquidität kann durch den gleichzeitigen Transfer von Geld und Antigeld zu einem ausgehandelten Liquiditä̈tspreis zwischen Geld und Antigeld bereitgestellt werden. Der variierende Liquiditätspreis ersetzt Zinssätze und Kreditschöpfung. Die wirtschaftliche Stabilität eines solchen Systems wurde mithilfe eines agentenbasierten Zufallsökonomiemodells untersucht. In diesem handeln Haushalte und Unternehmen stochastisch Güter auf einer zentralen Limit-Orderbuch Börse äquivalent zu Handelsschemata an Aktienmärkten. Der Vergleich der Marktsimulationen für Gleichgewichts- und externe Schockszenarien des vorherrschenden Geldsystems mit dem Geld- Antigeld-System verdeutlicht zwei Kernergebnisse: Erstens, die Notwendigkeit individueller schuldenbegrenzender Randbedingungen, um effiziente Gleichgewichte in den Märkten zu erzielen und zweitens eine ähnliche Preisdynamik zum vorherrschenden Geldsystem, die auf eine grundlegende Funktionalität des Geld-Antigeldsystems hinweist. Die Bildung, Stabilität und Dissoziation biologischer Liganden-Binder-Systeme spielen eine fundamentale Rolle in fast allen Aspekten belebter Materie. Während die Bindungsaffinitäten für viele molekulare Liganden-Binder-Systeme gut beschrieben sind, sind die kinetischen Assoziations- und Dissoziationsraten weniger gut erforscht. Mit der weit verbreiteten Microscale Thermophoresis Technik MST, werden mittels mikroskaliger Temperaturgradienten die Affinitäten von Ligand-Bindersystemen bestimmt. Durch eine im Vergleich zur MST Technik um eine Größenordnung verbesserte thermische Anbdingung des Samples kö̈nnen mit der neuartigen kinetische mikroskalige Thermophorese Technik KMST zusätzlich quantitativ die kinetischen Raten auf rein optischer und immo- bilisationsfreier Basis bestimmt werden. Bei einer KMST-Messung wird der kinetische Fingerabdruck aus der Fluoreszenzänderung zurück zum Gleichgewicht innerhalb eines zuvor IR-laserbeheizten Spots extrahiert. Gemessene kinetische Relaxationszeitkonstanten zwischen 0.01–0.5 /s lassen die Bestimmung von Assoziationsraten 10^4–10^6 /Ms und Dissoziationsraten 10^-4 - 10^-1 /s prinzipiell zu. Für Cy5-markierte komplementäre DNA-Stränge wurde die erwartete exponentielle Abhängigkeit der Dissozitationsraten und Dissoziationskonstanten von Salzkonzentration, Stranglänge und inverser Temperatur bestätigt und Raten in makromolekular gedrängten Lösungen gemessen. Die gemessenen Assoziationsraten weisen eine lineare Abhängigkeit von der Salzkonzentration auf, eine schwache Abhängigkeit von der Stranglänge und der inversen Temperatur. Für biologische Reaktionsprozesse mit ausreichender enthalpischer Komponente bietet KMST eine immobilisationsfreie Bestimmung der kinetischen Raten

Recent Advances in General Game Playing

The goal of General Game Playing (GGP) has been to develop computer programs that can perform well across various game types. It is natural for human game players to transfer knowledge from games they already know how to play to other similar games. GGP research attempts to design systems that work well across different game types, including unknown new games. In this review, we present a survey of recent advances (2011 to 2014) in GGP for both traditional games and video games. It is notable that research on GGP has been expanding into modern video games. Monte-Carlo Tree Search and its enhancements have been the most influential techniques in GGP for both research domains. Additionally, international competitions have become important events that promote and increase GGP research. Recently, a video GGP competition was launched. In this survey, we review recent progress in the most challenging research areas of Artificial Intelligence (AI) related to universal game playing