The association between liking, learning and creativity in music

Aesthetic preference is intricately linked to learning and creativity. Previous studies have largely examined the perception of novelty in terms of pleasantness and the generation of novelty via creativity separately. The current study examines the connection between perception and generation of novelty in music; specifically, we investigated how pleasantness judgements and brain responses to musical notes of varying probability (estimated by a computational model of auditory expectation) are linked to learning and creativity. To facilitate learning de novo, 40 non-musicians were trained on an unfamiliar artificial music grammar. After learning, participants evaluated the pleasantness of the final notes of melodies, which varied in probability, while their EEG was recorded. They also composed their own musical pieces using the learned grammar which were subsequently assessed by experts. As expected, there was an inverted U-shaped relationship between liking and probability: participants were more likely to rate the notes with intermediate probabilities as pleasant. Further, intermediate probability notes elicited larger N100 and P200 at posterior and frontal sites, respectively, associated with prediction error processing. Crucially, individuals who produced less creative compositions preferred higher probability notes, whereas individuals who composed more creative pieces preferred notes with intermediate probability. Finally, evoked brain responses to note probability were relatively independent of learning and creativity, suggesting that these higher-level processes are not mediated by brain responses related to performance monitoring. Overall, our findings shed light on the relationship between perception and generation of novelty, offering new insights into aesthetic preference and its neural correlates

Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making

In multi-objective decision planning and learning, much attention is paid to producing optimal solution sets that contain an optimal policy for every possible user preference profile. We argue that the step that follows, i.e, determining which policy to execute by maximising the user's intrinsic utility function over this (possibly infinite) set, is under-studied. This paper aims to fill this gap. We build on previous work on Gaussian processes and pairwise comparisons for preference modelling, extend it to the multi-objective decision support scenario, and propose new ordered preference elicitation strategies based on ranking and clustering. Our main contribution is an in-depth evaluation of these strategies using computer and human-based experiments. We show that our proposed elicitation strategies outperform the currently used pairwise methods, and found that users prefer ranking most. Our experiments further show that utilising monotonicity information in GPs by using a linear prior mean at the start and virtual comparisons to the nadir and ideal points, increases performance. We demonstrate our decision support framework in a real-world study on traffic regulation, conducted with the city of Amsterdam.Comment: AAMAS 2018, Source code at https://github.com/lmzintgraf/gp_pref_elici

Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach

Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor $\gamma < 1$, or in episodic settings, with $\gamma = 1$. While this has proven effective for specific tasks with well-defined objectives (e.g., games), it has never been established that fixed discounting is suitable for general purpose use (e.g., as a model of human preferences). This paper characterizes rationality in sequential decision making using a set of seven axioms and arrives at a form of discounting that generalizes traditional fixed discounting. In particular, our framework admits a state-action dependent "discount" factor that is not constrained to be less than 1, so long as there is eventual long run discounting. Although this broadens the range of possible preference structures in continuous settings, we show that there exists a unique "optimizing MDP" with fixed $\gamma < 1$ whose optimal value function matches the true utility of the optimal policy, and we quantify the difference between value and utility for suboptimal policies. Our work can be seen as providing a normative justification for (a slight generalization of) Martha White's RL task formalism (2017) and other recent departures from the traditional RL, and is relevant to task specification in RL, inverse RL and preference-based RL.Comment: 8 pages + 1 page supplement. In proceedings of AAAI 2019. Slides, poster and bibtex available at https://silviupitis.com/#rethinking-the-discount-factor-in-reinforcement-learning-a-decision-theoretic-approac

A Voting-Based System for Ethical Decision Making

We present a general approach to automating ethical decisions, drawing on machine learning and computational social choice. In a nutshell, we propose to learn a model of societal preferences, and, when faced with a specific ethical dilemma at runtime, efficiently aggregate those preferences to identify a desirable choice. We provide a concrete algorithm that instantiates our approach; some of its crucial steps are informed by a new theory of swap-dominance efficient voting rules. Finally, we implement and evaluate a system for ethical decision making in the autonomous vehicle domain, using preference data collected from 1.3 million people through the Moral Machine website.Comment: 25 pages; paper has been reorganized, related work and discussion sections have been expande

A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences

Implicit cognition is impaired and dissociable in a head-injured group with executive deficits

Implicit or non-conscious cognition is traditionally assumed to be robust to pathology but Gomez-Beldarrain et al (1999, 2002) recently showed deficits on a single implicit task after head injury. Laboratory research suggests that implicit processes dissociate. This study therefore examined implicit cognition in 20 head-injured patients and age- and I.Q.-matched controls using a battery of four implicit cognition tasks: a Serial Reaction Time task (SRT), mere exposure effect task, automatic stereotype activation and hidden co-variation detection. Patients were assessed on an extensive neuropsychological battery, and MRI scanned. Inclusion criteria included impairment on at least one measure of executive function. The patient group was impaired relative to the control group on all the implicit cognition tasks except automatic stereotype activation. Effect size analyses using the control mean and standard deviation for reference showed further dissociations across patients and across implicit tasks. Patients impaired on implicit tasks had more cognitive deficits overall than those unimpaired, and a larger Dysexecutive Self/Other discrepancy (DEX) score suggesting greater behavioural problems. Performance on the SRT task correlated with a composite measure of executive function. Head-injury thus produced heterogeneous impairments in the implicit acquisition of new information. Implicit activation of existing knowledge structures appeared intact. Impairments in implicit cognition and executive function may interact to produce dysfunctional behaviour after head-injury. Future comparisons of implicit and explicit cognition should use several measures of each function, to ensure that they measure the latent variable of interest

The contribution of injury severity, executive and implicit functions to awareness of defi cits after traumatic brain injury (TBI)

Deficits in self-awareness are commonly seen after Traumatic Brain Injury (TBI) and adversely affect rehabilitative efforts, independence and quality of life (Ponsford, 2004). Awareness models predict that executive and implicit functions are important cognitive components of awareness though the putative relationship between implicit and awareness processes has not been subject to empirical investigation (Crosson et al., 1989; Ownsworth, Clare, & Morris, 2006; Toglia & Kirk, 2000). Severity of injury, also thought to be a crucial determinant of awareness outcome post-insult, is under-explored in awareness studies (Sherer, Boake, Levin, Silver, Ringholz, & Walter, 1998 ). The present study measured the contribution of injury severity, IQ, mood state, executive and implicit functions to awareness in head-injured patients assigned to moderate/severe head-injured groups using several awareness, executive, and implicit measures. Severe injuries resulted in greater impairments across most awareness, executive and implicit measures compared with moderate injuries, although deficits were still seen in the moderate group. Hierarchical regression results showed that severity of injury, IQ, mood state, executive and implicit functions made signifi cant unique contributions to selective aspects of awareness. Future models of awareness should account for both implicit and executive contributions to awareness and the possibility that both are vulnerable to disruption after neuropathology. ( JINS , 2010, 16 , 1– 10 .