Building machines that learn and think about morality

Lake et al. propose three criteria which, they argue, will bring artificial intelligence (AI) systems closer to human cognitive abilities. In this paper, we explore the application of these criteria to a particular domain of human cognition: our capacity for moral reasoning. In doing so, we explore a set of considerations relevant to the development of AI moral decision-making. Our main focus is on the relation between dual-process accounts of moral reasoning and model-free/model-based forms of machine learning. We also discuss how work in embodied and situated cognition could provide a valu- able perspective on future research

Modeling Human Understanding of Complex Intentional Action with a Bayesian Nonparametric Subgoal Model

Most human behaviors consist of multiple parts, steps, or subtasks. These structures guide our action planning and execution, but when we observe others, the latent structure of their actions is typically unobservable, and must be inferred in order to learn new skills by demonstration, or to assist others in completing their tasks. For example, an assistant who has learned the subgoal structure of a colleague's task can more rapidly recognize and support their actions as they unfold. Here we model how humans infer subgoals from observations of complex action sequences using a nonparametric Bayesian model, which assumes that observed actions are generated by approximately rational planning over unknown subgoal sequences. We test this model with a behavioral experiment in which humans observed different series of goal-directed actions, and inferred both the number and composition of the subgoal sequences associated with each goal. The Bayesian model predicts human subgoal inferences with high accuracy, and significantly better than several alternative models and straightforward heuristics. Motivated by this result, we simulate how learning and inference of subgoals can improve performance in an artificial user assistance task. The Bayesian model learns the correct subgoals from fewer observations, and better assists users by more rapidly and accurately inferring the goal of their actions than alternative approaches.Comment: Accepted at AAAI 1

Session 5: Development, Neuroscience and Evolutionary Psychology

Proceedings of the Pittsburgh Workshop in History and Philosophy of Biology, Center for Philosophy of Science, University of Pittsburgh, March 23-24 2001 Session 5: Development, Neuroscience and Evolutionary Psycholog

Hierarchical Influences on Human Decision-Making

Deciding how to act is complicated because people often hold simultaneous intentions to meet multiple goals. These many goals can be arranged in a hierarchy of goals and sub-goals, and a hierarchy of behaviours can be established to attain them. The hierarchical structure of human behaviour is well established, but the precise form of that hierarchical structure remains unclear. Further, we do not know whether and how this hierarchical organisation of action influences the cognitive processes of deciding between candidate actions. This thesis aims to address these two open questions. In Chapter 2, I tackle the first of these two questions. Using behavioural experiments in combination with hierarchical reinforcement learning models of behaviour, I demonstrate that people can learn entirely novel sequences of action without practice, and that this ability requires a hierarchical organisation of action built from two distinct operations. First, the brain must sequence low-level components into higher-level routines of action. Second, the brain must have a method of abstracting the relational structure of a sequence away from its content. In sum, this chapter provides evidence for a theoretical framework which can be used to understand hierarchically structured action more deeply. In Chapters 3 and 4, I tackle the second question: does hierarchical structure influence decision-making? I begin (in Chapter 3) by investigating how hierarchical structure and self-efficacy interact to influence choice between candidate actions. I find that higher level actions are associated with lesser self-efficacy and therefore a lesser willingness to commit to them. This effect arises not only because higher-level actions are more difficult to carry out due to their length, but also because the restrictions that they place on future choices represent a cost. I then (in Chapter 4) investigate whether there are any subjective biases in how outcomes at high or low hierarchical levels are evaluated. I find no overall subjective bias in the evaluation of such outcomes, but I find that social context can prompt strong biases to weight evaluation of outcomes according to their hierarchical level. In sum, I find that hierarchical structure can and does influence decision-making, and I provide evidence for two distinct processes that play a part in this. These findings establish both a novel theoretical framework for future investigations of hierarchically structured action, and a novel set of interactions between the structure of behaviour and how people make action decisions

The role of prediction and outcomes in adaptive cognitive control

Humans adaptively perform actions to achieve their goals. This flexible behaviour requires two core abilities: the ability to anticipate the outcomes of candidate actions and the ability to select and implement actions in a goal-directed manner. The ability to predict outcomes has been extensively researched in reinforcement learning paradigms, but this work has often focused on simple actions that are not embedded in hierarchical and sequential structures that are characteristic of goal-directed human behaviour. On the other hand, the ability to select actions in accordance with high-level task goals, particularly in the presence of alternative responses and salient distractors, has been widely researched in cognitive control paradigms. Cognitive control research, however, has often paid less attention to the role of action outcomes. The present review attempts to bridge these accounts by proposing an outcome-guided mechanism for selection of extended actions. Our proposal builds on constructs from the hierarchical reinforcement learning literature, which emphasises the concept of reaching and evaluating informative states, i.e., states that constitute subgoals in complex actions. We develop an account of the neural mechanisms that allow outcome-guided action selection to be achieved in a network that relies on projections from cortical areas to the basal ganglia and back-projections from the basal ganglia to the cortex. These cortico-basal ganglia-thalamo-cortical ‘loops’ allow convergence – and thus integration – of information from non-adjacent cortical areas (for example between sensory and motor representations). This integration is essential in action sequences, for which achieving an anticipated sensory state signals the successful completion of an action. We further describe how projection pathways within the basal ganglia allow selection between representations, which may pertain to movements, actions, or extended action plans. The model lastly envisages a role for hierarchical projections from the striatum to dopaminergic midbrain areas that enable more rostral frontal areas to bias the selection of inputs from more posterior frontal areas via their respective representations in the basal ganglia.This work is supported by the Biotechnology and Biological Sciences Research Council (BBSRC) Grant BB/I019847/1, awarded to NY and FW

Hierarchical control over effortful behavior by rodent medial frontal cortex : a computational model

The anterior cingulate cortex (ACC) has been the focus of intense research interest in recent years. Although separate theories relate ACC function variously to conflict monitoring, reward processing, action selection, decision making, and more, damage to the ACC mostly spares performance on tasks that exercise these functions, indicating that they are not in fact unique to the ACC. Further, most theories do not address the most salient consequence of ACC damage: impoverished action generation in the presence of normal motor ability. In this study we develop a computational model of the rodent medial prefrontal cortex that accounts for the behavioral sequelae of ACC damage, unifies many of the cognitive functions attributed to it, and provides a solution to an outstanding question in cognitive control research-how the control system determines and motivates what tasks to perform. The theory derives from recent developments in the formal study of hierarchical control and learning that highlight computational efficiencies afforded when collections of actions are represented based on their conjoint goals. According to this position, the ACC utilizes reward information to select tasks that are then accomplished through top-down control over action selection by the striatum. Computational simulations capture animal lesion data that implicate the medial prefrontal cortex in regulating physical and cognitive effort. Overall, this theory provides a unifying theoretical framework for understanding the ACC in terms of the pivotal role it plays in the hierarchical organization of effortful behavior