Reinforcement Learning for Value Alignment

[eng] As autonomous agents become increasingly sophisticated and we allow them to perform more complex tasks, it is of utmost importance to guarantee that they will act in alignment with human values. This problem has received in the AI literature the name of the value alignment problem. Current approaches apply reinforcement learning to align agents with values due to its recent successes at solving complex sequential decision-making problems. However, they follow an agent-centric approach by expecting that the agent applies the reinforcement learning algorithm correctly to learn an ethical behaviour, without formal guarantees that the learnt ethical behaviour will be ethical. This thesis proposes a novel environment-designer approach for solving the value alignment problem with theoretical guarantees. Our proposed environment-designer approach advances the state of the art with a process for designing ethical environments wherein it is in the agent's best interest to learn ethical behaviours. Our process specifies the ethical knowledge of a moral value in terms that can be used in a reinforcement learning context. Next, our process embeds this knowledge in the agent's learning environment to design an ethical learning environment. The resulting ethical environment incentivises the agent to learn an ethical behaviour while pursuing its own objective. We further contribute to the state of the art by providing a novel algorithm that, following our ethical environment design process, is formally guaranteed to create ethical environments. In other words, this algorithm guarantees that it is in the agent's best interest to learn value- aligned behaviours. We illustrate our algorithm by applying it in a case study environment wherein the agent is expected to learn to behave in alignment with the moral value of respect. In it, a conversational agent is in charge of doing surveys, and we expect it to ask the users questions respectfully while trying to get as much information as possible. In the designed ethical environment, results confirm our theoretical results: the agent learns an ethical behaviour while pursuing its individual objective.[cat] A mesura que els agents autònoms es tornen cada cop més sofisticats i els permetem realitzar tasques més complexes, és de la màxima importància garantir que actuaran d'acord amb els valors humans. Aquest problema ha rebut a la literatura d'IA el nom del problema d'alineació de valors. Els enfocaments actuals apliquen aprenentatge per reforç per alinear els agents amb els valors a causa dels seus èxits recents a l'hora de resoldre problemes complexos de presa de decisions seqüencials. Tanmateix, segueixen un enfocament centrat en l'agent en esperar que l'agent apliqui correctament l'algorisme d'aprenentatge de reforç per aprendre un comportament ètic, sense garanties formals que el comportament ètic après serà ètic. Aquesta tesi proposa un nou enfocament de dissenyador d'entorn per resoldre el problema d'alineació de valors amb garanties teòriques. El nostre enfocament de disseny d'entorns proposat avança l'estat de l'art amb un procés per dissenyar entorns ètics en què és del millor interès de l'agent aprendre comportaments ètics. El nostre procés especifica el coneixement ètic d'un valor moral en termes que es poden utilitzar en un context d'aprenentatge de reforç. A continuació, el nostre procés incorpora aquest coneixement a l'entorn d'aprenentatge de l'agent per dissenyar un entorn d'aprenentatge ètic. L'entorn ètic resultant incentiva l'agent a aprendre un comportament ètic mentre persegueix el seu propi objectiu. A més, contribuïm a l'estat de l'art proporcionant un algorisme nou que, seguint el nostre procés de disseny d'entorns ètics, està garantit formalment per crear entorns ètics. En altres paraules, aquest algorisme garanteix que és del millor interès de l'agent aprendre comportaments alineats amb valors. Il·lustrem el nostre algorisme aplicant-lo en un estudi de cas on s'espera que l'agent aprengui a comportar-se d'acord amb el valor moral del respecte. En ell, un agent de conversa s'encarrega de fer enquestes, i esperem que faci preguntes als usuaris amb respecte tot intentant obtenir la màxima informació possible. En l'entorn ètic dissenyat, els resultats confirmen els nostres resultats teòrics: l'agent aprèn un comportament ètic mentre persegueix el seu objectiu individual

Case Study on Design Management: Inefficiencies and Possible Remedies

Delivering better products with a reduced lead time and less resources has become the primary focus of design management. The aim of this work is to revisit typical design management inefficiencies and discuss possible remedies for these problems. To this end, a case study and interviews with seven Estonian architects were carried out. The data obtained was analyzed within the framework of the transformation-flow-value theory of production. Despite its failure to deliver customer value, a single-minded transformation view of operations has been the dominant approach taken in design management and processes, leading to inefficiencies in design practices

Developing and Testing Digital Ethical Reflection in Long-Term Care: Nurses’ Experiences

publishedVersio

Evolutionary games on graphs

Game theory is one of the key paradigms behind many scientific disciplines from biology to behavioral sciences to economics. In its evolutionary form and especially when the interacting agents are linked in a specific social network the underlying solution concepts and methods are very similar to those applied in non-equilibrium statistical physics. This review gives a tutorial-type overview of the field for physicists. The first three sections introduce the necessary background in classical and evolutionary game theory from the basic definitions to the most important results. The fourth section surveys the topological complications implied by non-mean-field-type social network structures in general. The last three sections discuss in detail the dynamic behavior of three prominent classes of models: the Prisoner's Dilemma, the Rock-Scissors-Paper game, and Competing Associations. The major theme of the review is in what sense and how the graph structure of interactions can modify and enrich the picture of long term behavioral patterns emerging in evolutionary games.Comment: Review, final version, 133 pages, 65 figure

Cognition and norms: toward a developmental account of moral agency in social dilemmas

Most recent developments in the study of social dilemmas give an increasing amount of attention to cognition, belief systems, valuations, and language. However, developments in this field operate almost entirely under epistemological assumptions which only recognize the instrumental form of rationality and deny that “value judgments” or “moral questions” have cognitive content. This standpoint erodes the moral aspect of the choice situation and obstructs acknowledgment of the links connecting cognition, inner growth, and moral reasoning, and the significance of such links in reaching cooperative solutions to many social dilemmas. Concurrently, this standpoint places the role of communication and mutual understanding in promoting cooperation in morally relevant conflicts of action in a rather mysterious situation. This paper draws on Habermas’s critique of instrumental action, and on the most recent developments in institutional and behavioral economics with a view to enhancing our knowledge of the interventions used to cope with social dilemmas. We conclude the paper with a brief presentation of a research strategy for examining the capacity of alternative developmental models to predict dissimilar choices under similar incentive conditions in social dilemmas

The influence of culture and ethical ideology on ethical decision making process of Malaysian managers

The ethical decision making (EDM) process of individuals has proven to be very challenging due to the multitude of complex and varied factors that contribute to this behaviour. Nevertheless, many theoretical frameworks have been proposed to describe such behaviour. Despite the increasing number of EDM models and practices, reports of unethical incidents and illegal behaviour, especially in organizations, continue to appear. This problem is in part related to the failure of EDM models that do not adequately explain the antecedent factors of ethical behaviour by individuals in organizations. As a result, there is a gap between theory and practice. Hence, the main objective of this study is to address these shortcomings by exploring and empirically examining the antecedents in the EDM process and by developing a behavioural model that encompasses a fully functional model of individual ethical behaviour.Furthermore, most of the major frameworks have been formulated from a Western perspective. The current research was conducted in the Malaysian context, using a model that accounts for cultural differences. The most generally accepted concept is that culture is a key determinant of an individual's ethical ideology, which affects an individual’s inclination to behave ethically. In other words, culture acts as a guideline in determining whether certain practices are appropriate and acceptable. However, individuals in organizations also have an obligation to comply the culture in their work setting. Therefore, the culture of ethical practices in an organization is expected to play an important role in the process of EDM. In addition, literature has established the moderating effect of gender, age and level of education in such behaviour. Therefore, the current research also further investigated the relationship among the components of the EDM process, which has thus far not been given proper attention.A two-phase sequential mixed-method approach, consisting of qualitative and quantitative approaches, was carried out in this research. In the qualitative part, a field study of semi-structured interviews was conducted with the objective of testing the applicability of the initial model besides exploring the dimensionality of the construct. The data was analyzed using content analysis through Nvivo software. Based on the analysis, the dimensionality of the constructs was identified and two more relevant antecedent factors were detected, namely parental values and religiosity. Thus, these two factors were added to the comprehensive research model. A survey was conducted among managers from Malaysia large organizations to collect quantitative data. The data was analyzed using a Partial Least Square (PLS) based Structural Equations Modelling (SEM) tool. In addition, multi-group analysis of PLS was also employed.The major findings of this research have confirmed the influence of ethical ideology as the major determinant of the EDM process. Furthermore, results have demonstrated the role of culture as the antecedent of an individual’s ethical ideology. Interestingly, the influence of parental values and religiosity, which was derived from the field study, was also confirmed. Findings also verified the moderating effect of the organizational ethical culture in enhancing a moral awareness of an individual’s EDM process. With regards to the relationship among the components of the EDM process, findings revealed sequential and interrelationship links between the components. On the other hand, the moderating effect of gender, age and level of education in the EDM process was not detected. Overall results confirm the multidimensional construct and the complexity of such a process. This research provides a significant contribution to the existing knowledge in the EDM area. Most importantly, an understanding of the antecedents and the processes of EDM provides guidelines for organizations in developing better ethical programs and policies in order to promote and encourage ethical behaviour. Perhaps a major contribution of this research is the implication for managers to enhance the process of EDM in organizations

Ethnographic Intimacy: Thinking Through the Ethics of Social Research in Sex Worlds

Ethnographic researchers entering sensitive fields of research become entangled in ethical dilemmas when they encounter \'sticky\' questions, situations and issues. In undertaking research within two distinct sex worlds: female sex work and male sexual negotiation/risk and HIV, we struggled to manage the contingent links between our relationships with the people who inhabit these worlds, the ethical requirements of our institutional ethics committees, and our hybrid selves. In the context of \'doing\' intimate ethnography, we were required to craft ourselves into the field and establish a number of intimate and prolonged relationships. While the participants in our studies were active in giving their consent, this did not obviate the risk that they would become objectified within the field relationship and the texts the research generated. These issues are central to our discussion as we consider the lack of fit between ethical guidelines and the practical reality of fieldwork.Ethnography; Informed Consent; Ethics Committees; Reflexivity; Sex Work; Risk and HIV

An Inquiry into the Piagetian Tradition in America as a Basis for a Philosophy of Education at the Communityy College Level: A Quasi-Experimental Approach

The purpose of this study was to investigate the utility of a structural-developmental approach as a basis for a philosophy of education for the public community college of the Midwestern United States. The structural developmental approach was explored within the broader cognitive studies of the Piagetian tradition in America. The method of research employed was a combination of analysis of relevant literature and empirical investigation at a community college. This combination was explored in order to determine the possible abstraction of a structural-developmental pattern as a potential base for theory building. A research instrument was used to assess the cognitive stage of development of critical groups of students. The drop, course withdrawal. Grade point average, and cognitive stage data of lower ability vocational-technical, lower ability transfer, and upper ability students were compared and analyzed, with a .05 level of significance as the criterion for differences. Regarding the relationships of vocational-technical courses and transfer courses, the following observations seemed justified concerning the abstraction level required: (1) no evidence was gained to support the action of a true difference in abstraction level required (2) limited data from other literature offered no substantial support for a true difference, and (3) an argument from silence suggested no true difference. Regarding supportive evidence for the utility of a cognitive-developmental-structural approach to the rural community college educational task, the following summary of findings seemed justified: (l) course withdrawal, grade point average, and cognitive stage data of upper ability and lower ability student’s appeared to indicate a utility for the approach (2) consistency theory offered the possibility of a favorable climate for an interactional (naturalistic and environmental) approach such as was investigated (3) the literature within the community college framework suggested the need for a more comprehensive philosophy than had crystalized; and (4) the literature which had explored the ramifications of the Piagetian tradition in America provided a rationale for a cognitive-developmental-structural approach. From the findings and the results of the analyzed information in this study, the following conclusions seemed justified: (l) evidence from liter3ture and empirical investigations demonstrated the need for a broadly based psychology of education not presently apparent in the community college movement (2) vocational-technical programs presented no unique solution to the breadth of the educational task: (3) diversity in student capability demanded a more universal basis for a philosophy of education: (4) evidence from the literature and empirical inquiry destroyed the myth that the Piagetian tradition was age-bound at a level below community college functions: (5) Piagetian tradition provided a universal kind of basis for a philosophy of education: (6) natural diversity of the college setting studied was conducive to a broadly based psychology of the individual student; (7) cognitive stages were abstracted from the mental functions of students involved in the study and found to be relevant to the educational processes of the community college: and (8) implementation of a definitive cognitive-developmental approach to the educational task of grades thirteen and fourteen would have the advantage of providing a continuity with the educational modes of grades one through twelve. Recommendations that seemed warranted were (1) recognition of the utility of a structural-developmental approach (2) acceptance of qualitative differences in stages of mental development (3) an incremental approach to the developmental tasks, (4) instructional design accommodating invariant stages of thought development: (5) involvement of instructors in affective and motivational teaching (6) exploration of new techniques and approaches and (7) replication of notions and techniques of the structural-developmental approach to the educational task of the rural community college