Learning to Avoid Risky Actions

When a reinforcement learning agent executes actions that can cause frequent damage to itself, it can learn, by using Q-learning, that these actions must not be executed again. However, there are other actions that do not cause damage frequently but only once in a while, for example, risky actions such as parachuting. These actions may imply punishment to the agent and, depending on its personality, it would be better to avoid them. Nevertheless, using the standard Q-learning algorithm, the agent is not able to learn to avoid them, because the result of these actions can be positive on average. In this article, an additional mechanism of Q-learning, inspired by the emotion of fear, is introduced in order to deal with those risky actions by considering the worst results. Moreover, there is a daring factor for adjusting the consideration of the risk. This mechanism is implemented on an autonomous agent living in a virtual environment. The results present the performance of the agent with different daring degrees.The funds provided by the Spanish Government through the project called “A New Approach to Social Robotics” (AROS), of MICINN (Ministry of Science and Innovation) and through the RoboCity2030-IICM project (S2009/DPI-1559), funded by Programas de Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of the EU

Learning Behaviors by an Autonomous Social Robot with Motivations

In this study, an autonomous social robot is living in a laboratory where it can interact with several items (people included). Its goal is to learn by itself the proper behaviors in order to maintain its well-being at as high a quality as possible. Several experiments have been conducted to test the performance of the system. The Object Q-Learning algorithm has been implemented in the robot as the learning algorithm. This algorithm is a variation of the traditional Q-Learning because it considers a reduced state space and collateral effects. The comparison of the performance of both algorithms is shown in the first part of the experiments. Moreover, two mechanisms intended to reduce the learning session durations have been included: Well-Balanced Exploration and Amplified Reward. Their advantages are justified in the results obtained in the second part of the experiments. Finally, the behaviors learned by our robot are analyzed. The resulting behaviors have not been preprogrammed. In fact, they have been learned by real interaction in the real world and are related to the motivations of the robot. These are natural behaviors in the sense that they can be easily understood by humans observing the robot.The authors gratefully acknowledge the funds provided by the Spanish Government through the project call "Aplicaciones de los robots sociales", DPI2011-26980 from the Spanish Ministry of Economy and Competitiveness.Publicad

An autonomous social robot in fear

Currently artificial emotions are being extensively used in robots. Most of these implementations are employed to display affective states. Nevertheless, their use to drive the robot's behavior is not so common. This is the approach followed by the authors in this work. In this research, emotions are not treated in general but individually. Several emotions have been implemented in a real robot, but in this paper, authors focus on the use of the emotion of fear as an adaptive mechanism to avoid dangerous situations. In fact, fear is used as a motivation which guides the behavior during specific circumstances. Appraisal of fear is one of the cornerstones of this work. A novel mechanism learns to identify the harmful circumstances which cause damage to the robot. Hence, these circumstances elicit the fear emotion and are known as fear releasers. In order to prove the advantages of considering fear in our decision making system, the robot's performance with and without fear are compared and the behaviors are analyzed. The robot's behaviors exhibited in relation to fear are natural, i.e., the same kind of behaviors can be observed on animals. Moreover, they have not been preprogrammed, but learned by real interactions in the real world. All these ideas have been implemented in a real robot living in a laboratory and interacting with several items and people.The funds have been provided by the Spanish Government through the project called "A new approach to social robotics" (AROS), of MICINN (Ministry of Science and Innovation) and through the RoboCity2030- II-CM project (S2009/DPI-1559), funded by Programas de Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of the EU

An Empirical Assessment of Users\u27 Information Security Protection Behavior towards Social Engineering Breaches

User behavior is one of the most significant information security risks. Information Security is all about being aware of who and what to trust and behaving accordingly. Due to technology becoming an integral part of nearly everything in people\u27s daily lives, the organization\u27s need for protection from security threats has continuously increased. Social engineering is the act of tricking a user into revealing information or taking action. One of the riskiest aspects of social engineering is that it depends mainly upon user errors and is not necessarily a technology shortcoming. User behavior should be one of the first apprehensions when it comes to social engineering. Unfortunately, there are few specific studies to understand factors that affect users\u27 information security protection behavior towards social engineering breaches. The focus of the information security literature is shifting from technology to user behavior in recent times. SETA (Security Education Training Awareness) program aids organizations in teaching their users about information security issues and expectations to prevent information security breaches. Information security policies depict the rules and regulations that everyone must follow utilizing an organization\u27s information technology resources. This research study used Protection Motivation Theory (PMT) combined with the SETA program and security policies to determine factors that affect users\u27 information security protection behavior towards social engineering breaches. This research study was an empirical and quantitative study to congregate data utilizing a web survey and PLS-SEM (Partial Least Squares Structural Equation Modeling) technique. As a result, the research study supported all three hypotheses associated with fear, including a positive impact of perceived severity on fear, perceived vulnerability on fear, and fear on protection motivation. Moreover, the research study substantiated the positive impact of perceived severity, perceived vulnerability, and response efficacy on protection motivation. Furthermore, the research study also confirmed the positive impact of protection motivation and the SETA program on protection behavior. The findings of this research study derived that, unswerving with the literature, social engineering has arisen as one of the biggest threats in information security. This research study explored factors impacting users\u27 information security protection behavior towards social engineering breaches. Support of all hypotheses for fear appeal is a substantial contribution in view of a lesser-researched fear appeal in preceding research using PMT. This research study provided the groundwork for encouraging and nurturing users\u27 information security protection behavior to prevent social engineering breaches. Finally, this research study contributes to the increasing phenomenon of social engineering in practice and future research

Bio-inspired decision making system for an autonomous social robot: the role of fear

Robotics is an emergent field which is currently in vogue. In the near future, many researchers anticipate the spread of robots coexisting with humans in the real world. This requires a considerable level of autonomy in robots. Moreover, in order to provide a proper interaction between robots and humans without technical knowledge, these robots must behave according to the social and cultural norms. This results in social robots with cognitive capabilities inspired by biological organisms such as humans or animals. The work presented in this dissertation tries to extend the autonomy of a social robot by implementing a biologically inspired decision making system which allows the robot to make its own decisions. Considering this kind of decision making system, the robot will not be considered as a slave any more, but as a partner. The decisionmaking systemis based on drives,motivations, emotions, and self-learning. According to psychological theories, drives are deficits of internal variables or needs (e.g. energy) and the urge to correct these deficits are the motivations (e.g. survival). Following a homeostatic approach, the goal of the robot is to satisfy its drives maintaining its necessities within an acceptable range, i.e. to keep the robot’s wellbeing as high as possible. The learning process provides the robot with the proper behaviors to cope with each motivation in order to achieve the goal. In this dissertation, emotions are individually treated following a functional approach. This means that, considering some of the different functions of emotions in animals or humans, each artificial emotion plays a different role. Happiness and sadness are employed during learning as the reward or punishment respectively, so they evaluate the performance of the robot. On the other hand, fear plays a motivational role, that is, it is considered as a motivation which impels the robot to avoid dangerous situations. The benefits of these emotions in a real robot are detailed and empirically tested. The robot decides its future actions based on what it has learned from previous experiences. Although the current context of this robot is limited to a laboratory, the social robot cohabits with humans in a potentially non-deterministic environment. The robot is endowed with a repertory of actions but, initially, it does not know what action to execute either when to do it. Actually, it has to learn the policy of behavior, i.e. what action to execute in different world configuration, that is, in every state, in order to satisfy the drive related to the highest motivation. Since the robot will be learning in a real environment interacting with several objects, it is desired to achieve the policy of behavior in an acceptable range of time. The learning process is performed using a variation of the well-known Q-Learning algorithm, the Object Q-Learning. By using this algorithm, the robot learns the value of every state-action pair through its interaction with the environment. This means, it learns the value that every action has in every possible state; the higher the value, the better the action is in that state. At the beginning of the learning process these values, called the Q values, can all be set to the same value, or some of them can be fixed to another value. In the first case, this implies that the robot will learn from scratch; in the second case, the robot has some kind of previous information about the action selection. These values are updated during the learning process. The emotion of fear is particularly studied. The generation process of this emotion (the appraisal) and the reactions to fear are really useful to endow the robot with an adaptive reliable mechanism of “survival”. This dissertation presents a social robot which benefits from a particular learning process of new releasers of fear, i.e. the capacity to identify new dangerous situations. In addition, by means of the decision making system, the robot learns different reactions to prevent danger according to different unpredictable events. In fact, these reactions to fear are quite similar to the fear reactions observed in nature. Another challenge is to design a solution for the decision making system in such a way that it is flexible enough to easily change the configuration or even apply it to different robots. Considering the bio-inspiration of this work, this research (and other related works) was born as a try to better understand the brain processes. It is the author’s hope that it sheds some light in the study of mental processes, in particular those which may lead to mental or cognitive disorders. -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------La robótica es un área emergente que actualmente se encuentra en boga. Muchos científicos pronostican que, en un futuro próximo, los robot cohabitarán con las personas en el mundo real. Para que esto llegue a suceder, se necesita que los robots tengan un nivel de autonomía considerable. Además, para que exista una interacción entre robots y personas sin conocimientos técnicos, estos robots deben comportarse de acuerdo a las normas sociales y culturales. Esto nos lleva a robots sociales con capacidades cognitivas inspiradas en organismos biológicos, como los humanos o los animales. El trabajo que se presenta en esta tesis pretende aumentar la autonomía de un robot social mediante la implementación de un sistema de toma de decisiones bioinspirado que permita a un robot tomar sus propias decisiones. Desde este punto de vista, el robot no se considerará más como un esclavo, sino como un compañero. El sistema de toma de decisiones está basado en necesidades (drives), motivaciones, emociones y auto-aprendizaje. De acuerdo a diversas teorías psicológicas, las necesidades son carencias o déficits de variables internas (por ejemplo, la energía) y el impulso para corregir estas necesidades son las motivaciones (como por ejemplo la supervivencia). Considerando un enfoque homeostático, el objetivo del robot es satisfacer sus carencias manteniéndolas en un nivel aceptable. Esto quiere decir que el bienestar del robot debe ser lo más alto posible. El proceso de aprendizaje permite al robot desarrollar el comportamiento necesario según las distintas motivaciones para lograr su objetivo. En esta tesis, las emociones son consideradas de forma individual desde un punto de vista funcional. Esto significa que, considerando las diferentes funciones de las emociones en animales y humanos, cada una de las emociones artificiales juega un papel diferente. Por un lado, la felicidad y la tristeza se usan durante el aprendizaje como refuerzo o castigo respectivamente y, por tanto, evaluan el comportamiento del robot. Por otro lado, el miedo juega un papel motivacional, es decir, es considerado como una motivación la cual “empuja” el robot a evitar las situaciones peligrosas. Los detalles y las ventajas de estas emociones en un robot real se muestran empíricamente a lo largo de este libro. El robot decide sus acciones futuras en base a lo que ha aprendido en experiencias pasadas. A pesar de que el contexto actual del robot está limitado a un laboratorio, el robot social cohabita con personas en un entorno potencialmente no-determinístico. El robot está equipado con un repertorio de acciones pero, inicialmente, no sabe qué acción ejecutar ni cuando hacerlo. De echo, tiene que aprender la política de comportamiento, esto es, qué acción ejecutar en diferentes configuraciones del mundo (en cada estado) para satisfacer la necesidad relacionada con la motivación más alta. Puesto que el robot aprende en un entorno real interaccionando con distintos objetos, es necesario que este aprendizaje se realice en un tiempo aceptable. El algoritmo de aprendizaje que se utiliza es una variación del conocido Q-Learning, el Object Q-Learning. Mediante este algoritmo el robot aprende el valor de cada par estadoacción a través de interacción con el entorno. Esto significa, que aprende el valor de cada acción in cada posible estado. Cuanto más alto sea el valor, mejor es la acción en ese estado. Al inicio del proceso de aprendizaje, estos valores, llamados valores Q, pueden tener todos el mismo valor o pueden pueden tener asignados distintos valores. En el primer caso, el robot no dispone de conocimientos previos; en el segundo, el robot dispone de cierta información sobre la acción a elegir. Estos valores serán actualizados durante el aprendizaje. La emoción de miedo es especialmente estudiada en esta tesis. La forma de generarse esta emoción (el appraisal) y las reacciones al miedo resultan realmente útiles a la hora de dotar al robot con un mecanismo de supervivencia adaptable y fiable. Esta tesis presenta un robot social que utiliza un proceso particular para el aprendizaje de nuevos “liberadores” del miedo, es decir, dispone de la capacidad de identificar nuevas situaciones peligrosas. Además, mediante el sistema de toma de decisiones, el robot aprende diferente reacciones para protegerse ante posibles daños causados por diversos eventos impredecibles. De echo, estas reacciones al miedo son bastante similares a las reacciones al miedo que se pueden observar en la naturaleza. Otro reto importante es el diseño de la solución: el sistema de toma de decisiones tiene que diseñarse de forma que sea suficientemente flexible para permitir cambiar fácilmente la configuración o incluso para aplicarse a distintos robots. Teniendo en cuenta el enfoque bioinspirado de este trabajo, esta investigación (y muchos otros trabajos relacionados) surge como un intento de entender un poco más lo que sucede en el cerebro. El autor espera que esta tesis pueda ayudar en el estudio de los procesos mentales, en particular aquellos que pueden llevar a desórdenes mentales o cognitivos