Ecological adaptation in the context of an actor-critic

Biological beings are the result of an evolutionary and developmental process of adaptation to the environment they perceive and where they act. Animals and plants have successfully adapted to a large variety of environments, which supports the ideal of inspiring artificial agents after biology and ethology. This idea has been already suggested by previous studies and is extended throughout this thesis. However, the role of perception in the process of adaptation and its integration in an agent capable of acting for survival is not clear.Robotic architectures in AI proposed throughout the last decade have broadly addressed the problems of behaviour selection, namely deciding "what to do next", and of learning as the two main adaptive processes. Behaviour selection has been commonly related to theories of motivation, and learning has been bound to theories of reinforcement. However, the formulation of a general theory including both processes as particular cases of the same phenomenon is still an incomplete task. This thesis focuses again on behaviour selection and learning; however it proposes to integrate both processes by stressing the ecological relationship between the agent and its environment. If the selection of behaviour is an expression of the agent's motivations, the feedback of the environment due to behaviour execution can be viewed as part of the same process, since it also influences the agent's internal motivations and the learning processes via reinforcement. I relate this to an argument supporting the existence of a common neural substrate to compute motivation and reward, and therefore relating the elicitation of a behaviour to the perception of reward resulting from its executionAs in previous studies, behaviour selection is viewed as a competition among parallel pathways to gain control over the agent's actuators. Unlike for the previous cases, the computation of every motivation in this thesis is not anymore the result of an additive or multiplicative formula combining inner and outer stimuli. Instead, the ecological principle is proposed to constrain the combination of stimuli in a novel fashion that leads to adaptive behavioural patterns. This method aims at overcoming the intrinsic limitations of any formula, the use of which results in behavioural responses restricted to a set of specific patterns, and therefore to the set of ethological cases they can justify. External stimuli and internal physiology in the model introduced in this thesis are not combined a priori. Instead, these are viewed from the perspective of the agent as modulatory elements biasing the selection of one behaviour over another guided by the reward provided by the environment, being the selection performed by an actor-critic reinforcement learning algorithm aiming at the maximum cumulative reward.In this context, the agent's drives are the expression of the deficit or excess of internal resources and the reference of the agent to define its relationship with the environment. The schema to learn object affordances is integrated in an actor-critic reinforcement learning algorithm, which is the core of a motivation and reinforcement framework driving behaviour selection and learning. Its working principle is based on the capacity of perceiving changes in the environment via internal hormonal responses and of modifying the agent's behavioural patterns accordingly. To this end, the concept of reward is defined in the framework of the agent's internal physiology and is related to the condition of physiological stability introduced by Ashby, and supported by Dawkins and Meyer as a requirement for survival. In this light, the definition of the reward used for learning is defined in the physiological state, where the effect of interacting with the environment can be quantified in an ethologically consistent manner.The above ideas on motivation, behaviour selection, learning and perception have been made explicit in an architecture integrated in an simulated robotic platform. To demonstrate the reach of their validity, extensive simulation has been performed to address the affordance learning paradigm and the adaptation offered by the framework of the actor-critic. To this end, three different metrics have been proposed to measure the effect of external and internal perception on the learning and behaviour selection processes: the performance in terms of flexibility of adaptation, the physiological stability and the cycles of behaviour execution at every situation. In addition to this, the thesis has begun to frame the integration of behaviours of an appetitive and consummatory nature in a single schema. Finally, it also contributes to the arguments disambiguating the role of dopamine as a neurotransmitter in the Basal Ganglia

The primacy of Knowing-how : cognition, know-how and an enactive action first epistemology

The Enactive Approach (EA) is a project of naturalization of the mind. EA should be able to offer a naturalization of knowledge, such underlying naturalization is what is found in this dissertation. The result is an epistemology where the most basal aspect of knowledge is not to accurately represent. For an enactive epistemology, the primary relation is how knowers relate, contact or engage with what is known. I argue in the final chapter that knowing is a perspectival, affectively entangled, historically situated relation between knower and known. Knower, known and knowing are characterized in broad naturalistic terms. EA is first presented in the context of a larger trend of studying cognition in an ecological way. The understanding of mind in the context of the living leads me to argue that living systems and precarious autonomous systems in general are intrinsically teleological systems whose defining activity consists in being responsive to the viability boundaries or conditions of their own existence. Cognitive systems skillfully change in adaptive manners to not disintegrate, even if their changes are not optimal. The account provides a relational account of adaptive behavior as the basis for an account of know-how. The more general notion of know-how can be articulated from the notion of perception as mastery of sensorimotor contingencies. Know how in general is understood as the organization and reorganization of bodily processes and structures that enables reliable successful action. Know-how as the bodily sensitivities and capabilities relative to the cognitive domain that reliably result in the success of action is a feature of all forms of cognitive engagement. The cognition or knowing-how of languaging consists in acquiring, producing, interpreting and modifying the know-how shared within linguistic communities. Crucially, the influence of the interactive context in a participant’s sense-making varies in a continuum of participation. In one end of the spectrum one finds sense-making that remains largely (but not absolutely) individual and in the other end where what characterizes the activity is a joint process of sense-making. Knowing-how to language is knowing-how to be in dialogue with plural and idiosyncratic identities while being both yourself. A shared community of practices emerges as the basis of objectivity; knowing-how is a communal affair. If cognition is the skillful and not necessarily optimal adaptation of a precarious systemic identity to an always changing environment, all cognition rests on know how. Cognition rests on know-how in the sense that all cognition is understood in terms of skillful transition between states of a system struggling with possible disintegration. Intelligent behavior is not based on symbolic structures and context-free knowledge, it is based on richly detailed, context-specific know-how. The knowledgeable interaction with the world is the responsiveness to the now that incorporates the history leading up to it.A Abordagem Enativa (AE) é um projeto de naturalização da mente. AE deveria ser capaz de oferecer uma naturalização do conhecimento, tal naturalização subjacente é o que apresento ao leitor nesta tese. O resultado é uma epistemologia na qual o aspecto mais básico do conhecimento não é representar acuradamente. Para uma epistemologia enativa, a relação privilegiada é como conhecedores se relacionam, entram em contato ou engajam com o que é conhecido. Argumento no capítulo final que o conhecimento é uma relação perspectival, afectivamente emaranhada, historicamente situada entre conhecedor e conhecido. Conhecedor, conhecido e conhecer são caracterizados em termos liberalmente naturalistas. AE é primeiro apresentada no contexto de uma ampla tendência de estudar-se ecologicamente a cognição. O entendimento da vida no contexto do vivo me leva a argumentar que sistemas vivos e sistemas autônomos precários em geral são teleológicos e sua atividade definidora consiste em ser responsivo às fronteiras de viabilidade de sua própria existência. Sistemas cognitivos habilidosamente mudam de modos adaptativos evitando a desintegração, mesmo que as mudanças não sejam optimais. A abordagem provê uma visão relacional do comportamento adaptativo como base para o conhecimento prático [know-how]. A noção mais geral de conhecimento prático pode ser elaborada a partir da noção de percepção como maestria de contingências sensório-motoras. Conhecimento prático em geral é compreendido como a organização e reorganização de processos e estruturas corporais que possibilita de modo confiável a ação bem-sucedida. Conhecimento prático como as sensibilidades e capacidades corporais para o confiável sucesso da ação é uma característica de todas as formas de engajamento cognitivo. A cognição ou o sabendo-fazer do lingueajear consiste em adquirir, produzir, interpretar e modificar o conhecimento prático compartilhado entre comunidades linguísticas. Crucialmente, a influência do contexto interativo na produção de sentido de um participante de uma comunidade varia em um contínuo de participação. Num extremo encontra-se produção de sentido que permanece majoritariamente (mas não absolutamente individual, e no outro encontra-se atividades caracterizadas como processos conjuntos de produção de sentido. Sabendo-fazer linguagem é saber como estar em diálogo com identidades plurais e idiossincráticas enquanto se é uma você mesmo. Uma comunidade de práticas compartilhadas emerge como a base da objetividade, saber-como é um assunto comunal. Se a cognição é a adaptação habilidosa e não necessariamente optimal de uma identidade sistêmica precária em um ambiente constantemente mudando, toda cognição apoia se em conhecimento prático. Cognição apoia-se em conhecimento prático na medida que toda cognição é entendida em termos da transição habilidosa entre estados de um sistema sob a possibilidade de desintegração Comportamento inteligente não é baseado em estruturas simbólicas e conhecimento geral, baseia-se em conhecimento prático ricamente detalhado e relevante ao contexto específico. A interação com o mundo dotada de conhecimento é a responsividade para o agora que incorpora a história que nos levou até aqui

Wavelength assignment in optical burst switching networks using neuro-dynamic programming

Cataloged from PDF version of article.All-optical networks are the most promising architecture for building large-size, hugebandwidth transport networks that are required for carrying the exponentially increasing Internet traffic. Among the existing switching paradigms in the literature, the optical burst switching is intended to leverage the attractive properties of optical communications, and at the same time, take into account its limitations. One of the major problems in optical burst switching is high blocking probability that results from one-way reservation protocol used. In this thesis, this problem is solved in wavelength domain by using smart wavelength assignment algorithms. Two heuristic wavelength assignment algorithms prioritizing available wavelengths according to reservation tables at the network nodes are proposed. The major contribution of the thesis is the formulation of the wavelength assignment problem as a continuous-time, average cost dynamic programming problem and its solution based on neuro-dynamic programming. Experiments are done over various traffic loads, burst lengths, and number of wavelength converters with a pool structure. The simulation results show that the wavelength assignment algorithms proposed for optical burst switching networks in the thesis perform better than the wavelength assignment algorithms in the literature that are developed for circuit-switched optical networks.Keçeli, FeyzaM.S

What to bid and when to stop

Negotiation is an important activity in human society, and is studied by various disciplines, ranging from economics and game theory, to electronic commerce, social psychology, and artificial intelligence. Traditionally, negotiation is a necessary, but also time-consuming and expensive activity. Therefore, in the last decades there has been a large interest in the automation of negotiation, for example in the setting of e-commerce. This interest is fueled by the promise of automated agents eventually being able to negotiate on behalf of human negotiators.Every year, automated negotiation agents are improving in various ways, and there is now a large body of negotiation strategies available, all with their unique strengths and weaknesses. For example, some agents are able to predict the opponent's preferences very well, while others focus more on having a sophisticated bidding strategy. The problem however, is that there is little incremental improvement in agent design, as the agents are tested in varying negotiation settings, using a diverse set of performance measures. This makes it very difficult to meaningfully compare the agents, let alone their underlying techniques. As a result, we lack a reliable way to pinpoint the most effective components in a negotiating agent.There are two major advantages of distinguishing between the different components of a negotiating agent's strategy: first, it allows the study of the behavior and performance of the components in isolation. For example, it becomes possible to compare the preference learning component of all agents, and to identify the best among them. Second, we can proceed to mix and match different components to create new negotiation strategies., e.g.: replacing the preference learning technique of an agent and then examining whether this makes a difference. Such a procedure enables us to combine the individual components to systematically explore the space of possible negotiation strategies.To develop a compositional approach to evaluate and combine the components, we identify structure in most agent designs by introducing the BOA architecture, in which we can develop and integrate the different components of a negotiating agent. We identify three main components of a general negotiation strategy; namely a bidding strategy (B), possibly an opponent model (O), and an acceptance strategy (A). The bidding strategy considers what concessions it deems appropriate given its own preferences, and takes the opponent into account by using an opponent model. The acceptance strategy decides whether offers proposed by the opponent should be accepted.The BOA architecture is integrated into a generic negotiation environment called Genius, which is a software environment for designing and evaluating negotiation strategies. To explore the negotiation strategy space of the negotiation research community, we amend the Genius repository with various existing agents and scenarios from literature. Additionally, we organize a yearly international negotiation competition (ANAC) to harvest even more strategies and scenarios. ANAC also acts as an evaluation tool for negotiation strategies, and encourages the design of negotiation strategies and scenarios.We re-implement agents from literature and ANAC and decouple them to fit into the BOA architecture without introducing any changes in their behavior. For each of the three components, we manage to find and analyze the best ones for specific cases, as described below. We show that the BOA framework leads to significant improvements in agent design by wining ANAC 2013, which had 19 participating teams from 8 international institutions, with an agent that is designed using the BOA framework and is informed by a preliminary analysis of the different components.In every negotiation, one of the negotiating parties must accept an offer to reach an agreement. Therefore, it is important that a negotiator employs a proficient mechanism to decide under which conditions to accept. When contemplating whether to accept an offer, the agent is faced with the acceptance dilemma: accepting the offer may be suboptimal, as better offers may still be presented before time runs out. On the other hand, accepting too late may prevent an agreement from being reached, resulting in a break off with no gain for either party. We classify and compare state-of-the-art generic acceptance conditions. We propose new acceptance strategies and we demonstrate that they outperform the other conditions. We also provide insight into why some conditions work better than others and investigate correlations between the properties of the negotiation scenario and the efficacy of acceptance conditions.Later, we adopt a more principled approach by applying optimal stopping theory to calculate the optimal decision on the acceptance of an offer. We approach the decision of whether to accept as a sequential decision problem, by modeling the bids received as a stochastic process. We determine the optimal acceptance policies for particular opponent classes and we present an approach to estimate the expected range of offers when the type of opponent is unknown. We show that the proposed approach is able to find the optimal time to accept, and improves upon all existing acceptance strategies.Another principal component of a negotiating agent's strategy is its ability to take the opponent's preferences into account. The quality of an opponent model can be measured in two different ways. One is to use the agent's performance as a benchmark for the model's quality. We evaluate and compare the performance of a selection of state-of-the-art opponent modeling techniques in negotiation. We provide an overview of the factors influencing the quality of a model and we analyze how the performance of opponent models depends on the negotiation setting. We identify a class of simple and surprisingly effective opponent modeling techniques that did not receive much previous attention in literature.The other way to measure the quality of an opponent model is to directly evaluate its accuracy by using similarity measures. We review all methods to measure the accuracy of an opponent model and we then analyze how changes in accuracy translate into performance differences. Moreover, we pinpoint the best predictors for good performance. This leads to new insights concerning how to construct an opponent model, and what we need to measure when optimizing performance.Finally, we take two different approaches to gain more insight into effective bidding strategies. We present a new classification method for negotiation strategies, based on their pattern of concession making against different kinds of opponents. We apply this technique to classify some well-known negotiating strategies, and we formulate guidelines on how agents should bid in order to be successful, which gives insight into the bidding strategy space of negotiating agents. Furthermore, we apply optimal stopping theory again, this time to find the concessions that maximize utility for the bidder against particular opponents. We show there is an interesting connection between optimal bidding and optimal acceptance strategies, in the sense that they are mirrored versions of each other.Lastly, after analyzing all components separately, we put the pieces back together again. We take all BOA components accumulated so far, including the best ones, and combine them all together to explore the space of negotiation strategies.We compute the contribution of each component to the overall negotiation result, and we study the interaction between components. We find that combining the best agent components indeed makes the strongest agents. This shows that the component-based view of the BOA architecture not only provides a useful basis for developing negotiating agents but also provides a useful analytical tool. By varying the BOA components we are able to demonstrate the contribution of each component to the negotiation result, and thus analyze the significance of each. The bidding strategy is by far the most important to consider, followed by the acceptance conditions and finally followed by the opponent model.Our results validate the analytical approach of the BOA framework to first optimize the individual components, and then to recombine them into a negotiating agent

The interactionist approach to virtue

In this dissertation, I took sides with virtue ethicists and argued that virtue is possible despite the mounting empirical evidence of how situational features impact human behavior. The main innovation I bring into the character debate is the idea that humans are creatures with various species-specific and socio-cultural constraints, and that this dimension should be integrated into theorizing about virtue. To do this, I extended and refined the concept of human limitations, to encompass not only natural disasters, as Aristotle did it, but also contain psychological and socio-cultural elements that impose limits to the way we see the social world and navigate it. Respectively, so was my argument, the idea of virtue should be refined as well, as an aspiration of creatures like us, and not those of heroes with a divine power or even half-gods. In a nutshell, I proposed to rethink three core concepts: moral failure, human limitations, and moral virtues

Investigations into controllers for adaptive autonomous agents based on artificial neural networks.

This thesis reports the development and study of novel architectures for the simulation of adaptive behaviour based on artificial neural networks. There are two distinct levels of enquiry. At the primary level, the initial aim was to design and implement a unified architecture integrating sensorimotor learning and overall control. This was intended to overcome shortcomings of typical behaviour-based approaches in reactive control settings. It was achieved in two stages. Initially, feedforward neural networks were used at the sensorimotor level of a modular architecture and overall control was provided by an algorithm. The algorithm was then replaced by a recurrent neural network. For training, a form of reinforcement learning was used. This posed an intriguing composite of the well-known action selection and credit assignment problems. The solution was demonstrated in two sets of simulation studies involving variants of each architecture. These studies also showed: firstly that the expected advantages over the standard behaviour-based approach were realised, and secondly that the new integrated architecture preserved these advantages, with the added value of a unified control approach. The secondary level of enquiry addressed the more foundational question of whether the choice of processing mechanism is critical if the simulation of adaptive behaviour is to progress much beyond the reactive stage in more than a trivial sense. It proceeded by way of a critique of the standard behaviourbased approach to make a positive assessment of the potential for recurrent neural networks to fill such a role. The findings were used to inform further investigations at the primary level of enquiry. These were based on a framework for the simulation of delayed response learning using supervised learning techniques. A further new architecture, based on a second-order recurrent neural network, was designed for this set of studies. It was then compared with existing architectures. Some interesting results are presented to indicate the appropriateness of the design and the potential of the approach, though limitations in the long run are not discounted

Fuzzy Computational Model for Emotion Regulation Based on Affect Control Theory

Emotion modeling is a multi-disciplinary problem that has managed to attract a great deal of research work spanned to a wide spectrum of scholarly areas starting at humanistic science fields passing through applied sciences and engineering and arriving at health care and wellbeing. Emotion research under the umbrella of IT and Computer Science was extensively successful with a handful of achievements especially in the last two decades. Affective Computing is an IT originated systematic research area that strives to best model emotions in a way that fits the needs for computer applications enriched with affective component. A comprehensive Affective Computing based system is made of three major components: a component for emotion detection, a component for emotion modeling, and finally a component to generating affective responses in artificial agents. The major focus of this dissertation is on developing efficient computational models for emotions. In fact most of the research works presented in this dissertation were focused on a sub problem of emotion modeling known as emotion regulation at which we strive to model the dynamics of changes in the emotional response levels of individuals as a result of the overt or covert situational changes. In this dissertation, several emotion related problems were addressed. Modeling the dynamics for emotion elicitation from a pure appraisal approach, investigating individualistic differences in emotional processes, and modeling emotion contagion as a type of social contagion phenomena are a few to name from those conducted research works. The main contribution of this dissertation was to propose a new computational model for the problem of emotion regulation that is based on Affect Control Theory. The new approach utilized a hybrid appraisal-dimensional architecture. By using a fuzzy modeling approach, the natural fuzziness in perceiving, representing and expressing emotions was effectively and efficiently addressed. Furthermore, the combination of automata framework with the concept of bipolar emotional channels used at the heart of the modeling processes of the proposed model has further contributed to promote the behavior of the model in order to exhibit an accepted degree of human-like affective behavior

A Review of Platforms for the Development of Agent Systems

Agent-based computing is an active field of research with the goal of building autonomous software of hardware entities. This task is often facilitated by the use of dedicated, specialized frameworks. For almost thirty years, many such agent platforms have been developed. Meanwhile, some of them have been abandoned, others continue their development and new platforms are released. This paper presents a up-to-date review of the existing agent platforms and also a historical perspective of this domain. It aims to serve as a reference point for people interested in developing agent systems. This work details the main characteristics of the included agent platforms, together with links to specific projects where they have been used. It distinguishes between the active platforms and those no longer under development or with unclear status. It also classifies the agent platforms as general purpose ones, free or commercial, and specialized ones, which can be used for particular types of applications.Comment: 40 pages, 2 figures, 9 tables, 83 reference

Approaches to multi-agent learning

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.Includes bibliographical references (leaves 165-171).Systems involving multiple autonomous entities are becoming more and more prominent. Sensor networks, teams of robotic vehicles, and software agents are just a few examples. In order to design these systems, we need methods that allow our agents to autonomously learn and adapt to the changing environments they find themselves in. This thesis explores ideas from game theory, online prediction, and reinforcement learning, tying them together to work on problems in multi-agent learning. We begin with the most basic framework for studying multi-agent learning: repeated matrix games. We quickly realize that there is no such thing as an opponent-independent, globally optimal learning algorithm. Some form of opponent assumptions must be necessary when designing multi-agent learning algorithms. We first show that we can exploit opponents that satisfy certain assumptions, and in a later chapter, we show how we can avoid being exploited ourselves. From this beginning, we branch out to study more complex sequential decision making problems in multi-agent systems, or stochastic games. We study environments in which there are large numbers of agents, and where environmental state may only be partially observable.(cont.) In fully cooperative situations, where all the agents receive a single global reward signal for training, we devise a filtering method that allows each individual agent to learn using a personal training signal recovered from this global reward. For non-cooperative situations, we introduce the concept of hedged learning, a combination of regret-minimizing algorithms with learning techniques, which allows a more flexible and robust approach for behaving in competitive situations. We show various performance bounds that can be guaranteed with our hedged learning algorithm, thus preventing our agent from being exploited by its adversary. Finally, we apply some of these methods to problems involving routing and node movement in a mobilized ad-hoc networking domain.by Yu-Han Chang.Ph.D