Learning control knowledge by observation in software agents

This thesis is the outcome of research on providing software agents with learning by observation capabilities. It presents an agent architecture that allows software agents to learn control knowledge by direct observation of the actions executed by expert agents while performing a task. The proposed architecture makes it possible for software agents to observe each other. It displays information that is essential for observation, such as the agent constituents and capabilities, the actions performed and the conditions holding for them. The displayed information is accessible to all agents that want to observe. The proposed approach combines two methods of learning from the observed data. The first one relies on the sequence which the actions were observed. The second one categorizes the information in the observed data and determines which set of categories the new problems belong. The two learning methods are incorporated into a learning process that covers all aspects of learning by observation such as the discovery and observation of experts, storage of the acquired information, learning and application of the acquired knowledge. The learning process also includes an evaluation of the agent’s progress which provides control over the decision to obtain new knowledge or apply the acquired knowledge to new problems. The process is extended with external feedback on the actions executed by the agent. The approach was tested on three different scenarios that show that learning by observation can be of key importance whenever agents sharing similar features want to learn from each other.Esta tese resulta da investigação da aplicação da aprendizagem por observação em agentes de software. A tese apresenta uma arquitetura que permite a agentes de software aprender mecanismos de controlo por observação direta das acções realizadas por agentes especialistas enquanto estes realizam uma tarefa. A arquitetura proposta permite que agentes de software se observem uns aos outros ao exibir informações que são essenciais para a observação, tais como os constituintes e as capacidades do agente, as ações realizadas e as condições existentes aquando a realização das mesmas. Esta informação é acessível a todos os agentes que queiram observar. A abordagem proposta combina dois métodos de aprendizagem. O primeiro baseia-se na sequência em que as ações foram observadas. O segundo categoriza a informação observada e determina o conjunto de categorias aos quais os novos problemas pertencem. Os dois métodos de aprendizagem são incorporados num processo de aprendizagem que cobre todos os aspetos da aprendizagem por observação tais como a descoberta e observação de especialistas, o armazenamento da informaçãao adquirida, a aprendizagem e a aplicação do conhecimento adquirido. O processo de aprendizagem inclui também uma avaliação do progresso do agente que controla a decisão de obter novo conhecimento ou de aplicar o conhecimento adquirido em novos problemas. O processo é alargado com feedback externo sobre as acões executadas. A abordagem foi testada em três diferentes cenários que mostram a importância da aprendizagem por observação em situações onde agentes que compartilham características semelhantes querem aprender uns com os outros.This thesis reports PhD research work, for the Doctoral Program on Information Science and Technology of ISCTE-Instituto Universitario de Lisboa. It is partially supported by Fundação para a Ciencia e a Tecnologia through the PhD Grant number SFRH/BD/44779/2008 and the Associated Laboratory number 12 - Instituto de Telecomunicações - PEst-OE/EEI/LA0008/2013

A Contextual Approach To Learning Collaborative Behavior Via Observation

This dissertation describes a novel technique to creating a simulated team of agents through observation. Simulated human teamwork can be used for a number of purposes, such as expert examples, automated teammates for training purposes and realistic opponents in games and training simulation. Current teamwork simulations require the team member behaviors be programmed into the simulation, often requiring a great deal of time and effort. None are able to observe a team at work and replicate the teamwork behaviors. Machine learning techniques for learning by observation and learning by demonstration have proven successful at observing behavior of humans or other software agents and creating a behavior function for a single agent. The research described here combines current research in teamwork simulations and learning by observation to effectively train a multi-agent system in effective team behavior. The dissertation describes the background and work by others as well as a detailed description of the learning method. A prototype built to evaluate the developed approach as well as the extensive experimentation conducted is also described

INVESTIGATING AGENT AND TASK OPENNESS IN ADHOC TEAM FORMATION

When deciding which ad hoc team to join, agents are often required to consider rewards from accomplishing tasks as well as potential benefits from learning when working with others, when solving tasks. We argue that, in order to decide when to learn or when to solve task, agents have to consider the existing agents’ capabilities and tasks available in the environment, and thus agents have to consider agent and task openness—the rate of new, previously unknown agents (and tasks) that are introduced into the environment. We further assume that agents evolve their capabilities intrinsically through learning by observation or learning by doing when working in a team. Thus, an agent will need to consider which task to do or which team to join would provide the best situation for such learning to occur. In this thesis, we develop an auction-based multiagent simulation framework, a mechanism to simulate openness in our environment, and conduct comprehensive experiments to investigate the impact of agent and task openness. We propose several agent task selection strategies to leverage the environmental openness. Furthermore, we present a multiagent solution for agent-based collaborative human task assignment when finding suitable tasks for users in complex environments is made especially challenging by agent openness and task openness. Using an auction-based protocol to fairly assign tasks, software agents model uncertainty in the outcomes of bids caused by openness, then acquire tasks for people that maximize both the user’s utility gain and learning opportunities for human users (who improve their abilities to accomplish future tasks through learning by experience and by observing more capable humans). Experimental results demonstrate the effects of agent and task openness on collaborative task assignment, the benefits of reasoning about openness, and the value of non-myopically choosing tasks to help people improve their abilities for uncertain future tasks

A plan classifier based on Chi-square distribution tests

To make good decisions in a social context, humans often need to recognize the plan underlying the behavior of others, and make predictions based on this recognition. This process, when carried out by software agents or robots, is known as plan recognition, or agent modeling. Most existing techniques for plan recognition assume the availability of carefully hand-crafted plan libraries, which encode the a-priori known behavioral repertoire of the observed agents; during run-time, plan recognition algorithms match the observed behavior of the agents against the plan-libraries, and matches are reported as hypotheses. Unfortunately, techniques for automatically acquiring plan-libraries from observations, e.g., by learning or data-mining, are only beginning to emerge. We present an approach for automatically creating the model of an agent behavior based on the observation and analysis of its atomic behaviors. In this approach, observations of an agent behavior are transformed into a sequence of atomic behaviors (events). This stream is analyzed in order to get the corresponding behavior model, represented by a distribution of relevant events. Once the model has been created, the proposed approach presents a method using a statistical test for classifying an observed behavior. Therefore, in this research, the problem of behavior classification is examined as a problem of learning to characterize the behavior of an agent in terms of sequences of atomic behaviors. The experiment results of this paper show that a system based on our approach can efficiently recognize different behaviors in different domains, in particular UNIX command-line data, and RoboCup soccer simulationThis work has been partially supported by the Spanish Government under project TRA2007-67374-C02-0

Web-based learning through mixed-initiative interactions : design and implimentation

Mixed-initiative interaction is a naturally-occurring feature of human-human interactions. It characterize by turn-taking, frequent change of focus, agenda and control among the “speakers”. This human-based mixed-initiative interaction can be implemented through a mixed-initiative systems which are a popular approach to building intelligent systems that can collaborate naturally and effectively with people. Mixed-initiative systems exhibit various degrees of involvement in regards to the initiatives taken by the user or the system. In any discourse, the initiative may be shared between either, a learner and a system agent, or between two independent system agents. Both the parties in question establish and maintain a common goal and context, and proceed with an interaction mechanism involving initiative taking that optimizes their progress towards the goal. However, the application of mixed-initiative interaction in web-based learning is very much limited. In this paper, we discuss the design and implementation of a web-based learning system through mixedinitiative system known as JavaLearn. JavaLearn allows the interaction between the system (in the form of a software agent) and the individual learner. Here, the system supports the learning through a problem solving activity by demanding active learning behaviour from the learner with minimal natural language understanding by the agent and embodies the application-dependent aspects of the discourse. It guides the learner to solve the problem by giving adaptive advice, hints and engage the learner in the real time interaction in the form of “conversation”. The principal features of this system are: It is adaptive and are based on reflection, observation and relation. The system acquires its intelligence through the finite state machine and rule-based agents. (Abstract by authors