Optimal Blends of History and Intelligence for Robust Antiterrorism Policy

Abstract Antiterrorism analysis requires that security agencies blend evidence on historical patterns of terrorist behavior with incomplete intelligence on terrorist adversaries to predict possible terrorist operations and devise appropriate countermeasures. We model interactions between reactive, adaptive and intelligent adversaries embedded in minimally sufficient organizational settings to study the optimal analytic mixture, expressed as historical memory reach-back and the number of anticipatory scenarios, that should be used to design antiterrorism policy. We show that history is a valuable source of information when the terrorist organization evolves and acquires new capabilities at such a rapid pace that makes optimal strategies advocated by game-theoretic reasoning unlikely to succeed

Higher-order theory of mind is especially useful in unpredictable negotiations

In social interactions, people often reason about the beliefs, goals and intentions of others. This theory of mind allows them to interpret the behavior of others, and predict how they will behave in the future. People can also use this ability recursively: they use higher-order theory of mind to reason about the theory of mind abilities of others, as in "he thinks that I don’t know that he sent me an anonymous letter". Previous agent-based modeling research has shown that the usefulness of higher-order theory of mind reasoning can be useful across competitive, cooperative, and mixed-motive settings. In this paper, we cast a new light on these results by investigating how the predictability of the environment influences the effectiveness of higher-order theory of mind. Our results show that the benefit of (higher-order) theory of mind reasoning is strongly dependent on the predictability of the environment. We consider agent-based simulations in repeated one-shot negotiations in a particular negotiation setting known as Colored Trails. When this environment is highly predictable, agents obtain little benefit from theory of mind reasoning. However, if the environment has more observable features that change over time, agents without the ability to use theory of mind experience more difficulties predicting the behavior of others accurately. This in turn allows theory of mind agents to obtain higher scores in these more dynamic environments. These results suggest that the human-specific ability for higher-order theory of mind reasoning may have evolved to allow us to survive in more complex and unpredictable environments

Estimating the Use of Higher-Order Theory of Mind Using Computational Agents

When people make decisions in a social context, they often make use of theory of mind, by reasoning about unobservable mental content of others. For example, the behavior of a pedestrian who wants to cross the street depends on whether or not he believes that the driver of an oncoming car has seen him or not. People can also reason about the theory of mind abilities of others, leading to recursive thinking of the sort 'I think that you think that I think.'. Previous research suggests that this ability may be especially effective in simple competitive settings. In this paper, we use a combination of computational agents and Bayesian model selection to determine to what extent people make use of higher-order theory of mind reasoning in a particular competitive game known as matching pennies. We find that while many children and adults appear to make use of theory of mind, participants are also often classified as using a simpler reactive strategy based only on the actions of the directly preceding round. This may indicate that human reasoners do not primarily use their theory of mind abilities to compete with others

Rational Coordination in Multi-Agent Environments

We adopt the decision-theoretic principle of expected utility maximization as a paradigm for designing autonomous rational agents, and present a framework that uses this paradigm to determine the choice of coordinated action. We endow an agent with a specialized representation that captures the agent's knowledge about the environment and about the other agents, including its knowledge about their states of knowledge, which can include what they know about the other agents, and so on. This reciprocity leads to a recursive nesting of models. Our framework puts forth a representation for the recursive models and, under the assumption that the nesting of models is finite, uses dynamic programming to solve this representation for the agent's rational choice of action. Using a decision-theoretic approach, our work addresses concerns of agent decision-making about coordinated action in unpredictable situations, without imposing upon agents pre-designed prescriptions, or protocols, about standard rules of interaction. We implemented our method in a number of domains and we show results of coordination among our automated agents, among human-controlled agents, and among our agents coordinating with human-controlled agents.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/44002/1/10458_2004_Article_272540.pd

Fuzzy and tile coding approximation techniques for coevolution in reinforcement learning

PhDThis thesis investigates reinforcement learning algorithms suitable for learning in large state space problems and coevolution. In order to learn in large state spaces, the state space must be collapsed to a computationally feasible size and then generalised about. This thesis presents two new implementations of the classic temporal difference (TD) reinforcement learning algorithm Sarsa that utilise fuzzy logic principles for approximation, FQ Sarsa and Fuzzy Sarsa. The effectiveness of these two fuzzy reinforcement learning algorithms is investigated in the context of an agent marketplace. It presents a practical investigation into the design of fuzzy membership functions and tile coding schemas. A critical analysis of the fuzzy algorithms to a related technique in function approximation, a coarse coding approach called tile coding is given in the context of three different simulation environments; the mountain-car problem, a predator/prey gridworld and an agent marketplace. A further comparison between Fuzzy Sarsa and tile coding in the context of the nonstationary environments of the agent marketplace and predator/prey gridworld is presented. This thesis shows that the Fuzzy Sarsa algorithm achieves a significant reduction of state space over traditional Sarsa, without loss of the finer detail that the FQ Sarsa algorithm experiences. It also shows that Fuzzy Sarsa and gradient descent Sarsa(λ) with tile coding learn similar levels of distinction against a stationary strategy. Finally, this thesis demonstrates that Fuzzy Sarsa performs better in a competitive multiagent domain than the tile coding solution

Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems

Much research in artificial intelligence is concerned with the development of autonomous agents that can interact effectively with other agents. An important aspect of such agents is the ability to reason about the behaviours of other agents, by constructing models which make predictions about various properties of interest (such as actions, goals, beliefs) of the modelled agents. A variety of modelling approaches now exist which vary widely in their methodology and underlying assumptions, catering to the needs of the different sub-communities within which they were developed and reflecting the different practical uses for which they are intended. The purpose of the present article is to provide a comprehensive survey of the salient modelling methods which can be found in the literature. The article concludes with a discussion of open problems which may form the basis for fruitful future research.Comment: Final manuscript (46 pages), published in Artificial Intelligence Journal. The arXiv version also contains a table of contents after the abstract, but is otherwise identical to the AIJ version. Keywords: autonomous agents, multiagent systems, modelling other agents, opponent modellin

Modélisation de dialogues à l'aide d'un modèle Markovien caché

La modélisation de dialogue humain-machine est un domaine de recherche qui englobe plusieurs disciplines telles que la philosophie, les sciences cognitives et sociales, et l’informatique. Elle a pour but de reproduire la capacité humaine afin d’apprendre des stratégies optimales de dialogue. De plus, elle vise à concevoir et à évaluer des systèmes de gestion de dialogue ou d’étudier plus en détails la nature des conversations. Par ailleurs, peu de modèles de simulation de dialogues existants ont été jugé bons. Ce mémoire présente un modèle de Markov caché qui prédit l’action de l’utilisateur dans les systèmes de dialogue étant donné l’action du système précédente. L’apprentissage du modèle a été réalisé selon une approche d’apprentissage non supervisé en utilisant différentes méthodes de la validation croisée. Quant à l’évaluation du modèle, elle a été faite en utilisant différentes métriques. Les résultats de l’évaluation ont été en dessous des attentes mais tout de même satisfaisants par rapport aux travaux antérieurs. Par conséquent, des avenues de recherches futures seront proposées pour surpasser cette problématique. Mots-clés : traitement de la langue naturelle, dialogue oral homme-machine, modèle de Markov caché, apprentissage non supervisé, validation croisée.Modeling human-machine dialogue is a research area that encompasses several disciplines such as philosophy, computer science, as well as cognitive and social sciences. It aims to replicate the human ability to learn optimal strategies of dialogue. Furthermore, it aims to design and evaluate management systems for dialogue, and to study the nature of the conversations in more detail. Moreover, few simulation models of existing dialogues were considered good. This thesis presents a hidden Markov model that predicts the action of the user in dialogue systems on the basis of the previous system action. The learning model has been realized through an approach to unsupervised learning using different methods of cross validation. As for model evaluation, it has been done using different metrics. The evaluation results were below expectation. Nonetheless, they are satisfactory compared to previous work. Ultimately, avenues for future research are proposed to overcome this problem. Keywords: natural language processing, spoken dialogue human-machine, Hidden Markov Model (HMM), unsupervised learning, cross validation

Probabilistic modelling of oil rig drilling operations for business decision support: a real world application of Bayesian networks and computational intelligence.

This work investigates the use of evolved Bayesian networks learning algorithms based on computational intelligence meta-heuristic algorithms. These algorithms are applied to a new domain provided by the exclusive data, available to this project from an industry partnership with ODS-Petrodata, a business intelligence company in Aberdeen, Scotland. This research proposes statistical models that serve as a foundation for building a novel operational tool for forecasting the performance of rig drilling operations. A prototype for a tool able to forecast the future performance of a drilling operation is created using the obtained data, the statistical model and the experts' domain knowledge. This work makes the following contributions: applying K2GA and Bayesian networks to a real-world industry problem; developing a well-performing and adaptive solution to forecast oil drilling rig performance; using the knowledge of industry experts to guide the creation of competitive models; creating models able to forecast oil drilling rig performance consistently with nearly 80% forecast accuracy, using either logistic regression or Bayesian network learning using genetic algorithms; introducing the node juxtaposition analysis graph, which allows the visualisation of the frequency of nodes links appearing in a set of orderings, thereby providing new insights when analysing node ordering landscapes; exploring the correlation factors between model score and model predictive accuracy, and showing that the model score does not correlate with the predictive accuracy of the model; exploring a method for feature selection using multiple algorithms and drastically reducing the modelling time by multiple factors; proposing new fixed structure Bayesian network learning algorithms for node ordering search-space exploration. Finally, this work proposes real-world applications for the models based on current industry needs, such as recommender systems, an oil drilling rig selection tool, a user-ready rig performance forecasting software and rig scheduling tools