A POMDP approach to the hide and seek game

Projecte final de Màster Oficial fet en col.laboració amb Institut de Robàtica i Informàtica IndustrialPartially observable Markov decision processes (POMDPs) provide an elegant mathematical framework for modeling complex decision and planning problems in uncertain and dynamic environments. They have been successfully applied to various robotic tasks. The modeling advantage of POMDPs, however, comes at a price exact methods for solving them are computationally very expensive and thus applicable in practice only to simple problems. A major challenge is to scale up POMDP algorithms for more complex robotic systems. Our goal is to make an autonomous mobile robot to learn and play the children's game hide and seek with opponent a human agent. Motion planning in uncertain and dynamic envi- ronments is an essential capability for autonomous robots. We focus on an e cient point-based POMDP algorithm, SARSOP, that exploits the notion of optimally reachable belief spaces to improve computational efficiency. Moreover we explore the mixed observability MDPs (MOMDPs) model, a special class of POMDPs. Robotic systems often have mixed observability: even when a robots state is not fully observable, some components of the state may still be fully observable. Ex- ploiting this, we use the factored model, proposed in the literature, to represent separately the fully and partially observable components of a robots state and derive a compact lower dimensional representation of its belief space. We then use this factored representation in conjunction with the point-based algorithm to com- pute approximate POMDP solutions. Experiments show that on our problem, the new algorithm is many times faster than a leading point-based POMDP algorithm without important losses in the quality of the solutio

A game-theoretic analysis of DoS attacks on driverless vehicles

Driverless vehicles are expected to form the foundation of future connected transport infrastructure. A key weakness of connected vehicles is their vulnerability to physical-proximity attacks such as sensor saturation attacks. It is natural to study whether such attacks can be used to disrupt swarms of autonomous vehicles used as part of a large fleet providing taxi and courier delivery services. In this paper, we start to examine the strategic options available to attackers and defenders (autonomous-fleet operators) in such conflicts. We find that attackers have the upper hand in most cases and are able to carry out crippling denial-of-service attacks on fleets, by leveraging the inherent deficiencies of road networks identified by techniques from graph analysis. Experimental results on ten cities using real-world courier traces shows that most cities will require upgraded infrastructure to defend driverless vehicles against denial-of-service attacks. We found several hidden costs that impact equipment designers and operators of driverless vehicles - not least, that road-networks need to be redesigned for robustness against attacks thus raising some fundamental questions about the benefits

Towards Generalist Robots through Visual World Modeling

Moving from narrow robots specializing in specific tasks to generalist robots excelling in multiple tasks in various environmental conditions is the future of next-generation robotics. The key to generalist robots is the ability to learn world models that are reusable, generalizable, and adaptable. Having a general understanding of how the physical world works will enable robots to acquire transferable knowledge across different tasks, predict possible outcomes of future actions before execution, and constantly update their knowledge through continual interactions. While the majority of robot learning frameworks tend to mix task-related and task-agnostic components altogether throughout the learning process, these two components are often not intertwined when one of them is changed. For example, a task-agnostic component such as the computational model of the robot body remains the same even under different task settings, while a task-related component such as the dynamics of a moving object remains the same for different embodiments. This thesis studies the key steps towards building generalist robots by decomposing the world modeling problem into task-agnostic and task-related elements: (1) robot self-modeling; (2) robot modeling other agents; and (3) robot modeling the physical environment. This framework has produced powerful and efficient learning-based robotic systems for a variety of tasks and physical embodiments, such as computational models of physical robots that can be reused and adapted to numerous task objectives and changing environments, behavior modeling frameworks for complex multi-robot applications, and dynamical system understanding algorithms to distill compact physics knowledge from high-dimensional and multi-modal sensory data. The approach in this thesis could help catalyze the understanding, prediction, and control of increasingly complex systems

Communication between nodes for autonomic and distributed management

Doutoramento conjunto MAPi em InformáticaOver the last decade, the most widespread approaches for traditional management were based on the Simple Network Management Protocol (SNMP) or Common Management Information Protocol (CMIP). However, they both have several problems in terms of scalability, due to their centralization characteristics. Although the distributed management approaches exhibit better performance in terms of scalability, they still underperform regarding communication costs, autonomy, extensibility, exibility, robustness, and cooperation between network nodes. The cooperation between network nodes normally requires excessive overheads for synchronization and dissemination of management information in the network. For emerging dynamic and large-scale networking environments, as envisioned in Next Generation Networks (NGNs), exponential growth in the number of network devices and mobile communications and application demands is expected. Thus, a high degree of management automation is an important requirement, along with new mechanisms that promote it optimally and e ciently, taking into account the need for high cooperation between the nodes. Current approaches for self and autonomic management allow the network administrator to manage large areas, performing fast reaction and e ciently facing unexpected problems. The management functionalities should be delegated to a self-organized plane operating within the network, that decrease the network complexity and the control information ow, as opposed to centralized or external servers. This Thesis aims to propose and develop a communication framework for distributed network management which integrates a set of mechanisms for initial communication, exchange of management information, network (re) organization and data dissemination, attempting to meet the autonomic and distributed management requirements posed by NGNs. The mechanisms are lightweight and portable, and they can operate in di erent hardware architectures and include all the requirements to maintain the basis for an e cient communication between nodes in order to ensure autonomic network management. Moreover, those mechanisms were explored in diverse network conditions and events, such as device and link errors, di erent tra c/network loads and requirements. The results obtained through simulation and real experimentation show that the proposed mechanisms provide a lower convergence time, smaller overhead impact in the network, faster dissemination of management information, increase stability and quality of the nodes associations, and enable the support for e cient data information delivery in comparison to the base mechanisms analyzed. Finally, all mechanisms for communication between nodes proposed in this Thesis, that support and distribute the management information and network control functionalities, were devised and developed to operate in completely decentralized scenarios.Durante a última década, protocolos como Simple Network Management Protocol (SNMP) ou Common Management Information Protocol (CMIP) foram as abordagens mais comuns para a gestão tradicional de redes. Essas abordagens têm vários problemas em termos de escalabilidade, devido às suas características de centralização. Apresentando um melhor desempenho em termos de escalabilidade, as abordagens de gestão distribuída, por sua vez, são vantajosas nesse sentido, mas também apresentam uma série de desvantagens acerca do custo elevado de comunicação, autonomia, extensibilidade, exibilidade, robustez e cooperação entre os nós da rede. A cooperação entre os nós presentes na rede é normalmente a principal causa de sobrecarga na rede, uma vez que necessita de colectar, sincronizar e disseminar as informações de gestão para todos os nós nela presentes. Em ambientes dinâmicos, como é o caso das redes atuais e futuras, espera-se um crescimento exponencial no número de dispositivos, associado a um grau elevado de mobilidade dos mesmos na rede. Assim, o grau elevado de funções de automatiza ção da gestão da rede é uma exigência primordial, bem como o desenvolvimento de novos mecanismos e técnicas que permitam essa comunicação de forma optimizada e e ciente. Tendo em conta a necessidade de elevada cooperação entre os elementos da rede, as abordagens atuais para a gestão autonómica permitem que o administrador possa gerir grandes áreas de forma rápida e e ciente frente a problemas inesperados, visando diminuir a complexidade da rede e o uxo de informações de controlo nela gerados. Nas gestões autonómicas a delegação de operações da rede é suportada por um plano auto-organizado e não dependente de servidores centralizados ou externos. Com base nos tipos de gestão e desa os acima apresentados, esta Tese tem como principal objetivo propor e desenvolver um conjunto de mecanismos necessários para a criação de uma infra-estrutura de comunicação entre nós, na tentativa de satisfazer as exigências da gestão auton ómica e distribuída apresentadas pelas redes de futura geração. Nesse sentido, mecanismos especí cos incluindo inicialização e descoberta dos elementos da rede, troca de informação de gestão, (re) organização da rede e disseminação de dados foram elaborados e explorados em diversas condições e eventos, tais como: falhas de ligação, diferentes cargas de tráfego e exigências de rede. Para além disso, os mecanismos desenvolvidos são leves e portáveis, ou seja, podem operar em diferentes arquitecturas de hardware e contemplam todos os requisitos necessários para manter a base de comunicação e ciente entre os elementos da rede. Os resultados obtidos através de simulações e experiências reais comprovam que os mecanismos propostos apresentam um tempo de convergência menor para descoberta e troca de informação, um menor impacto na sobrecarga da rede, disseminação mais rápida da informação de gestão, aumento da estabilidade e a qualidade das ligações entre os nós e entrega e ciente de informações de dados em comparação com os mecanismos base analisados. Finalmente, todos os mecanismos desenvolvidos que fazem parte da infrastrutura de comunicação proposta foram concebidos e desenvolvidos para operar em cenários completamente descentralizados

Optimal search in discrete locations:extensions and new findings

A hidden target needs to be found by a searcher in many real-life situations, some of which involve large costs and significant consequences with failure. Therefore, efficient search methods are paramount. In our search model, the target lies in one of several discrete locations according to some hiding distribution, and the searcher's goal is to discover the target in minimum expected time by making successive searches of individual locations. In Part I of the thesis, the searcher knows the hiding distribution. Here, if there is only one way to search each location, the solution to the search problem, discovered in the 1960s, is simple; search next any location with a maximal probability per unit time of detecting the target. An equivalent solution is derived by viewing the search problem as a multi-armed bandit and following a Gittins index policy. Motivated by modern search technology, we introduce two modes---fast and slow---to search each location. The fast mode takes less time, but the slow mode is more likely to find the target. An optimal policy is difficult to obtain in general, because it requires an optimal sequence of search modes for each location, in addition to a set of sequence-dependent Gittins indices for choosing between locations. For each mode, we identify a sufficient condition for a location to use only that search mode in an optimal policy. For locations meeting neither sufficient condition, an optimal choice of search mode is extremely complicated, depending both on the hiding distribution and the search parameters of the other locations. We propose several heuristic policies motivated by our analysis, and demonstrate their near-optimal performance in an extensive numerical study. In Part II of the thesis, the searcher has only one search mode per location, but does not know the hiding distribution, which is chosen by an intelligent hider who aims to maximise the expected time until the target is discovered. Such a search game, modelled via two-person, zero-sum game theory, is relevant if the target is a bomb, intruder, or, of increasing importance due to advances in technology, a computer hacker. By Part I, if the hiding distribution is known, an optimal counter strategy for the searcher is any corresponding Gittins index policy. To develop an optimal search strategy in the search game, the searcher must account for the hider’s motivation to choose an optimal hiding distribution, and consider the set of corresponding Gittins index policies. %It follows that an optimal search strategy in the search game must be some Gittins index policy if the hiding distribution is assumed to be chosen optimally by the hider. However, the searcher must choose carefully from this set of Gittins index policies to ensure the same expected time to discover the target regardless of where it is hidden by the hider. %It follows that an optimal search strategy in the search game must be a Gittins index policy applied to a hiding distribution which is optimal from the hider's perspective. However, to avoid giving the hider any advantage, the searcher must carefully choose such a Gittins index policy among the many available. As a result, finding an optimal search strategy, or even proving one exists, is difficult. We extend several results for special cases from the literature to the fully-general search game; in particular, we show an optimal search strategy exists and may take a simple form. Using a novel test, we investigate the frequency of the optimality of a particular hiding strategy that gives the searcher no preference over any location at the beginning of the search