Aprendizagem de coordenação em sistemas multi-agente

The ability for an agent to coordinate with others within a system is a valuable property in multi-agent systems. Agents either cooperate as a team to accomplish a common goal, or adapt to opponents to complete different goals without being exploited. Research has shown that learning multi-agent coordination is significantly more complex than learning policies in singleagent environments, and requires a variety of techniques to deal with the properties of a system where agents learn concurrently. This thesis aims to determine how can machine learning be used to achieve coordination within a multi-agent system. It asks what techniques can be used to tackle the increased complexity of such systems and their credit assignment challenges, how to achieve coordination, and how to use communication to improve the behavior of a team. Many algorithms for competitive environments are tabular-based, preventing their use with high-dimension or continuous state-spaces, and may be biased against specific equilibrium strategies. This thesis proposes multiple deep learning extensions for competitive environments, allowing algorithms to reach equilibrium strategies in complex and partially-observable environments, relying only on local information. A tabular algorithm is also extended with a new update rule that eliminates its bias against deterministic strategies. Current state-of-the-art approaches for cooperative environments rely on deep learning to handle the environment’s complexity and benefit from a centralized learning phase. Solutions that incorporate communication between agents often prevent agents from being executed in a distributed manner. This thesis proposes a multi-agent algorithm where agents learn communication protocols to compensate for local partial-observability, and remain independently executed. A centralized learning phase can incorporate additional environment information to increase the robustness and speed with which a team converges to successful policies. The algorithm outperforms current state-of-the-art approaches in a wide variety of multi-agent environments. A permutation invariant network architecture is also proposed to increase the scalability of the algorithm to large team sizes. Further research is needed to identify how can the techniques proposed in this thesis, for cooperative and competitive environments, be used in unison for mixed environments, and whether they are adequate for general artificial intelligence.A capacidade de um agente se coordenar com outros num sistema é uma propriedade valiosa em sistemas multi-agente. Agentes cooperam como uma equipa para cumprir um objetivo comum, ou adaptam-se aos oponentes de forma a completar objetivos egoístas sem serem explorados. Investigação demonstra que aprender coordenação multi-agente é significativamente mais complexo que aprender estratégias em ambientes com um único agente, e requer uma variedade de técnicas para lidar com um ambiente onde agentes aprendem simultaneamente. Esta tese procura determinar como aprendizagem automática pode ser usada para encontrar coordenação em sistemas multi-agente. O documento questiona que técnicas podem ser usadas para enfrentar a superior complexidade destes sistemas e o seu desafio de atribuição de crédito, como aprender coordenação, e como usar comunicação para melhorar o comportamento duma equipa. Múltiplos algoritmos para ambientes competitivos são tabulares, o que impede o seu uso com espaços de estado de alta-dimensão ou contínuos, e podem ter tendências contra estratégias de equilíbrio específicas. Esta tese propõe múltiplas extensões de aprendizagem profunda para ambientes competitivos, permitindo a algoritmos atingir estratégias de equilíbrio em ambientes complexos e parcialmente-observáveis, com base em apenas informação local. Um algoritmo tabular é também extendido com um novo critério de atualização que elimina a sua tendência contra estratégias determinísticas. Atuais soluções de estado-da-arte para ambientes cooperativos têm base em aprendizagem profunda para lidar com a complexidade do ambiente, e beneficiam duma fase de aprendizagem centralizada. Soluções que incorporam comunicação entre agentes frequentemente impedem os próprios de ser executados de forma distribuída. Esta tese propõe um algoritmo multi-agente onde os agentes aprendem protocolos de comunicação para compensarem por observabilidade parcial local, e continuam a ser executados de forma distribuída. Uma fase de aprendizagem centralizada pode incorporar informação adicional sobre ambiente para aumentar a robustez e velocidade com que uma equipa converge para estratégias bem-sucedidas. O algoritmo ultrapassa abordagens estado-da-arte atuais numa grande variedade de ambientes multi-agente. Uma arquitetura de rede invariante a permutações é também proposta para aumentar a escalabilidade do algoritmo para grandes equipas. Mais pesquisa é necessária para identificar como as técnicas propostas nesta tese, para ambientes cooperativos e competitivos, podem ser usadas em conjunto para ambientes mistos, e averiguar se são adequadas a inteligência artificial geral.Apoio financeiro da FCT e do FSE no âmbito do III Quadro Comunitário de ApoioPrograma Doutoral em Informátic

Enabling technologies and cyber-physical systems for mission-critical scenarios

Programa Oficial de Doutoramento en Tecnoloxías da Información e Comunicacións en Redes Móbiles . 5029P01[Abstract] Reliable transport systems, defense, public safety and quality assurance in the Industry 4.0 are essential in a modern society. In a mission-critical scenario, a mission failure would jeopardize human lives and put at risk some other assets whose impairment or loss would significantly harm society or business results. Even small degradations of the communications supporting the mission could have large and possibly dire consequences. On the one hand, mission-critical organizations wish to utilize the most modern, disruptive and innovative communication systems and technologies, and yet, on the other hand, need to comply with strict requirements, which are very different to those of non critical scenarios. The aim of this thesis is to assess the feasibility of applying emerging technologies like Internet of Things (IoT), Cyber-Physical Systems (CPS) and 4G broadband communications in mission-critical scenarios along three key critical infrastructure sectors: transportation, defense and public safety, and shipbuilding. Regarding the transport sector, this thesis provides an understanding of the progress of communications technologies used for railways since the implantation of Global System for Mobile communications-Railways (GSM-R). The aim of this work is to envision the potential contribution of Long Term Evolution (LTE) to provide additional features that GSM-R would never support. Furthermore, the ability of Industrial IoT for revolutionizing the railway industry and confront today's challenges is presented. Moreover, a detailed review of the most common flaws found in Radio Frequency IDentification (RFID) based IoT systems is presented, including the latest attacks described in the literature. As a result, a novel methodology for auditing security and reverse engineering RFID communications in transport applications is introduced. The second sector selected is driven by new operational needs and the challenges that arise from modern military deployments. The strategic advantages of 4G broadband technologies massively deployed in civil scenarios are examined. Furthermore, this thesis analyzes the great potential for applying IoT technologies to revolutionize modern warfare and provide benefits similar to those in industry. It identifies scenarios where defense and public safety could leverage better commercial IoT capabilities to deliver greater survivability to the warfighter or first responders, while reducing costs and increasing operation efficiency and effectiveness. The last part is devoted to the shipbuilding industry. After defining the novel concept of Shipyard 4.0, how a shipyard pipe workshop works and what are the requirements for building a smart pipe system are described in detail. Furthermore, the foundations for enabling an affordable CPS for Shipyards 4.0 are presented. The CPS proposed consists of a network of beacons that continuously collect information about the location of the pipes. Its design allows shipyards to obtain more information on the pipes and to make better use of it. Moreover, it is indicated how to build a positioning system from scratch in an environment as harsh in terms of communications as a shipyard, showing an example of its architecture and implementation.[Resumen] En la sociedad moderna, los sistemas de transporte fiables, la defensa, la seguridad pública y el control de la calidad en la Industria 4.0 son esenciales. En un escenario de misión crítica, el fracaso de una misión pone en peligro vidas humanas y en riesgo otros activos cuyo deterioro o pérdida perjudicaría significativamente a la sociedad o a los resultados de una empresa. Incluso pequeñas degradaciones en las comunicaciones que apoyan la misión podrían tener importantes y posiblemente terribles consecuencias. Por un lado, las organizaciones de misión crítica desean utilizar los sistemas y tecnologías de comunicación más modernos, disruptivos e innovadores y, sin embargo, deben cumplir requisitos estrictos que son muy diferentes a los relativos a escenarios no críticos. El objetivo principal de esta tesis es evaluar la viabilidad de aplicar tecnologías emergentes como Internet of Things (IoT), Cyber-Physical Systems (CPS) y comunicaciones de banda ancha 4G en escenarios de misión crítica en tres sectores clave de infraestructura crítica: transporte, defensa y seguridad pública, y construcción naval. Respecto al sector del transporte, esta tesis permite comprender el progreso de las tecnologías de comunicación en el ámbito ferroviario desde la implantación de Global System for Mobile communications-Railway (GSM-R). El objetivo de este trabajo es analizar la contribución potencial de Long Term Evolution (LTE) para proporcionar características adicionales que GSM-R nunca podría soportar. Además, se presenta la capacidad de la IoT industrial para revolucionar la industria ferroviaria y afrontar los retos actuales. Asimismo, se estudian con detalle las vulnerabilidades más comunes de los sistemas IoT basados en Radio Frequency IDentification (RFID), incluyendo los últimos ataques descritos en la literatura. Como resultado, se presenta una metodología innovadora para realizar auditorías de seguridad e ingeniería inversa de las comunicaciones RFID en aplicaciones de transporte. El segundo sector elegido viene impulsado por las nuevas necesidades operacionales y los desafíos que surgen de los despliegues militares modernos. Para afrontarlos, se analizan las ventajas estratégicas de las tecnologías de banda ancha 4G masivamente desplegadas en escenarios civiles. Asimismo, esta tesis analiza el gran potencial de aplicación de las tecnologías IoT para revolucionar la guerra moderna y proporcionar beneficios similares a los alcanzados por la industria. Se identifican escenarios en los que la defensa y la seguridad pública podrían aprovechar mejor las capacidades comerciales de IoT para ofrecer una mayor capacidad de supervivencia al combatiente o a los servicios de emergencias, a la vez que reduce los costes y aumenta la eficiencia y efectividad de las operaciones. La última parte se dedica a la industria de construcción naval. Después de definir el novedoso concepto de Astillero 4.0, se describe en detalle cómo funciona el taller de tubería de astillero y cuáles son los requisitos para construir un sistema de tuberías inteligentes. Además, se presentan los fundamentos para posibilitar un CPS asequible para Astilleros 4.0. El CPS propuesto consiste en una red de balizas que continuamente recogen información sobre la ubicación de las tuberías. Su diseño permite a los astilleros obtener más información sobre las tuberías y hacer un mejor uso de las mismas. Asimismo, se indica cómo construir un sistema de posicionamiento desde cero en un entorno tan hostil en términos de comunicaciones, mostrando un ejemplo de su arquitectura e implementación

Testing of Materials and Elements in Civil Engineering

This book was proposed and organized as a means to present recent developments in the field of testing of materials and elements in civil engineering. For this reason, the articles highlighted in this editorial relate to different aspects of testing of different materials and elements in civil engineering, from building materials to building structures. The current trend in the development of testing of materials and elements in civil engineering is mainly concerned with the detection of flaws and defects in concrete elements and structures, and acoustic methods predominate in this field. As in medicine, the trend is towards designing test equipment that allows one to obtain a picture of the inside of the tested element and materials. Interesting results with significance for building practices were obtained

Sistemas automáticos de informação e segurança para apoio na condução de veículos

Doutoramento em Engenharia MecânicaO objeto principal desta tese é o estudo de algoritmos de processamento e representação automáticos de dados, em particular de informação obtida por sensores montados a bordo de veículos (2D e 3D), com aplicação em contexto de sistemas de apoio à condução. O trabalho foca alguns dos problemas que, quer os sistemas de condução automática (AD), quer os sistemas avançados de apoio à condução (ADAS), enfrentam hoje em dia. O documento é composto por duas partes. A primeira descreve o projeto, construção e desenvolvimento de três protótipos robóticos, incluindo pormenores associados aos sensores montados a bordo dos robôs, algoritmos e arquitecturas de software. Estes robôs foram utilizados como plataformas de ensaios para testar e validar as técnicas propostas. Para além disso, participaram em várias competições de condução autónoma tendo obtido muito bons resultados. A segunda parte deste documento apresenta vários algoritmos empregues na geração de representações intermédias de dados sensoriais. Estes podem ser utilizados para melhorar técnicas já existentes de reconhecimento de padrões, deteção ou navegação, e por este meio contribuir para futuras aplicações no âmbito dos AD ou ADAS. Dado que os veículos autónomos contêm uma grande quantidade de sensores de diferentes naturezas, representações intermédias são particularmente adequadas, pois podem lidar com problemas relacionados com as diversas naturezas dos dados (2D, 3D, fotométrica, etc.), com o carácter assíncrono dos dados (multiplos sensores a enviar dados a diferentes frequências), ou com o alinhamento dos dados (problemas de calibração, diferentes sensores a disponibilizar diferentes medições para um mesmo objeto). Neste âmbito, são propostas novas técnicas para a computação de uma representação multi-câmara multi-modal de transformação de perspectiva inversa, para a execução de correcção de côr entre imagens de forma a obter mosaicos de qualidade, ou para a geração de uma representação de cena baseada em primitivas poligonais, capaz de lidar com grandes quantidades de dados 3D e 2D, tendo inclusivamente a capacidade de refinar a representação à medida que novos dados sensoriais são recebidos.The main object of this thesis is the study of algorithms for automatic information processing and representation, in particular information provided by onboard sensors (2D and 3D), to be used in the context of driving assistance. The work focuses on some of the problems facing todays Autonomous Driving (AD) systems and Advanced Drivers Assistance Systems (ADAS). The document is composed of two parts. The first part describes the design, construction and development of three robotic prototypes, including remarks about onboard sensors, algorithms and software architectures. These robots were used as test beds for testing and validating the developed techniques; additionally, they have participated in several autonomous driving competitions with very good results. The second part of this document presents several algorithms for generating intermediate representations of the raw sensor data. They can be used to enhance existing pattern recognition, detection or navigation techniques, and may thus benefit future AD or ADAS applications. Since vehicles often contain a large amount of sensors of different natures, intermediate representations are particularly advantageous; they can be used for tackling problems related with the diverse nature of the data (2D, 3D, photometric, etc.), with the asynchrony of the data (multiple sensors streaming data at different frequencies), or with the alignment of the data (calibration issues, different sensors providing different measurements of the same object). Within this scope, novel techniques are proposed for computing a multicamera multi-modal inverse perspective mapping representation, executing color correction between images for obtaining quality mosaics, or to produce a scene representation based on polygonal primitives that can cope with very large amounts of 3D and 2D data, including the ability of refining the representation as new information is continuously received

Enabling NATO’s Collective Defense: Critical Infrastructure Security and Resiliency (NATO COE-DAT Handbook 1)

In 2014 NATO’s Center of Excellence-Defence Against Terrorism (COE-DAT) launched the inaugural course on “Critical Infrastructure Protection Against Terrorist Attacks.” As this course garnered increased attendance and interest, the core lecturer team felt the need to update the course in critical infrastructure (CI) taking into account the shift from an emphasis on “protection” of CI assets to “security and resiliency.” What was lacking in the fields of academe, emergency management, and the industry practitioner community was a handbook that leveraged the collective subject matter expertise of the core lecturer team, a handbook that could serve to educate government leaders, state and private-sector owners and operators of critical infrastructure, academicians, and policymakers in NATO and partner countries. Enabling NATO’s Collective Defense: Critical Infrastructure Security and Resiliency is the culmination of such an effort, the first major collaborative research project under a Memorandum of Understanding between the US Army War College Strategic Studies Institute (SSI), and NATO COE-DAT. The research project began in October 2020 with a series of four workshops hosted by SSI. The draft chapters for the book were completed in late January 2022. Little did the research team envision the Russian invasion of Ukraine in February this year. The Russian occupation of the Zaporizhzhya nuclear power plant, successive missile attacks against Ukraine’s electric generation and distribution facilities, rail transport, and cyberattacks against almost every sector of the country’s critical infrastructure have been on world display. Russian use of its gas supplies as a means of economic warfare against Europe—designed to undermine NATO unity and support for Ukraine—is another timely example of why adversaries, nation-states, and terrorists alike target critical infrastructure. Hence, the need for public-private sector partnerships to secure that infrastructure and build the resiliency to sustain it when attacked. Ukraine also highlights the need for NATO allies to understand where vulnerabilities exist in host nation infrastructure that will undermine collective defense and give more urgency to redressing and mitigating those fissures.https://press.armywarcollege.edu/monographs/1951/thumbnail.jp