Adversarial Attacks to Reward Machine-based Reinforcement Learning

In recent years, Reward Machines (RMs) have stood out as a simple yet effective automata-based formalism for exposing and exploiting task structure in reinforcement learning settings. Despite their relevance, little to no attention has been directed to the study of their security implications and robustness to adversarial scenarios, likely due to their recent appearance in the literature. With my thesis, I aim to provide the first analysis of the security of RM-based reinforcement learning techniques, with the hope of motivating further research in the field, and I propose and evaluate a novel class of attacks on RM-based techniques: blinding attacks.Comment: Thesis Supervisor: Prof. Federico Cerutti (Universit\`a degli Studi di Brescia, IT

Analysis of Square-Root Kalman Filters for Angles-Only Orbital Navigation and the Effects of Sensor Accuracy on State Observability

Angles-only navigation is simple, robust, and well proven in many applications. However, it is sometimes ill-conditioned for orbital rendezvous and proximity operations because, without a direct range measurement, the distance to approaching satellites must be estimated by firing thrusters and observing the change in the target\u27s bearing. Nevertheless, the simplicity of angles-only navigation gives it great appeal. The viability of this technique for relative navigation is examined by building a high-fidelity simulation and evaluating the sensitivity of the system to sensor errors. The relative performances of square-root filtering methods, including Potter, Carlson, and UD factorization filters, are compared to the conventional and Joseph formulations. Filter performance is evaluated during closed-loop station keeping operations in simulation

Procedural-Reasoning Architecture for Applied Behavior Analysis-based Instructions

Autism Spectrum Disorder (ASD) is a complex developmental disability affecting as many as 1 in every 88 children. While there is no known cure for ASD, there are known behavioral and developmental interventions, based on demonstrated efficacy, that have become the predominant treatments for improving social, adaptive, and behavioral functions in children. Applied Behavioral Analysis (ABA)-based early childhood interventions are evidence based, efficacious therapies for autism that are widely recognized as effective approaches to remediation of the symptoms of ASD. They are, however, labor intensive and consequently often inaccessible at the recommended levels. Recent advancements in socially assistive robotics and applications of virtual intelligent agents have shown that children with ASD accept intelligent agents as effective and often preferred substitutes for human therapists. This research is nascent and highly experimental with no unifying, interdisciplinary, and integral approach to development of intelligent agents based therapies, especially not in the area of behavioral interventions. Motivated by the absence of the unifying framework, we developed a conceptual procedural-reasoning agent architecture (PRA-ABA) that, we propose, could serve as a foundation for ABA-based assistive technologies involving virtual, mixed or embodied agents, including robots. This architecture and related research presented in this disser- tation encompass two main areas: (a) knowledge representation and computational model of the behavioral aspects of ABA as applicable to autism intervention practices, and (b) abstract architecture for multi-modal, agent-mediated implementation of these practices

Building and monitoring an event-driven microservices ecosystem

Throughout the years, software architectures have evolved deeply to attempt to address the main issues that have been emerging, mainly due to the ever-changing market needs. The need to provide a way for organizations and teams to build applications independently and with greater agility and speed led to the adoption of microservices, particularly endorsing an asynchronous methodology of communication between them via events. Moreover, the evergrowing demands for high-quality resilient and highly available systems helped pave the path towards a greater focus on strict quality measures, particularly monitoring and other means of assuring the well-functioning of components in production in real-time. Although techniques like logging, monitoring, and alerting are essential to be employed for each microservice, it may not be enough considering an event-driven architecture. Studies have shown that although organizations have been adopting this type of software architecture, they still struggle with the lack of visibility into end-to-end business processes that span multiple microservices. This thesis explores how to guarantee observability over such architecture, thus keeping track of the business processes. It shall do so by providing a tool that facilitates the analysis of the current situation of the ecosystem, as well as allow to view and possibly act upon the data. Two solutions have been explored and are therefore presented thoroughly, alongside a detailed comparison with the purpose of drawing conclusions and providing some guidance to the readers. These outcomes that were produced by the thesis resulted in a paper published and registered to be presented at this year’s edition of the SEI hosted at ISEP.Ao longo dos últimos anos, as arquiteturas de software têm evoluído significativamente de forma a tentar resolver os principais problemas que têm surgindo, principalmente derivados nas necessidades do mercado que estão em constante mudança. A necessidade de providenciar uma forma das organizações e suas equipas construírem aplicações independentemente e com uma maior agilidade e rapidez levou à adoção de microserviços, geralmente aplicando uma metodologia de comunicação assíncrona através de eventos. Para além disso, a constante evolução da necessidade de ter sistemas de qualidade e altamente resilientes e disponíveis, ajudou a direcionar um maior foco para padrões de qualidade mais rigorosos, particularmente no que toca a monitorização e outros meios para assegurar o correto funcionamento de componentes em produção em tempo-real. Embora técnicas como a produção de logs, monitorização e alarmística sejam essenciais para ser aplicadas a cada microserviço, poderá não ser suficiente quando consideramos uma arquitetura baseada em eventos. Estudos recentes apontam para que organizações, apesar de estarem a adotar cada vez mais este tipo de arquiteturas de software, ainda encontram bastantes dificuldades devido à falta de visibilidade que possuem dos processos de negócio que envolvem e se propagam por diversos microserviços. Esta tese explora como garantir visibilidade sobre uma arquitetura como a descrita, e assim conseguir seguir os processos de negócio. O resultado da mesma deverá atender a isso providenciando uma ferramenta que facilita a análise da situação atual do ecossistema, e que possibilita a visualização e a intervenção sobre os dados que são disponibilizados. Foram desenvolvidas duas soluções que serão apresentadas detalhadamente juntamente com uma comparação entre as duas com o propósito de tirar mais conclusões e providenciar alguma orientação ao leitor. A tese originou a criação de um artigo submetido para ser apresentado na edição deste ano do SEI

Um estudo sobre métodos de determinação de estados e parâmetros de máquinas síncronas de polos salientes

Orientador: Mateus GiesbrechtDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: As máquinas síncronas de polos salientes desempenham um papel fundamental na análise de estabilidade de sistemas elétricos de potência, especialmente em países cuja maior parte da energia gerada provém de fontes hidráulicas. Os modelos elétricos equivalentes que descrevem o comportamento dessas máquinas são compostos por diversos parâmetros, os quais são utilizados em uma ampla gama de estudos. No presente trabalho, estudam-se e propõem-se técnicas de estimação de estados e parâmetros de máquinas síncronas de polos salientes. A princípio, as equações de tensão, de fluxos concatenados, de potência e de movimento são desenvolvidas com as devidas unidades de medida, tanto em variáveis de máquina quanto em variáveis projetadas sobre um plano ortogonal que gira na velocidade elétrica do rotor. Na maior parte da literatura, essas unidades não são explicitadas no equacionamento. Dentre os parâmetros elétricos dos modelos das máquinas síncronas de polos salientes, as reatâncias de magnetização são os que mais influenciam o comportamento da máquina em condições de regime permanente senoidal. Desta forma, apresenta-se uma nova abordagem à estimação do ângulo de carga dessas máquinas e o subsequente cálculo das reatâncias de magnetização a partir de condições de carga específicas -- o desempenho do método proposto é avaliado em dados de simulação e em dados reais de operação de um gerador síncrono de grande porte. Algumas abordagens à determinação de parâmetros requerem que a máquina seja posta fora de operação para que ensaios específicos possam ser realizados. Dentre eles, um dos mais empregados na determinação de parâmetros transitórios e de regime permanente é o ensaio de rejeição de carga; assim, este ensaio também é analisado e aperfeiçoado por um método automatizado de separação de soma de exponenciais baseado em projeção de variáveis. Por tratar-se de um sistema multivariável e altamente não linear, diferentes observadores de estado também são utilizados para se determinarem estados e parâmetros de máquinas síncronas em tempo hábil e com precisão satisfatória. Este trabalho apresenta uma abordagem não linear recursivamente aplicável à estimação de fluxos concatenados, correntes de enrolamentos amortecedores, ângulo de carga e reatâncias de magnetização de máquinas síncronas de polos salientes por meio da filtragem de partículas. Um modelo não linear de oitava ordem é considerado e apenas as medições realizadas nos terminais da armadura e do campo durante regime permanente se fazem necessárias para estimar as referidas grandezasAbstract: Salient-pole synchronous machines play a fundamental role in the stability analysis of electrical power systems, especially in countries where most of the generated energy comes from hydraulic sources. The electrical equivalent models that describe the behavior of these machines are composed of several electrical parameters, which are used in a wide range of studies. In the present work, techniques for estimating states and parameters of salient-pole synchronous machines are studied and proposed. A priori, the voltage, flux linkage, power, and motion equations are developed with the appropriate units included, both in machine variables and in variables projected on an orthogonal plane rotating in the rotor's electrical speed. In most of the literature, these units are not explained in the equation process. Among the electrical parameters, the magnetizing reactances are the ones that most influence the machine behavior under transient and steady-state conditions. In this way, a new approach to estimate the load angle of these machines and the subsequent calculation of the magnetizing reactances from specific load conditions are presented -- the performance of the proposed method is evaluated by means of simulation data and by operating data of a large synchronous generator. Some approaches to determine parameters require the machine to be taken out of operation, so that specific tests may be performed. Among them, one of the most used to determine transient and steady-state parameters is the load rejection test; thus, this test is also analyzed and refined by an automated method based on variable projection for separating the resulting sum-of-exponentials. Since the machines are highly nonlinear, multivariate, dynamic systems, different state observers seek to solve the state estimation problem in a timely manner and with satisfactory accuracy. This work presents a nonlinear and recursive approach for the estimation of flux linkages per second, amortisseur winding currents, load angle, and magnetizing reactances of salient-pole synchronous machines by means of the particle filtering. An eighth-order nonlinear model is considered, and only measurements taken at the machine terminals are necessary to estimate these quantitiesMestradoAutomaçãoMestre em Engenharia Elétrica162015/2018-6CNPq

Design of Discrete-time Chaos-Based Systems for Hardware Security Applications

Security of systems has become a major concern with the advent of technology. Researchers are proposing new security solutions every day in order to meet the area, power and performance specifications of the systems. The additional circuit required for security purposes can consume significant area and power. This work proposes a solution which utilizes discrete-time chaos-based logic gates to build a system which addresses multiple hardware security issues. The nonlinear dynamics of chaotic maps is leveraged to build a system that mitigates IC counterfeiting, IP piracy, overbuilding, disables hardware Trojan insertion and enables authentication of connecting devices (such as IoT and mobile). Chaos-based systems are also used to generate pseudo-random numbers for cryptographic applications.The chaotic map is the building block for the design of discrete-time chaos-based oscillator. The analog output of the oscillator is converted to digital value using a comparator in order to build logic gates. The logic gate is reconfigurable since different parameters in the circuit topology can be altered to implement multiple Boolean functions using the same system. The tuning parameters are control input, bifurcation parameter, iteration number and threshold voltage of the comparator. The proposed system is a hybrid between standard CMOS logic gates and reconfigurable chaos-based logic gates where original gates are replaced by chaos-based gates. The system works in two modes: logic locking and authentication. In logic locking mode, the goal is to ensure that the system achieves logic obfuscation in order to mitigate IC counterfeiting. The secret key for logic locking is made up of the tuning parameters of the chaotic oscillator. Each gate has 10-bit key which ensures that the key space is large which exponentially increases the computational complexity of any attack. In authentication mode, the aim of the system is to provide authentication of devices so that adversaries cannot connect to devices to learn confidential information. Chaos-based computing system is susceptible to process variation which can be leveraged to build a chaos-based PUF. The proposed system demonstrates near ideal PUF characteristics which means systems with large number of primary outputs can be used for authenticating devices

Direct Adaptive Control for a Trajectory Tracking UAV

This research focuses on the theoretical development and analysis of a direct adaptive control algorithm to enable a fixed-wing UAV to track reference trajectories while in the presence of persistent external disturbances. A typical application of this work is autonomous flight through urban environments, where reference trajectories would be provided by a path planning algorithm and the vehicle would be subjected to significant wind gust disturbances. Full 6-DOF nonlinear and linear UAV simulation models are developed and used to study the performance of the direct adaptive control system for various scenarios. A stability proof is developed to prove convergence of the direct adaptive control system under certain conditions. Specific adaptive controller implementation details are provided, including the use of a sensor blending algorithm to address the non-minimum phase properties of the UAV models. The robustness of the adaptive system pertaining to the amount of modeling error that can be accommodated by the controller is studied, and the disturbance rejection capabilities and limitations of the controllers are also analyzed. The overall results of this research demonstrate that the direct adaptive control algorithm can enable trajectory tracking in cases where there are both significant uncertainties in the external disturbances and considerable error in the UAV model

Stochastic Tools for Network Security: Anonymity Protocol Analysis and Network Intrusion Detection

With the rapid development of Internet and the sharp increase of network crime, network security has become very important and received a lot of attention. In this dissertation, we model security issues as stochastic systems. This allows us to find weaknesses in existing security systems and propose new solutions. Exploring the vulnerabilities of existing security tools can prevent cyber-attacks from taking advantages of the system weaknesses. We consider The Onion Router (Tor), which is one of the most popular anonymity systems in use today, and show how to detect a protocol tunnelled through Tor. A hidden Markov model (HMM) is used to represent the protocol. Hidden Markov models are statistical models of sequential data like network traffic, and are an effective tool for pattern analysis. New, flexible and adaptive security schemes are needed to cope with emerging security threats. We propose a hybrid network security scheme including intrusion detection systems (IDSs) and honeypots scattered throughout the network. This combines the advantages of two security technologies. A honeypot is an activity-based network security system, which could be the logical supplement of the passive detection policies used by IDSs. This integration forces us to balance security performance versus cost by scheduling device activities for the proposed system. By formulating the scheduling problem as a decentralized partially observable Markov decision process (DEC-POMDP), decisions are made in a distributed manner at each device without requiring centralized control. When using a HMM, it is important to ensure that it accurately represents both the data used to train the model and the underlying process. Current methods assume that observations used to construct a HMM completely represent the underlying process. It is often the case that the training data size is not large enough to adequately capture all statistical dependencies in the system. It is therefore important to know the statistical significance level that the constructed model represents the underlying process, not only the training set. We present a method to determine if the observation data and constructed model fully express the underlying process with a given level of statistical significance. We apply this approach to detecting the existence of protocols tunnelled through Tor. While HMMs are a powerful tool for representing patterns allowing for uncertainties, they cannot be used for system control. The partially observable Markov decision process (POMDP) is a useful choice for controlling stochastic systems. As a combination of two Markov models, POMDPs combine the strength of HMM (capturing dynamics that depend on unobserved states) and that of Markov decision process (MDP) (taking the decision aspect into account). Decision making under uncertainty is used in many parts of business and science. We use here for security tools. We propose three approximation methods for discrete-time infinite-horizon POMDPs. One of the main contributions of our work is high-quality approximation solution for finite-space POMDPs with the average cost criterion, and their extension to DEC-POMDPs. The solution of the first algorithm is built out of the observable portion when the underlying MDP operates optimally. The other two methods presented here can be classified as the policy-based approximation schemes, in which we formulate the POMDP planning as a quadratically constrained linear program (QCLP), which defines an optimal controller of a desired size. This representation allows a wide range of powerful nonlinear programming (NLP) algorithms to be used to solve POMDPs. Simulation results for a set of benchmark problems illustrate the effectiveness of the proposed method. We show how this tool could be used to design a network security framework

Wide-area monitoring and control of future smart grids

Application of wide-area monitoring and control for future smart grids with substantial wind penetration and advanced network control options through FACTS and HVDC (both point-to-point and multi-terminal) is the subject matter of this thesis. For wide-area monitoring, a novel technique is proposed to characterize the system dynamic response in near real-time in terms of not only damping and frequency but also mode-shape, the latter being critical for corrective control action. Real-time simulation in Opal-RT is carried out to illustrate the effectiveness and practical feasibility of the proposed approach. Potential problem with wide-area closed-loop continuous control using FACTS devices due to continuously time-varying latency is addressed through the proposed modification of the traditional phasor POD concept introduced by ABB. Adverse impact of limited bandwidth availability due to networked communication is established and a solution using an observer at the PMU location has been demonstrated. Impact of wind penetration on the system dynamic performance has been analyzed along with effectiveness of damping control through proper coordination of wind farms and HVDC links. For multi-terminal HVDC (MTDC) grids the critical issue of autonomous power sharing among the converter stations following a contingency (e.g. converter outage) is addressed. Use of a power-voltage droop in the DC link voltage control loops using remote voltage feedback is shown to yield proper distribution of power mismatch according to the converter ratings while use of local voltages turns out to be unsatisfactory. A novel scheme for adapting the droop coefficients to share the burden according to the available headroom of each converter station is also studied. The effectiveness of the proposed approaches is illustrated through detailed frequency domain analysis and extensive time-domain simulation results on different test systems