Smart Computer Security Audit: Reinforcement Learning with a Deep Neural Network Approximator

A significant challenge in modern computer security is the growing skill gap as intruder capabilities increase, making it necessary to begin automating elements of penetration testing so analysts can contend with the growing number of cyber threats. In this paper, we attempt to assist human analysts by automating a single host penetration attack. To do so, a smart agent performs different attack sequences to find vulnerabilities in a target system. As it does so, it accumulates knowledge, learns new attack sequences and improves its own internal penetration testing logic. As a result, this agent (AgentPen for simplicity) is able to successfully penetrate hosts it has never interacted with before. A computer security administrator using this tool would receive a comprehensive, automated sequence of actions leading to a security breach, highlighting potential vulnerabilities, and reducing the amount of menial tasks a typical penetration tester would need to execute. To achieve autonomy, we apply an unsupervised machine learning algorithm, Q-learning, with an approximator that incorporates a deep neural network architecture. The security audit itself is modelled as a Markov Decision Process in order to test a number of decisionmaking strategies and compare their convergence to optimality. A series of experimental results is presented to show how this approach can be effectively used to automate penetration testing using a scalable, i.e. not exhaustive, and adaptive approach

Smart Security Audit: Reinforcement Learning with a Deep Neural Network Approximator

No embargo require

A novel switched model predictive control of wind turbines using artificial neural network-Markov chains prediction with load mitigation

The existing model predictive control algorithm based on continuous control using quadratic programming is currently one of the most used modern control strategies applied to wind turbines. However, heavy computational time involved and complexity in implementation are still obstructions in existing model predictive control algorithm. Owing to this, a new switched model predictive control technique is developed for the control of wind turbines with the ability to reduce complexity while maintaining better efficiency. The proposed technique combines model predictive control operating on finite control set and artificial intelligence with reinforcement techniques (Markov Chains, MC) to design a new effective control law which allows to achieve the control objectives in different wind speed zones with minimization of computational complexity. The proposed method is compared with the existing model predictive control algorithm, and it has been found that the proposed algorithm is better in terms of computational time, load mitigation, and dynamic response. The proposed research is a forward step towards refining modern control techniques to achieve optimization in nonlinear process control using novel hybrid structures based on conventional control laws and artificial intelligence.© 2021 THE AUTHORS. Published by Elsevier BV on behalf of Faculty of Engineering, Ain Shams University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)fi=vertaisarvioitu|en=peerReviewed

Resource-Aware Predictive Models in Cyber-Physical Systems

Cyber-Physical Systems (CPS) are composed of computing devices interacting with physical systems. Model-based design is a powerful methodology in CPS design in the implementation of control systems. For instance, Model Predictive Control (MPC) is typically implemented in CPS applications, e.g., in path tracking of autonomous vehicles. MPC deploys a model to estimate the behavior of the physical system at future time instants for a specific time horizon. Ordinary Differential Equations (ODE) are the most commonly used models to emulate the behavior of continuous-time (non-)linear dynamical systems. A complex physical model may comprise thousands of ODEs that pose scalability, performance and power consumption challenges. One approach to address these model complexity challenges are frameworks that automate the development of model-to-model transformation. In this dissertation, a state-based model with tunable parameters is proposed to operate as a reconfigurable predictive model of the physical system. Moreover, we propose a run-time switching algorithm that selects the best model using machine learning. We employed a metric that formulates the trade-off between the error and computational savings due to model reduction. Building statistical models are constrained to having expert knowledge and an actual understanding of the modeled phenomenon or process. Also, statistical models may not produce solutions that are as robust in a real-world context as factors outside the model, like disruptions would not be taken into account. Machine learning models have emerged as a solution to account for the dynamic behavior of the environment and automate intelligence acquisition and refinement. Neural networks are machine learning models, well-known to have the ability to learn linear and nonlinear relations between input and output variables without prior knowledge. However, the ability to efficiently exploit resource-hungry neural networks in embedded resource-bound settings is a major challenge.Here, we proposed Priority Neuron Network (PNN), a resource-aware neural networks model that can be reconfigured into smaller sub-networks at runtime. This approach enables a trade-off between the model's computation time and accuracy based on available resources. The PNN model is memory efficient since it stores only one set of parameters to account for various sub-network sizes. We propose a training algorithm that applies regularization techniques to constrain the activation value of neurons and assigns a priority to each one. We consider the neuron's ordinal number as our priority criteria in that the priority of the neuron is inversely proportional to its ordinal number in the layer. This imposes a relatively sorted order on the activation values. We conduct experiments to employ our PNN as the predictive model in a CPS application. We can see that not only our technique will resolve the memory overhead of DNN architectures but it also reduces the computation overhead for the training process substantially. The training time is a critical matter especially in embedded systems where many NN models are trained on the fly

A machine learning approach for smart computer security audit

This thesis presents a novel application of machine learning technology to automate network security audit and penetration testing processes in particular. A model-free reinforcement learning approach is presented. It is characterized by the absence of the environmental model. The model is derived autonomously by the audit system while acting in the tested computer network. The penetration testing process is specified as a Markov decision process (MDP) without definition of reward and transition functions for every state/action pair. The presented approach includes application of traditional and modified Q-learning algorithms. A traditional Q-learning algorithm learns the action-value function stored in the table, which gives the expected utility of executing a particular action in a particular state of the penetration testing process. The modified Q-learning algorithm differs by incorporation of the state space approximator and representation of the action-value function as a linear combination of features. Two deep architectures of the approximator are presented: autoencoder joint with artificial neural network (ANN) and autoencoder joint with recurrent neural network (RNN). The autoencoder is used to derive the feature set defining audited hosts. ANN is intended to approximate the state space of the audit process based on derived features. RNN is a more advanced version of the approximator and differs by the existence of the additional loop connections from hidden to input layers of the neural network. Such architecture incorporates previously executed actions into new inputs. It gives the opportunity to audit system learn sequences of actions leading to the goal of the audit, which is defined as receiving administrator rights on the host. The model-free reinforcement learning approach based on traditional Q-learning algorithms was also applied to reveal new vulnerabilities, buffer overflow in particular. The penetration testing system showed the ability to discover a string, exploiting potential vulnerability, by learning its formation process on the go. In order to prove the concept and to test the efficiency of an approach, audit tool was developed. Presented results are intended to demonstrate the adaptivity of the approach, performance of the algorithms and deep machine learning architectures. Different sets of hyperparameters are compared graphically to test the ability of convergence to the optimal action policy. An action policy is a sequence of actions, leading to the audit goal (getting admin rights on the remote host). The testing environment is also presented. It consists of 80+ virtual machines based on a vSphere virtualization platform. This combination of hosts represents a typical corporate network with Users segment, Demilitarized zone (DMZ) and external segment (Internet). The network has typical corporate services available: web server, mail server, file server, SSH, SQL server. During the testing process, the audit system acts as an attacker from the Internet

Nonlinear moving-horizon state estimation for hardware implementation and a model predictive control application

Dissertação (mestrado)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Mecânica, 2021.Nesta dissertação, exploramos a aplicação de redes neurais artificiais de funções de base radial (RBFs) embutidas em hardware para estimação de estados e controle em tempo real utilizando os algoritmos de Moving-Horizon Estimation(MHE) e Model Predictive Control (MPC). Esses algoritmos foram posteriormente aproximados por RBFs e implementados em um Field Programmable Gate Array (FPGA), que tem mostrado bons resultados em termos de precisão e tempo ˜ computacional. Mostramos que a estimativa de estado usando a versão aproximada do MHE ˜ pode ser executada usando um kit em escala de laboratório de aproximadamente 500 kHz para ´ um pendulo invertido a uma taxa de clock de cerca de 110 MHz. A latência para fornecer uma estimativa pode ser reduzida ainda mais quando FPGAs com clocks mais altos são usados, pois a ˜ arquitetura da rede neural artificial e inerentemente paralela. Após uma inspeção mais detalhada, ˜ descobriu-se que era possível reduzir o custo da área de chip trocando a função de custo por uma ˜ com resultados mais facilmente representáveis. Ele poderia então utilizar uma representação em ˜ 32 bits e o modulo CORDIC poderia ser removido, usando apenas a aproximação mais simples da ˜ serie de Taylor de 2 ´ ª ordem. Em seguida, expandimos isso, investigando a ideia de usar uma única rede neural para substituir tanto o controle quanto o estimatidor de estados. Comparado a um MPC com informações completas, sua versão utilizando o MHE não teve um bom desempenho contra ˜ ruídos de saída. A princípio não foi possível aproximar o controle e a estimativa do pêndulo com um bom resultado, porem ao separar o controle em duas partes obtivemos melhores resultados. Por fim, verificamos que tal rede neural foi capaz de estabilizar o sistema de pendulo invertido, ˆ mas não de aproximar sua parte oscilante n ˜ ao linear. A solução aqui apresentada ˜ e encorajada a ser estendida para sistemas mais complexos e não lineares, uma vez que uma arquitetura com ˜ complexidade razoável é encontrada para a rede neural artificial para ser implementada.Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES).In this dissertation, we explore the application of radial basis functions (RBFs) artificial neural networks embedded in hardware for real-time estimation and control algorithms as the Moving- Horizon Estimation (MHE) and the Model Predictive Control (MPC). These algorithms are then approximated using RBFs and implemented in a Field Programmable Gate Array (FPGA), which has shown good results in terms of accuracy and computational time. We show that the state estimate using the approximate version of the MHE can be run using a laboratory-scale kit of approximately 500 kHz for an inverted pendulum at a clock rate of about 110 MHz. The latency to provide an estimate can be further reduced when FPGAs with higher clocks are used as the artificial neural network architecture is inherently parallel. Upon further inspection, it was found to be possible to reduce the chip area cost by switching the cost function for one with more easily representable results. It could then utilize a 32-bits representation and the CORDIC module could be removed, using instead only the simpler 2o order Taylor approximation. We then expand upon this, probing at the idea of using a single neural network to substitute both the control and state-estimation. Compared to a MPC with full information, its version utilizing the MHE did not perform well against output noises. At first, it was not possible to approximate the pendulum control and estimation with a good result, however when separating the control in two parts we gained better outcomes. Lastly, we verify that such a neural network was capable of stabilizing the inverted pendulum system, but not of approximating the non-linear swing-up part of it. The solution herein presented is encouraged to be further extended for more complex and nonlinear systems, given that an architecture is found for the artificial neural network with reasonable complexity to be implemented

Arquiteturas de hardware para aceleração de algoritmos de controle preditivo não-linear

Tese (doutorado)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Mecânica, 2018.O Controle Preditivo Baseado em Modelos (MPC) é uma técnica avançada de controle que vem ganhando espaço tanto na academia quanto na indústria ao longo das últimas décadas. O fato de incorporar restrições em sua lei de controle e de poder ser aplicada tanto para sistemas lineares simples quanto para sistemas não-lineares complexos com múltiplas entradas e múltiplas saídas tornam seu emprego bastante atraente. Porém, seu alto custo computacional muitas vezes impede sua aplicação a sistemas com dinâmicas rápidas, principalmente a sistemas não-lineares embarcados onde há restrições computacionais e de consumo de energia. Baseado nisso, este trabalho se propõe a desenvolver algoritmos e arquiteturas em hardware capazes de viabilizar a aplicação do Controle Preditivo Não-Linear (NMPC) para sistemas embarcados. Duas abordagens são desenvolvidas ao longo do trabalho. A primeira aplica técnicas de aprendizado de máquina utilizando Redes Neurais Artificiais (RNAs) e Máquinas de Vetor de Suporte (SVMs) para criar soluções que aproximam o comportamento do NMPC em hardware. Neste caso, técnicas para o treinamento das RNAs e SVMs são exploradas com o intuito de generalizar uma solução capaz de lidar com uma ampla faixa de referências de controle. Em seguida, arquiteturas de hardware em ponto-flutuante para a implementação de RNAs do tipo RBF (Radial Basis Functions) e SVMs são desenvolvidas juntamente com configurador automático capaz de gerar os códigos VHDL (VHSIC Hardware Description Language) das respectivas arquiteturas baseado nos resultados de treinamento e sua topologia. As arquiteturas resultantes são testadas em um FPGA (Field-Programmable Gate Array) de baixo custo e são capazes de computar soluções em menos de 1 s. Na segunda abordagem, o algoritmo heurístico de Otimização por Enxame de Partículas (PSO), é estudado e adaptado para etapa de busca da sequência de controle ótima do NMPC. Dentre as modificações estão incluídas a adição de funções de penalização para obedecer às restrições de estados do sistema, o aprimoramento da técnica KPSO (Knowledge-Based PSO), denominada KPSO+SS, onde resultados de períodos de soluções de períodos amostragem anteriores são combinados com informações sobre o sinal de controle em estado estacionário e seus valores máximos e mínimos para agilizar a busca pela solução ótima. Mais uma vez, arquiteturas de hardware em ponto-flutuante são desenvolvidas para viabilizar a aplicação do controlador NMPC-PSO a sistemas embarcados. Um gerador de códigos da solução NMPC-PSO é proposto para permitir a aplicação da mesma arquitetura a outros sistemas. Em seguida, a solução é testada para o procedimento de swing-up do pêndulo invertido utilizando uma plataforma hardware-inthe- loop (HIL) e apresentou bom desempenho em tempo-real calculando a solução em menos de 3 ms. Finalmente, a solução NMPC-PSO é validada em um sistema de pêndulos gêmeos e outro sistema de controle de atitudes de um satélite.Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) e Decanato de Pesquisa e Inovação -(DPI/ UnB).Model-based Predictive Control (MPC) is an advanced control technique that has been gaining adoption in industry and the academy along the last few decades. Its ability to incorporate system constraints in the control law and be applied from simple linear systems up to more complex nonlinear systems with multiple inputs and outputs attracts its usage. However, the high computational cost associated with this technique often hinders its use, especially in embedded nonlinear systems with fast dynamics with computational and restrictions. Based on these facts, this work aims to study and develop algorithms and hardware architectures that can enable the application of Nonlinear Model Predictive Control (NMPC) on embedded systems. Two approaches are developed throughout this work. The first one applies machine learning techniques using Artificial Neural Networks (ANNs) and Support Vector Machines (SVMs) to create solutions that approximate the NMPC behavior in hardware. In this case, ANN and SVM training techniques are explored with the aim to generalize the control solution and work on a large range of reference control inputs. Next, floating-point hardware architectures to implement Radial Basis Function ANNs and SVM solutions are developed along with an automatic architectural configuration too, capable of generating the VHDL (VHSIC Hardware Description Language) codes based on the training results and its topology. Resulting architectures are tested on a low-cost FPGA (Field-Programmable Gate Array) and are capable of computing the solution in under 1 s. In a second approach, the Particle Swarm Optimization (PSO), which is a heuristic algorithm, is studied and adapted to perform the optimal control sequence search phase of the NMPC. Among the main optimizations performed are the addition of penalty functions to address the controlled system state constraints, an improved KPSO (Knowledge-Based PSO) technique named KPSO+SS, where results from previous sampling periods are combined with steady-state control information to speed-up the optimal solution search. Hardware architectures with floating-point arithmetic to enable the application of the NMPC-PSO solution on embedded systems are developed. Once again, a hardware description configuration tool is created to allow the architecture to be applied to multiple systems. Then, the solution is applied to a real-time inverted pendulum swing-up procedure tested on a hardware-in-the-loop (HIL) platform. The experiment yielding good performance and control results and was able to compute the solutions in under 3 ms. Finally, the NMPC-PSO solution is further validated performing a swing-up procedure on a Twin Pendulum system and then on a satellite control platform, a system with multiple inputs and output

Machine Learning for Identification and Optimal Control of Advanced Automotive Engines.

The complexity of automotive engines continues to increase to meet increasing performance requirements such as high fuel economy and low emissions. The increased sensing capabilities associated with such systems generate a large volume of informative data. With advancements in computing technologies, predictive models of complex dynamic systems useful for diagnostics and controls can be developed using data based learning. Such models have a short development time and can serve as alternatives to traditional physics based modeling. In this thesis, the modeling and control problem of an advanced automotive engine, the homogeneous charge compression ignition (HCCI) engine, is addressed using data based learning techniques. Several frameworks including design of experiments for data generation, identification of HCCI combustion variables, modeling the HCCI operating envelope and model predictive control have been developed and analyzed. In addition, stable online learning algorithms for a general class of nonlinear systems have been developed using extreme learning machine (ELM) model structure.PHDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/102392/1/vijai_1.pd

Elman Neural Networks in Model Predictive Control

The goal of this paper is to present interesting way how to model and predict nonlinear systems using recurrent neural network. This type of artificial neural networks is underestimated and marginalized. Nevertheless, it offers superior modelling features at reasonable computational costs. This contribution is focused on Elman Neural Network, two-layered recurrent neural network. The abilities of this network are presented in the nonlinear system control. The task of the controller is to control the liquid level in the second of two interconnected cylindrical tanks. The mathematical model of the real-time system was derived in order to test predictor and consequently the controller in Matlab/Simulink simulations