A Radio-fingerprinting-based Vehicle Classification System for Intelligent Traffic Control in Smart Cities

The measurement and provision of precise and upto-date traffic-related key performance indicators is a key element and crucial factor for intelligent traffic controls systems in upcoming smart cities. The street network is considered as a highly-dynamic Cyber Physical System (CPS) where measured information forms the foundation for dynamic control methods aiming to optimize the overall system state. Apart from global system parameters like traffic flow and density, specific data such as velocity of individual vehicles as well as vehicle type information can be leveraged for highly sophisticated traffic control methods like dynamic type-specific lane assignments. Consequently, solutions for acquiring these kinds of information are required and have to comply with strict requirements ranging from accuracy over cost-efficiency to privacy preservation. In this paper, we present a system for classifying vehicles based on their radio-fingerprint. In contrast to other approaches, the proposed system is able to provide real-time capable and precise vehicle classification as well as cost-efficient installation and maintenance, privacy preservation and weather independence. The system performance in terms of accuracy and resource-efficiency is evaluated in the field using comprehensive measurements. Using a machine learning based approach, the resulting success ratio for classifying cars and trucks is above 99%

Learning Feedback Terms for Reactive Planning and Control

With the advancement of robotics, machine learning, and machine perception, increasingly more robots will enter human environments to assist with daily tasks. However, dynamically-changing human environments requires reactive motion plans. Reactivity can be accomplished through replanning, e.g. model-predictive control, or through a reactive feedback policy that modifies on-going behavior in response to sensory events. In this paper, we investigate how to use machine learning to add reactivity to a previously learned nominal skilled behavior. We approach this by learning a reactive modification term for movement plans represented by nonlinear differential equations. In particular, we use dynamic movement primitives (DMPs) to represent a skill and a neural network to learn a reactive policy from human demonstrations. We use the well explored domain of obstacle avoidance for robot manipulation as a test bed. Our approach demonstrates how a neural network can be combined with physical insights to ensure robust behavior across different obstacle settings and movement durations. Evaluations on an anthropomorphic robotic system demonstrate the effectiveness of our work.Comment: 8 pages, accepted to be published at ICRA 2017 conferenc

Resilient Autonomous Control of Distributed Multi-agent Systems in Contested Environments

An autonomous and resilient controller is proposed for leader-follower multi-agent systems under uncertainties and cyber-physical attacks. The leader is assumed non-autonomous with a nonzero control input, which allows changing the team behavior or mission in response to environmental changes. A resilient learning-based control protocol is presented to find optimal solutions to the synchronization problem in the presence of attacks and system dynamic uncertainties. An observer-based distributed H_infinity controller is first designed to prevent propagating the effects of attacks on sensors and actuators throughout the network, as well as to attenuate the effect of these attacks on the compromised agent itself. Non-homogeneous game algebraic Riccati equations are derived to solve the H_infinity optimal synchronization problem and off-policy reinforcement learning is utilized to learn their solution without requiring any knowledge of the agent's dynamics. A trust-confidence based distributed control protocol is then proposed to mitigate attacks that hijack the entire node and attacks on communication links. A confidence value is defined for each agent based solely on its local evidence. The proposed resilient reinforcement learning algorithm employs the confidence value of each agent to indicate the trustworthiness of its own information and broadcast it to its neighbors to put weights on the data they receive from it during and after learning. If the confidence value of an agent is low, it employs a trust mechanism to identify compromised agents and remove the data it receives from them from the learning process. Simulation results are provided to show the effectiveness of the proposed approach

Fast Reinforcement Learning for Energy-Efficient Wireless Communications

We consider the problem of energy-efficient point-to-point transmission of delay-sensitive data (e.g. multimedia data) over a fading channel. Existing research on this topic utilizes either physical-layer centric solutions, namely power-control and adaptive modulation and coding (AMC), or system-level solutions based on dynamic power management (DPM); however, there is currently no rigorous and unified framework for simultaneously utilizing both physical-layer centric and system-level techniques to achieve the minimum possible energy consumption, under delay constraints, in the presence of stochastic and a priori unknown traffic and channel conditions. In this report, we propose such a framework. We formulate the stochastic optimization problem as a Markov decision process (MDP) and solve it online using reinforcement learning. The advantages of the proposed online method are that (i) it does not require a priori knowledge of the traffic arrival and channel statistics to determine the jointly optimal power-control, AMC, and DPM policies; (ii) it exploits partial information about the system so that less information needs to be learned than when using conventional reinforcement learning algorithms; and (iii) it obviates the need for action exploration, which severely limits the adaptation speed and run-time performance of conventional reinforcement learning algorithms. Our results show that the proposed learning algorithms can converge up to two orders of magnitude faster than a state-of-the-art learning algorithm for physical layer power-control and up to three orders of magnitude faster than conventional reinforcement learning algorithms

How to Control Hydrodynamic Force on Fluidic Pinball via Deep Reinforcement Learning

Deep reinforcement learning (DRL) for fluidic pinball, three individually rotating cylinders in the uniform flow arranged in an equilaterally triangular configuration, can learn the efficient flow control strategies due to the validity of self-learning and data-driven state estimation for complex fluid dynamic problems. In this work, we present a DRL-based real-time feedback strategy to control the hydrodynamic force on fluidic pinball, i.e., force extremum and tracking, from cylinders' rotation. By adequately designing reward functions and encoding historical observations, and after automatic learning of thousands of iterations, the DRL-based control was shown to make reasonable and valid control decisions in nonparametric control parameter space, which is comparable to and even better than the optimal policy found through lengthy brute-force searching. Subsequently, one of these results was analyzed by a machine learning model that enabled us to shed light on the basis of decision-making and physical mechanisms of the force tracking process. The finding from this work can control hydrodynamic force on the operation of fluidic pinball system and potentially pave the way for exploring efficient active flow control strategies in other complex fluid dynamic problems

DESIGN, MODELING, AND CONTROL OF SOFT DYNAMIC SYSTEMS

Soft physical systems, be they elastic bodies, fluids, and compliant-bodied creatures, are ubiquitous in nature. Modeling and simulation of these systems with computer algorithms enable the creation of visually appealing animations, automated fabrication paradigms, and novel user interfaces and control mechanics to assist designers and engineers to develop new soft machines. This thesis develops computational methods to address the challenges emerged during the automation of the design, modeling, and control workflow supporting various soft dynamic systems. On the design/control side, we present a sketch-based design interface to enable non-expert users to design soft multicopters. Our system is endorsed by a data-driven algorithm to generate system identification and control policies given a novel shape prototype and rotor configurations. We show that our interactive system can automate the workflow of different soft multicopters\u27 design, simulation, and control with human designers involved in the loop. On the modeling side, we study the physical behaviors of fluidic systems from a local, collective perspective. We develop a prior-embedded graph network to uncover the local constraint relations underpinning a collective dynamic system such as particle fluid. We also proposed a simulation algorithm to model vortex dynamics with locally interacting Lagrangian elements. We demonstrate the efficacy of the two systems by learning, simulating and visualizing complicated dynamics of incompressible fluid

Behavior Planning For Connected Autonomous Vehicles Using Feedback Deep Reinforcement Learning

With the development of communication technologies, connected autonomous vehicles (CAVs) can share information with each other. We propose a novel behavior planning method for CAVs to decide actions such as whether to change lane or keep lane based on the observation and shared information from neighbors, and to make sure that there exist corresponding control maneuvers such as acceleration and steering angle to guarantee the safety of each individual autonomous vehicle. We formulate this problem as a hybrid partially observable Markov decision process (HPOMDP) to consider objectives such as improving traffic flow efficiency and driving comfort and safety requirements. The discrete state transition is determined by the proposed feedback deep Q-learning algorithm using the feedback action from an underlying controller based on control barrier functions. The feedback deep Q-learning algorithm we design aims to solve the critical challenge of reinforcement learning (RL) in a physical system: guaranteeing the safety of the system while the RL is exploring the action space to increase the reward. We prove that our method renders a forward invariant safe set for the continuous state physical dynamic model of the system while the RL agent is learning. In experiments, our behavior planning method can increase traffic flow and driving comfort compared with the intelligent driving model (IDM). We also validate that our method maintains safety during the learning process.Comment: conferenc

Automated design of complex dynamic systems

Several fields of study are concerned with uniting the concept of computation with that of the design of physical systems. For example, a recent trend in robotics is to design robots in such a way that they require a minimal control effort. Another example is found in the domain of photonics, where recent efforts try to benefit directly from the complex nonlinear dynamics to achieve more efficient signal processing. The underlying goal of these and similar research efforts is to internalize a large part of the necessary computations within the physical system itself by exploiting its inherent non-linear dynamics. This, however, often requires the optimization of large numbers of system parameters, related to both the system's structure as well as its material properties. In addition, many of these parameters are subject to fabrication variability or to variations through time. In this paper we apply a machine learning algorithm to optimize physical dynamic systems. We show that such algorithms, which are normally applied on abstract computational entities, can be extended to the field of differential equations and used to optimize an associated set of parameters which determine their behavior. We show that machine learning training methodologies are highly useful in designing robust systems, and we provide a set of both simple and complex examples using models of physical dynamical systems. Interestingly, the derived optimization method is intimately related to direct collocation a method known in the field of optimal control. Our work suggests that the application domains of both machine learning and optimal control have a largely unexplored overlapping area which envelopes a novel design methodology of smart and highly complex physical systems

Modern Power System Dynamic Performance Improvement through Big Data Analysis

Higher penetration of Renewable Energy (RE) is causing generation uncertainty and reduction of system inertia for the modern power system. This phenomenon brings more challenges on the power system dynamic behavior, especially the frequency oscillation and excursion, voltage and transient stability problems. This dissertation work extracts the most useful information from the power system features and improves the system dynamic behavior by big data analysis through three aspects: inertia distribution estimation, actuator placement, and operational studies.First of all, a pioneer work for finding the physical location of COI in the system and creating accurate and useful inertia distribution map is presented. Theoretical proof and dynamic simulation validation have been provided to support the proposed method for inertia distribution estimation based on measurement PMU data. Estimation results are obtained for a radial system, a meshed system, IEEE 39 bus-test system, the Chilean system, and a real utility system in the US. Then, this work provided two control actuator placement strategy using measurement data samples and machine learning algorithms. The first strategy is for the system with single oscillation mode. Control actuators should be placed at the bus that are far away from the COI bus. This rule increased damping ratio of eamples systems up to 14\% and hugely reduced the computational complexity from the simulation results of the Chilean system. The second rule is created for system with multiple dynamic problems. General and effective guidance for planners is obtained for IEEE 39-bus system and IEEE 118-bus system using machine learning algorithms by finding the relationship between system most significant features and system dynamic performance. Lastly, it studied the real-time voltage security assessment and key link identification in cascading failure analysis. A proposed deep-learning framework has Achieved the highest accuracy and lower computational time for real-time security analysis. In addition, key links are identified through distance matrix calculation and probability tree generation using 400,000 data samples from the Western Electricity Coordinating Council (WECC) system