Global Optimization for Value Function Approximation

Existing value function approximation methods have been successfully used in many applications, but they often lack useful a priori error bounds. We propose a new approximate bilinear programming formulation of value function approximation, which employs global optimization. The formulation provides strong a priori guarantees on both robust and expected policy loss by minimizing specific norms of the Bellman residual. Solving a bilinear program optimally is NP-hard, but this is unavoidable because the Bellman-residual minimization itself is NP-hard. We describe and analyze both optimal and approximate algorithms for solving bilinear programs. The analysis shows that this algorithm offers a convergent generalization of approximate policy iteration. We also briefly analyze the behavior of bilinear programming algorithms under incomplete samples. Finally, we demonstrate that the proposed approach can consistently minimize the Bellman residual on simple benchmark problems

Machine learning for helicopter dynamics models

Abstract Machine Learning for Helicopter Dynamics Models by Ali Punjani Master of Science in Computer Science University of California, Berkeley Professor Pieter Abbeel, Chair We consider the problem of system identification of helicopter dynamics. Helicopters are complex systems, coupling rigid body dynamics with aerodynamics, engine dynamics, vibration, and other phenomena. Resultantly, they pose a challenging system identification problem, especially when considering non-stationary flight regimes. We pose the dynamics modeling problem as direct high-dimensional regression, and take inspiration from recent results in Deep Learning to represent the helicopter dynamics with a Rectified Linear Unit (ReLU) Network Model, a hierarchical neural network model. We provide a simple method for initializing the parameters of the model, and optimization details for training. We describe three baseline models and show that they are significantly outperformed by the ReLU Network Model in experiments on real data, indicating the power of the model to capture useful structure in system dynamics across a rich array of aerobatic maneuvers. Specifically, the ReLU Network Model improves 58% overall in RMS acceleration prediction over state-of-the-art methods. Predicting acceleration along the helicopter's up-down axis is empirically found to be the most difficult, and the ReLU Network Model improves by 60% over the prior state-ofthe-art. We discuss explanations of these performance gains, and also investigate the impact of hyperparameters in the novel model.

Neuroevolutionary reinforcement learning for generalized control of simulated helicopters

This article presents an extended case study in the application of neuroevolution to generalized simulated helicopter hovering, an important challenge problem for reinforcement learning. While neuroevolution is well suited to coping with the domain’s complex transition dynamics and high-dimensional state and action spaces, the need to explore efficiently and learn on-line poses unusual challenges. We propose and evaluate several methods for three increasingly challenging variations of the task, including the method that won first place in the 2008 Reinforcement Learning Competition. The results demonstrate that (1) neuroevolution can be effective for complex on-line reinforcement learning tasks such as generalized helicopter hovering, (2) neuroevolution excels at finding effective helicopter hovering policies but not at learning helicopter models, (3) due to the difficulty of learning reliable models, model-based approaches to helicopter hovering are feasible only when domain expertise is available to aid the design of a suitable model representation and (4) recent advances in efficient resampling can enable neuroevolution to tackle more aggressively generalized reinforcement learning tasks

Driver Behavior in Traffic

DTFH61-09-H-00007Existing traffic analysis and management tools do not model the ability of drivers to recognize their environment and respond to it with behaviors that vary according to the encountered driving situation. The small body of literature on characterizing drivers behavior is typically limited to specific locations (i.e., by collecting data on specific intersections or freeway sections) and is very narrow in scope. This report documented the research performed to model driver behavior in traffic under naturalistic driving data. The research resulted in the development of hybrid car-following model. In addition, a neuro-fuzzy reinforcement learning, an agent-based artificial intelligence machine-learning technique, was used to model driving behavior. The naturalistic driving database was used to train and validate driver agents. The proposed methodology simulated events from different drivers and proved behavior heterogeneities. Robust agent activation techniques were also developed using discriminant analysis. The developed agents were implemented in VISSIM simulation platform and were evaluated by comparing the behavior of vehicles with and without agent activation. The results showed very close resemblance of the behavior of agents and driver data. Prototype agents prototype (spreadsheets and codes) were developed. Future research recommendations include training agents using more data to cover a wider region in the Wiedemann regime space, and sensitivity analysis of agent training parameters

5G wireless network support using umanned aerial vehicles for rural and low-Income areas

>Magister Scientiae - MScThe fifth-generation mobile network (5G) is a new global wireless standard that enables state-of-the-art mobile networks with enhanced cellular broadband services that support a diversity of devices. Even with the current worldwide advanced state of broadband connectivity, most rural and low-income settings lack minimum Internet connectivity because there are no economic incentives from telecommunication providers to deploy wireless communication systems in these areas. Using a team of Unmanned Aerial Vehicles (UAVs) to extend or solely supply the 5G coverage is a great opportunity for these zones to benefit from the advantages promised by this new communication technology. However, the deployment and applications of innovative technology in rural locations need extensive research