A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences

Evolvability signatures of generative encodings: beyond standard performance benchmarks

Evolutionary robotics is a promising approach to autonomously synthesize machines with abilities that resemble those of animals, but the field suffers from a lack of strong foundations. In particular, evolutionary systems are currently assessed solely by the fitness score their evolved artifacts can achieve for a specific task, whereas such fitness-based comparisons provide limited insights about how the same system would evaluate on different tasks, and its adaptive capabilities to respond to changes in fitness (e.g., from damages to the machine, or in new situations). To counter these limitations, we introduce the concept of "evolvability signatures", which picture the post-mutation statistical distribution of both behavior diversity (how different are the robot behaviors after a mutation?) and fitness values (how different is the fitness after a mutation?). We tested the relevance of this concept by evolving controllers for hexapod robot locomotion using five different genotype-to-phenotype mappings (direct encoding, generative encoding of open-loop and closed-loop central pattern generators, generative encoding of neural networks, and single-unit pattern generators (SUPG)). We observed a predictive relationship between the evolvability signature of each encoding and the number of generations required by hexapods to adapt from incurred damages. Our study also reveals that, across the five investigated encodings, the SUPG scheme achieved the best evolvability signature, and was always foremost in recovering an effective gait following robot damages. Overall, our evolvability signatures neatly complement existing task-performance benchmarks, and pave the way for stronger foundations for research in evolutionary robotics.Comment: 24 pages with 12 figures in the main text, and 4 supplementary figures. Accepted at Information Sciences journal (in press). Supplemental videos are available online at, see http://goo.gl/uyY1R

Resource-efficient strategies for mobile ad-hoc networking

The ubiquity and widespread availability of wireless mobile devices with ever increasing inter-connectivity (e. g. by means of Bluetooth, WiFi or UWB) have led to new and emerging next generation mobile communication paradigms, such as the Mobile Ad-hoc NETworks (MANETs). MANETs are differentiated from traditional mobile systems by their unique properties, e. g. unpredictable nodal location, unstable topology and multi-hop packet relay. The success of on-going research in communications involving MANETs has encouraged their applications in areas with stringent performance requirements such as the e-healthcare, e. g. to connect them with existing systems to deliver e-healthcare services anytime anywhere. However, given that the capacity of mobile devices is restricted by their resource constraints (e. g. computing power, energy supply and bandwidth), a fundamental challenge in MANETs is how to realize the crucial performance/Quality of Service (QoS) expectations of communications in a network of high dynamism without overusing the limited resources. A variety of networking technologies (e. g. routing, mobility estimation and connectivity prediction) have been developed to overcome the topological instability and unpredictability and to enable communications in MANETs with satisfactory performance or QoS. However, these technologies often feature a high consumption of power and/or bandwidth, which makes them unsuitable for resource constrained handheld or embedded mobile devices. In particular, existing strategies of routing and mobility characterization are shown to achieve fairly good performance but at the expense of excessive traffic overhead or energy consumption. For instance, existing hybrid routing protocols in dense MANETs are based in two-dimensional organizations that produce heavy proactive traffic. In sparse MANETs, existing packet delivery strategy often replicates too many copies of a packet for a QoS target. In addition, existing tools for measuring nodal mobility are based on either the GPS or GPS-free positioning systems, which incur intensive communications/computations that are costly for battery-powered terminals. There is a need to develop economical networking strategies (in terms of resource utilization) in delivering the desired performance/soft QoS targets. The main goal of this project is to develop new networking strategies (in particular, for routing and mobility characterization) that are efficient in terms of resource consumptions while being effective in realizing performance expectations for communication services (e. g. in the scenario of e-healthcare emergency) with critical QoS requirements in resource-constrained MANETs. The main contributions of the thesis are threefold: (1) In order to tackle the inefficient bandwidth utilization of hybrid service/routing discovery in dense MANETs, a novel "track-based" scheme is developed. The scheme deploys a one-dimensional track-like structure for hybrid routing and service discovery. In comparison with existing hybrid routing/service discovery protocols that are based on two-dimensional structures, the track-based scheme is more efficient in terms of traffic overhead (e. g. about 60% less in low mobility scenarios as shown in Fig. 3.4). Due to the way "provocative tracks" are established, the scheme has also the capability to adapt to the network traffic and mobility for a better performance. (2) To minimize the resource utilization of packet delivery in sparse MANETs where wireless links are intermittently connected, a store-and-forward based scheme, "adaptive multicopy routing", was developed for packet delivery in sparse mobile ad-hoc networks. Instead of relying on the source to control the delivery overhead as in the conventional multi-copy protocols, the scheme allows each intermediate node to independently decide whether to forward a packet according to the soft QoS target and local network conditions. Therefore, the scheme can adapt to varying networking situations that cannot be anticipated in conventional source-defined strategies and deliver packets for a specific QoS targets using minimum traffic overhead. ii (3) The important issue of mobility measurement that imposes heavy communication/computation burdens on a mobile is addressed with a set of resource-efficient "GPS-free" soluti ons, which provide mobility characterization with minimal resource utilization for ranging and signalling by making use of the information of the time-varying ranges between neighbouring mobile nodes (or groups of mobile nodes). The range-based solutions for mobility characterization consist of a new mobility metric for network-wide performance measurement, two velocity estimators for approximating the inter-node relative speeds, and a new scheme for characterizing the nodal mobility. The new metric and its variants are capable of capturing the mobility of a network as well as predicting the performance. The velocity estimators are used to measure the speed and orientation of a mobile relative to its neighbours, given the presence of a departing node. Based on the velocity estimators, the new scheme for mobility characterization is capable of characterizing the mobility of a node that are associated with topological stability, i. e. the node's speeds, orientations relative to its neighbouring nodes and its past epoch time. iiiBIOPATTERN EU Network of Excellence (EU Contract 508803

Dynamic Rate and Channel Selection in Cognitive Radio Systems

In this paper, we investigate dynamic channel and rate selection in cognitive radio systems which exploit a large number of channels free from primary users. In such systems, transmitters may rapidly change the selected (channel, rate) pair to opportunistically learn and track the pair offering the highest throughput. We formulate the problem of sequential channel and rate selection as an online optimization problem, and show its equivalence to a {\it structured} Multi-Armed Bandit problem. The structure stems from inherent properties of the achieved throughput as a function of the selected channel and rate. We derive fundamental performance limits satisfied by {\it any} channel and rate adaptation algorithm, and propose algorithms that achieve (or approach) these limits. In turn, the proposed algorithms optimally exploit the inherent structure of the throughput. We illustrate the efficiency of our algorithms using both test-bed and simulation experiments, in both stationary and non-stationary radio environments. In stationary environments, the packet successful transmission probabilities at the various channel and rate pairs do not evolve over time, whereas in non-stationary environments, they may evolve. In practical scenarios, the proposed algorithms are able to track the best channel and rate quite accurately without the need of any explicit measurement and feedback of the quality of the various channels.Comment: 19 page