Self-adaptive online virtual network migration in network virtualization environments

This is the peer reviewed version of the following article: Zangiabady, M, Garcia‐Robledo, A, Gorricho, J‐L, Serrat‐Fernandez, J, Rubio‐Loyola, J. Self‐adaptive online virtual network migration in network virtualization environments. Trans Emerging Tel Tech. 2019; 30:e3692. https://doi.org/10.1002/ett.3692, which has been published in final form at https://doi.org/10.1002/ett.3692. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.In Network Virtualization Environments, the capability of operators to allocate resources in the Substrate Network (SN) to support Virtual Networks (VNs) in an optimal manner is known as Virtual Network Embedding (VNE). In the same context, online VN migration is the process meant to reallocate components of a VN, or even an entire VN among elements of the SN in real time and seamlessly to end-users. Online VNE without VN migration may lead to either over- or under-utilization of the SN resources. However, VN migration is challenging due to its computational cost and the service disruption inherent to VN components reallocation. Online VN migration can reduce migration costs insofar it is triggered proactively, not reactively, at critical times, avoiding the negative effects of both under- and over-triggering. This paper presents a novel online cost-efficient mechanism that self-adaptively learns the exact moments when triggering VN migration is likely to be profitable in the long term. We propose a novel self-adaptive mechanism based on Reinforcement Learning that determines the right trigger online VN migration times, leading to the minimization of migration costs while simultaneously considering the online VNE acceptance ratio.Peer ReviewedPostprint (author's final draft

Fast exploration and learning of latent graphs with aliased observations

We consider the problem of recovering a latent graph where the observations at each node are \emph{aliased}, and transitions are stochastic. Observations are gathered by an agent traversing the graph. Aliasing means that multiple nodes emit the same observation, so the agent can not know in which node it is located. The agent needs to uncover the hidden topology as accurately as possible and in as few steps as possible. This is equivalent to efficient recovery of the transition probabilities of a partially observable Markov decision process (POMDP) in which the observation probabilities are known. An algorithm for efficiently exploring (and ultimately recovering) the latent graph is provided. Our approach is exponentially faster than naive exploration in a variety of challenging topologies with aliased observations while remaining competitive with existing baselines in the unaliased regime

Bayesian Nonparametric Methods for Partially-Observable Reinforcement Learning

Making intelligent decisions from incomplete information is critical in many applications: for example, robots must choose actions based on imperfect sensors, and speech-based interfaces must infer a user’s needs from noisy microphone inputs. What makes these tasks hard is that often we do not have a natural representation with which to model the domain and use for choosing actions; we must learn about the domain’s properties while simultaneously performing the task. Learning a representation also involves trade-offs between modeling the data that we have seen previously and being able to make predictions about new data. This article explores learning representations of stochastic systems using Bayesian nonparametric statistics. Bayesian nonparametric methods allow the sophistication of a representation to scale gracefully with the complexity in the data. Our main contribution is a careful empirical evaluation of how representations learned using Bayesian nonparametric methods compare to other standard learning approaches, especially in support of planning and control. We show that the Bayesian aspects of the methods result in achieving state-of-the-art performance in decision making with relatively few samples, while the nonparametric aspects often result in fewer computations. These results hold across a variety of different techniques for choosing actions given a representation

Resolving perceptual aliasing in the presence of noisy sensors

Agents learning to act in a partially observable domain may need to overcome the problem of perceptual aliasing – i.e., different states that appear similar but require different responses. This problem is exacerbated when the agent’s sensors are noisy, i.e., sensors may produce different observations in the same state. We show that many well-known reinforcement learning methods designed to deal with perceptual aliasing, such as Utile Suffix Memory, finite size history windows, eligibility traces, and memory bits, do not handle noisy sensors well. We suggest a new algorithm, Noisy Utile Suffix Memory (NUSM), based on USM, that uses a weighted classification of observed trajectories. We compare NUSM to the above methods and show it to be more robust to noise.