Adaptive reinforcement learning for heterogeneous network selection

Next generation 5G mobile wireless networks will consist of multiple technologies for devices to access the network at the edge. One of the keys to 5G is therefore the ability for device to intelligently select its Radio Access Technology (RAT). Current fully distributed algorithms for RAT selection although guaranteeing convergence to equilibrium states, are often slow, require high exploration times and may converge to undesirable equilibria. In this dissertation, we propose three novel reinforcement learning (RL) frameworks to improve the efficiency of existing distributed RAT selection algorithms in a heterogeneous environment, where users may potentially apply a number of different RAT selection procedures. Although our research focuses on solutions for RAT selection in the current and future mobile wireless networks, the proposed solutions in this dissertation are general and suitable to apply for any large scale distributed multi-agent systems. In the first framework, called RL with Non-positive Regret, we propose a novel adaptive RL for multi-agent non-cooperative repeated games. The main contribution is to use both positive and negative regrets in RL to improve the convergence speed and fairness of the well-known regret-based RL procedure. Significant improvements in performance compared to other related algorithms in the literature are demonstrated. In the second framework, called RL with Network-Assisted Feedback (RLNF), our core contribution is to develop a network feedback model that uses network-assisted information to improve the performance of the distributed RL for RAT selection. RLNF guarantees no-regret payoff in the long-run for any user adopting it, regardless of what other users might do and so can work in an environment where not all users use the same learning strategy. This is an important implementation advantage as RLNF can be implemented within current mobile network standards. In the third framework, we propose a novel adaptive RL-based mechanism for RAT selection that can effectively handle user mobility. The key contribution is to leverage forgetting methods to rapidly react to the changes in the radio conditions when users move. We show that our solution improves the performance of wireless networks and converges much faster when users move compared to the non-adaptive solutions. Another objective of the research is to study the impact of various network models on the performance of different RAT selection approaches. We propose a unified benchmark to compare the performances of different algorithms under the same computational environment. The comparative studies reveal that among all the important network parameters that influence the performance of RAT selection algorithms, the number of base stations that a user can connect to has the most significant impact. This finding provides some guidelines for the proper design of RAT selection algorithms for future 5G. Our evaluation benchmark can serve as a reference for researchers, network developers, and engineers. Overall, the thesis provides different reinforcement learning frameworks to improve the efficiency of current fully distributed algorithms for heterogeneous RAT selection. We prove the convergence of the proposed reinforcement learning procedures using the differential inclusion (DI) technique. The theoretical analyses demonstrate that the use of DI not only provides an effective method to study the convergence properties of adaptive procedures in game-theoretic learning, but also yields a much more concise and extensible proof as compared to the classical approaches.Thesis (Ph.D.) -- University of Adelaide, School of Electrical and Electronic Engineering, 201

Network operator intent : a basis for user-friendly network configuration and analysis

Two important network management activities are configuration (making the network behave in a desirable way) and analysis (querying the network’s state). A challenge common to these activities is specifying operator intent. Seemingly simple configurations such as “no network user should exceed their allocated bandwidth” or questions like “how many network devices are in the library?” are difficult to formulate in practice, e.g. they may require multiple tools (like access control lists, firewalls, databases, or accounting software) and a detailed knowledge of the network. This requires a high degree of expertise and experience, and even then, mistakes are common. An understanding of the core concepts that network operators manipulate and analyse is needed so that more effective, efficient, and user-friendly tools and processes can be created. To address this, we create a taxonomy of languages for configuring networks, and use it to evaluate three such languages to learn how operators can express their intent. We identify factors such as language features, testing, state modeling, documentation, and tool support. Then, we interview network operators to understand what they want to express. We analyse the interviews and identify nine orthogonal dimensions which frequently appear in expressions of operator intent. We use these concepts, and our taxonomy, as the basis for a language for querying both business- and network-domain data. We evaluate our language and find that it reduces the number and complexity of queries needed to answer questions about networks. We also conduct a user study, and find that our language reduces novices’ cognitive load while increasing their accuracy and efficiency. With our language, users better understand how to approach questions, can more easily express themselves, and make fewer mistakes when interpreting data. Overall, we find that operator intent can, at one extreme, be expressed directly, as primitives like flow rules, packet counters, or CLI commands, and at another extreme as human-readable statements which are automatically translated and implemented. The former gives operators precise control, but the latter may be easier to use. We also find that there is more to expressing intent than syntax and semantics as usability, redundancy, state manipulation, and ecosystems all play a role. Our findings also show the importance of incorporating business-domain concepts in network management tools. By understanding operator intent we can reduce errors, improve both human-human and human-computer communication, create more usable tools, and make network operators more effective