Packet Scheduling Algorithms in LTE/LTE-A cellular Networks: Multi-agent Q-learning Approach

Spectrum utilization is vital for mobile operators. It ensures an efficient use of spectrum bands, especially when obtaining their license is highly expensive. Long Term Evolution (LTE), and LTE-Advanced (LTE-A) spectrum bands license were auctioned by the Federal Communication Commission (FCC) to mobile operators with hundreds of millions of dollars. In the first part of this dissertation, we study, analyze, and compare the QoS performance of QoS-aware/Channel-aware packet scheduling algorithms while using CA over LTE, and LTE-A heterogeneous cellular networks. This included a detailed study of the LTE/LTE-A cellular network and its features, and the modification of an open source LTE simulator in order to perform these QoS performance tests. In the second part of this dissertation, we aim to solve spectrum underutilization by proposing, implementing, and testing two novel multi-agent Q-learning-based packet scheduling algorithms for LTE cellular network. The Collaborative Competitive scheduling algorithm, and the Competitive Competitive scheduling algorithm. These algorithms schedule licensed users over the available radio resources and un-licensed users over spectrum holes. In conclusion, our results show that the spectrum band could be utilized by deploying efficient packet scheduling algorithms for licensed users, and can be further utilized by allowing unlicensed users to be scheduled on spectrum holes whenever they occur

Reinforcement Learning in Self Organizing Cellular Networks

Self-organization is a key feature as cellular networks densify and become more heterogeneous, through the additional small cells such as pico and femtocells. Self- organizing networks (SONs) can perform self-configuration, self-optimization, and self-healing. These operations can cover basic tasks such as the configuration of a newly installed base station, resource management, and fault management in the network. In other words, SONs attempt to minimize human intervention where they use measurements from the network to minimize the cost of installation, configuration, and maintenance of the network. In fact, SONs aim to bring two main factors in play: intelligence and autonomous adaptability. One of the main requirements for achieving such goals is to learn from sensory data and signal measurements in networks. Therefore, machine learning techniques can play a major role in processing underutilized sensory data to enhance the performance of SONs. In the first part of this dissertation, we focus on reinforcement learning as a viable approach for learning from signal measurements. We develop a general framework in heterogeneous cellular networks agnostic to the learning approach. We design multiple reward functions and study different effects of the reward function, Markov state model, learning rate, and cooperation methods on the performance of reinforcement learning in cellular networks. Further, we look into the optimality of reinforcement learning solutions and provide insights into how to achieve optimal solutions. In the second part of the dissertation, we propose a novel architecture based on spatial indexing for system-evaluation of heterogeneous 5G cellular networks. We develop an open-source platform based on the proposed architecture that can be used to study large scale directional cellular networks. The proposed platform is used for generating training data sets of accurate signal-to-interference-plus-noise-ratio (SINR) values in millimeter-wave communications for machine learning purposes. Then, with taking advantage of the developed platform, we look into dense millimeter-wave networks as one of the key technologies in 5G cellular networks. We focus on topology management of millimeter-wave backhaul networks and study and provide multiple insights on the evaluation and selection of proper performance metrics in dense millimeter-wave networks. Finally, we finish this part by proposing a self-organizing solution to achieve k-connectivity via reinforcement learning in the topology management of wireless networks

Network resource allocation policies with energy transfer capabilities

During the last decades, mobile network operators have witnessed an exponential increase in the traffic demand, mainly due to the high request of services from a huge amount of users. The trend is of a further increase in both the traffic demand and the number of connected devices over the next years. The traffic load is expected to have an annual growth rate of 53% for the mobile network alone, and the upcoming industrial era, which will connect different types of devices to the mobile infrastructure including human and machine type communications, will definitely exacerbate such an increasing trend. The current directions anticipate that future mobile networks will be composed of ultra dense deployments of heterogeneous Base Stations (BSs), where BSs using different transmission powers coexist. Accordingly, the traditional Macro BSs layer will be complemented or replaced with multiple overlapping tiers of small BSs (SBSs), which will allow extending the system capacity. However, the massive use of Information and Communication Technology (ICT) and the dense deployment of network elements is going to increase the level of energy consumed by the telecommunication infrastructure and its carbon footprint on the environment. Current estimations indicates that 10% of the worldwide electricity generation is due to the ICT industry and this value is forecasted to reach 51% by 2030, which imply that 23% of the carbon footprint by human activity will be due to ICT. Environmental sustainability is thus a key requirement for designing next generation mobile networks. Recently, the use of Renewable Energy Sources (RESs) for supplying network elements has attracted the attention of the research community, where the interest is driven by the increased efficiency and the reduced costs of energy harvesters and storage devices, specially when installed to supply SBSs. Such a solution has been demonstrated to be environmentally and economically sustainable in both rural and urban areas. However, RESs will entail a higher management complexity. In fact, environmental energy is inherently erratic and intermittent, which may cause a fluctuating energy inflow and produce service outage. A proper control of how the energy is drained and balanced across network elements is therefore necessary for a self-sustainable network design. In this dissertation, we focus on energy harvested through solar panels that is deemed the most appropriate due to the good efficiency of commercial photovoltaic panels as well as the wide availability of the solar source for typical installations. The characteristics of this energy source are analyzed in the first technical part of the dissertation, by considering an approach based on the extraction of features from collected data of solar energy radiation. In the second technical part of the thesis we introduce our proposed scenario. A federation of BSs together with the distributed harvesters and storage devices at the SBS sites form a micro-grid, whose operations are managed by an energy management system in charge of controlling the intermittent and erratic energy budget from the RESs. We consider load control (i.e., enabling sleep mode in the SBSs) as a method to properly manage energy inflow and spending, based on the traffic demand. Moreover, in the third technical part, we introduce the possibility of improving the network energy efficiency by sharing the exceeding energy that may be available at some BS sites within the micro-grid. Finally, a centralized controller based on supervised and reinforcement learning is proposed in the last technical part of the dissertation. The controller is in charge of opportunistically operating the network to achieve efficient utilization of the harvested energy and prevent SBSs blackout.Durante las últimas décadas, los operadores de redes móviles han sido testigos de un aumento exponencial en la demanda de tráfico, principalmente debido a la gran solicitud de servicios de una gran cantidad de usuarios. La tendencia es un aumento adicional tanto en la demanda de tráfico como en la cantidad de dispositivos conectados en los próximos años. Se espera que la carga de tráfico tenga una tasa de crecimiento anual del 53% solo para la red móvil, y la próxima era industrial, que conectará diferentes tipos de dispositivos a la infraestructura móvil, definitivamente exacerbará tal aumento. Las instrucciones actuales anticipan que las redes móviles futuras estarán compuestas por despliegues ultra densos de estaciones base (BS) heterogéneas. En consecuencia, la capa tradicional de Macro BS se complementará o reemplazará con múltiples niveles superpuestos de pequeños BS (SBS), lo que permitirá ampliar la capacidad del sistema. Sin embargo, el uso masivo de la Tecnología de la Información y la Comunicación (TIC) y el despliegue denso de los elementos de la red aumentará el nivel de energía consumida por la infraestructura de telecomunicaciones y su huella de carbono en el medio ambiente. Las estimaciones actuales indican que el 10% de la generación mundial de electricidad se debe a la industria de las TIC y se prevé que este valor alcance el 51% para 2030, lo que implica que el 23% de la huella de carbono por actividad humana se deberá a las TIC. La sostenibilidad ambiental es, por lo tanto, un requisito clave para diseñar redes móviles de próxima generación. Recientemente, el uso de fuentes de energía renovables (RES) para suministrar elementos de red ha atraído la atención de la comunidad investigadora, donde el interés se ve impulsado por el aumento de la eficiencia y la reducción de los costos de los recolectores y dispositivos de almacenamiento de energía, especialmente cuando se instalan para suministrar SBS. Se ha demostrado que dicha solución es ambiental y económicamente sostenible tanto en áreas rurales como urbanas. Sin embargo, las RES conllevarán una mayor complejidad de gestión. De hecho, la energía ambiental es inherentemente errática e intermitente, lo que puede causar una entrada de energía fluctuante y producir una interrupción del servicio. Por lo tanto, es necesario un control adecuado de cómo se drena y equilibra la energía entre los elementos de la red para un diseño de red autosostenible. En esta disertación, nos enfocamos en la energía cosechada a través de paneles solares que se considera la más apropiada debido a la buena eficiencia de los paneles fotovoltaicos comerciales, así como a la amplia disponibilidad de la fuente solar para instalaciones típicas. Las características de esta fuente de energía se analizan en la primera parte técnica de la disertación, al considerar un enfoque basado en la extracción de características de los datos recopilados de radiación de energía solar. En la segunda parte técnica de la tesis presentamos nuestro escenario propuesto. Una federación de BS junto con los cosechadores distribuidos y los dispositivos de almacenamiento forman una microrred, cuyas operaciones son administradas por un sistema de administración de energía a cargo de controlar el presupuesto de energía intermitente y errático de las RES. Consideramos el control de carga como un método para administrar adecuadamente la entrada y el gasto de energía, en función de la demanda de tráfico. Además, en la tercera parte técnica, presentamos la posibilidad de mejorar la eficiencia energética de la red al compartir la energía excedente que puede estar disponible en algunos sitios dentro de la microrred. Finalmente, se propone un controlador centralizado basado en aprendizaje supervisado y de refuerzo en la última parte técnica de la disertación. El controlador está a cargo de operar la red para lograr una utilización eficiente de energía y previene el apagón de SB

Learning and Reasoning Strategies for User Association in Ultra-dense Small Cell Vehicular Networks

Recent vehicular ad hoc networks research has been focusing on providing intelligent transportation services by employing information and communication technologies on road transport. It has been understood that advanced demands such as reliable connectivity, high user throughput, and ultra-low latency required by these services cannot be met using traditional communication technologies. Consequently, this thesis reports on the application of artificial intelligence to user association as a technology enabler in ultra-dense small cell vehicular networks. In particular, the work focuses on mitigating mobility-related concerns and networking issues at different mobility levels by employing diverse heuristic as well as reinforcement learning (RL) methods. Firstly, driven by rapid fluctuations in the network topology and the radio environment, a conventional, three-step sequence user association policy is designed to highlight and explore the impact of vehicle speed and different performance indicators on network quality of service (QoS) and user experience. Secondly, inspired by control-theoretic models and dynamic programming, a real-time controlled feedback user association approach is proposed. The algorithm adapts to the changing vehicular environment by employing derived network performance information as a heuristic, resulting in improved network performance. Thirdly, a sequence of novel RL based user association algorithms are developed that employ variable learning rate, variable rewards function and adaptation of the control feedback framework to improve the initial and steady-state learning performance. Furthermore, to accelerate the learning process and enhance the adaptability and robustness of the developed RL algorithms, heuristically accelerated RL and case-based transfer learning methods are employed. A comprehensive, two-tier, event-based, system level simulator which is an integration of a dynamic vehicular network, a highway, and an ultra-dense small cell network is developed. The model has enabled the analysis of user mobility effects on the network performance across different mobility levels as well as served as a firm foundation for the evaluation of the empirical properties of the investigated approaches

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Energy sustainability of next generation cellular networks through learning techniques

The trend for the next generation of cellular network, the Fifth Generation (5G), predicts a 1000x increase in the capacity demand with respect to 4G, which leads to new infrastructure deployments. To this respect, it is estimated that the energy consumption of ICT might reach the 51% of global electricity production by 2030, mainly due to mobile networks and services. Consequently, the cost of energy may also become predominant in the operative expenses of a Mobile Network Operator (MNO). Therefore, an efficient control of the energy consumption in 5G networks is not only desirable but essential. In fact, the energy sustainability is one of the pillars in the design of the next generation cellular networks. In the last decade, the research community has been paying close attention to the Energy Efficiency (EE) of the radio communication networks, with particular care on the dynamic switch ON/OFF of the Base Stations (BSs). Besides, 5G architectures will introduce the Heterogeneous Network (HetNet) paradigm, where Small BSs (SBSs) are deployed to assist the standard macro BS for satisfying the high traffic demand and reducing the impact on the energy consumption. However, only with the introduction of Energy Harvesting (EH) capabilities the networks might reach the needed energy savings for mitigating both the high costs and the environmental impact. In the case of HetNets with EH capabilities, the erratic and intermittent nature of renewable energy sources has to be considered, which entails some additional complexity. Solar energy has been chosen as reference EH source due to its widespread adoption and its high efficiency in terms of energy produced compared to its costs. To this end, in the first part of the thesis, a harvested solar energy model has been presented based on accurate stochastic Markov processes for the description of the energy scavenged by outdoor solar sources. The typical HetNet scenario involves dense deployments with a high level of flexibility, which suggests the usage of distributed control systems rather than centralized, where the scalability can become rapidly a bottleneck. For this reason, in the second part of the thesis, we propose to model the SBS tier as a Multi-agent Reinforcement Learning (MRL) system, where each SBS is an intelligent and autonomous agent, which learns by directly interacting with the environment and by properly utilizing the past experience. The agents implemented in each SBS independently learn a proper switch ON/OFF control policy, so as to jointly maximize the system performance in terms of throughput, drop rate and energy consumption, while adapting to the dynamic conditions of the environment, in terms of energy inflow and traffic demand. However, MRL might suffer the problem of coordination when finding simultaneously a solution among all the agents that is good for the whole system. In consequence, the Layered Learning paradigm has been adopted to simplify the problem by decomposing it in subtasks. In particular, the global solution is obtained in a hierarchical fashion: the learning process of a subtask is aimed at facilitating the learning of the next higher subtask layer. The first layer implements an MRL approach and it is in charge of the local online optimization at SBS level as function of the traffic demand and the energy incomes. The second layer is in charge of the network-wide optimization and it is based on Artificial Neural Networks aimed at estimating the model of the overall network.Con la llegada de la nueva generación de redes móviles, la quinta generación (5G), se predice un aumento por un factor 1000 en la demanda de capacidad respecto a la 4G, con la consecuente instalación de nuevas infraestructuras. Se estima que el gasto energético de las tecnologías de la información y la comunicación podría alcanzar el 51% de la producción mundial de energía en el año 2030, principalmente debido al impacto de las redes y servicios móviles. Consecuentemente, los costes relacionados con el consumo de energía pasarán a ser una componente predominante en los gastos operativos (OPEX) de las operadoras de redes móviles. Por lo tanto, un control eficiente del consumo energético de las redes 5G, ya no es simplemente deseable, sino esencial. En la última década, la comunidad científica ha enfocado sus esfuerzos en la eficiencia energética (EE) de las redes de comunicaciones móviles, con particular énfasis en algoritmos para apagar y encender las estaciones base (BS). Además, las arquitecturas 5G introducirán el paradigma de las redes heterogéneas (HetNet), donde pequeñas BSs, o small BSs (SBSs), serán desplegadas para ayudar a las grandes macro BSs en satisfacer la gran demanda de tráfico y reducir el impacto en el consumo energético. Sin embargo, solo con la introducción de técnicas de captación de la energía ambiental, las redes pueden alcanzar los ahorros energéticos requeridos para mitigar los altos costes de la energía y su impacto en el medio ambiente. En el caso de las HetNets alimentadas mediante energías renovables, la naturaleza errática e intermitente de esta tipología de energías constituye una complejidad añadida al problema. La energía solar ha sido utilizada como referencia debido a su gran implantación y su alta eficiencia en términos de cantidad de energía producida respecto costes de producción. Por consiguiente, en la primera parte de la tesis se presenta un modelo de captación de la energía solar basado en un riguroso modelo estocástico de Markov que representa la energía capturada por paneles solares para exteriores. El escenario típico de HetNet supondrá el despliegue denso de SBSs con un alto nivel de flexibilidad, lo cual sugiere la utilización de sistemas de control distribuidos en lugar de aquellos que están centralizados, donde la adaptabilidad podría convertirse rápidamente en un reto difícilmente gestionable. Por esta razón, en la segunda parte de la tesis proponemos modelar las SBSs como un sistema multiagente de aprendizaje automático por refuerzo, donde cada SBS es un agente inteligente y autónomo que aprende interactuando directamente con su entorno y utilizando su experiencia acumulada. Los agentes en cada SBS aprenden independientemente políticas de control del apagado y encendido que les permiten maximizar conjuntamente el rendimiento y el consumo energético a nivel de sistema, adaptándose a condiciones dinámicas del ambiente tales como la energía renovable entrante y la demanda de tráfico. No obstante, los sistemas multiagente sufren problemas de coordinación cuando tienen que hallar simultáneamente una solución de forma distribuida que sea buena para todo el sistema. A tal efecto, el paradigma de aprendizaje por niveles ha sido utilizado para simplificar el problema dividiéndolo en subtareas. Más detalladamente, la solución global se consigue de forma jerárquica: el proceso de aprendizaje de una subtarea está dirigido a ayudar al aprendizaje de la subtarea del nivel superior. El primer nivel contempla un sistema multiagente de aprendizaje automático por refuerzo y se encarga de la optimización en línea de las SBSs en función de la demanda de tráfico y de la energía entrante. El segundo nivel se encarga de la optimización a nivel de red del sistema y está basado en redes neuronales artificiales diseñadas para estimar el modelo de todas las BSsPostprint (published version