Reinforcement Learning for Resource Allocation in Steerable Laser-based Optical Wireless Systems

Vertical Cavity Surface Emitting Lasers (VCSELs) have demonstrated suitability for data transmission in indoor optical wireless communication (OWC) systems due to the high modulation bandwidth and low manufacturing cost of these sources. Specifically, resource allocation is one of the major challenges that can affect the performance of multi-user optical wireless systems. In this paper, an optimisation problem is formulated to optimally assign each user to an optical access point (AP) composed of multiple VCSELs within a VCSEL array at a certain time to maximise the signal to interference plus noise ratio (SINR). In this context, a mixed-integer linear programming (MILP) model is introduced to solve this optimisation problem. Despite the optimality of the MILP model, it is considered impractical due to its high complexity, high memory and full system information requirements. Therefore, reinforcement Learning (RL) is considered, which recently has been widely investigated as a practical solution for various optimization problems in cellular networks due to its ability to interact with environments with no previous experience. In particular, a Q-learning (QL) algorithm is investigated to perform resource management in a steerable VCSEL-based OWC systems. The results demonstrate the ability of the QL algorithm to achieve optimal solutions close to the MILP model. Moreover, the adoption of beam steering, using holograms implemented by exploiting liquid crystal devices, results in further enhancement in the performance of the network considered

Q-learning algorithm for resource allocation in WDMA-based optical wireless communication networks

Visible Light Communication (VLC) has been widely investigated during the last decade due to its ability to provide high data rates with low power consumption. In general, resource management is an important issue in cellular networks that can highly effect their performance. In this paper, an optimisation problem is formulated to assign each user to an optimal access point and a wavelength at a given time. This problem can be solved using mixed integer linear programming (MILP). However, using MILP is not considered a practical solution due to its complexity and memory requirements. In addition, accurate information must be provided to perform the resource allocation. Therefore, the optimisation problem is reformulated using reinforcement learning (RL), which has recently received tremendous interest due to its ability to interact with any environment without prior knowledge. In this paper, the resource allocation optimisation problem in VLC systems is investigated using the basic Q-learning algorithm. Two scenarios are simulated to compare the results with the previously proposed MILP model. The results demonstrate the ability of the Q-learning algorithm to provide optimal solutions close to the MILP model without prior knowledge of the system

Resource Allocation in Uplink NOMA-IoT Networks: A Reinforcement-Learning Approach

Non-orthogonal multiple access (NOMA) exploits the potential of the power domain to enhance the connectivity for the Internet of Things (IoT). Due to time-varying communication channels, dynamic user clustering is a promising method to increase the throughput of NOMA-IoT networks. This paper develops an intelligent resource allocation scheme for uplink NOMA-IoT communications. To maximise the average performance of sum rates, this work designs an efficient optimization approach based on two reinforcement learning algorithms, namely deep reinforcement learning (DRL) and SARSA-learning. For light traffic, SARSA-learning is used to explore the safest resource allocation policy with low cost. For heavy traffic, DRL is used to handle traffic-introduced huge variables. With the aid of the considered approach, this work addresses two main problems of fair resource allocation in NOMA techniques: 1) allocating users dynamically and 2) balancing resource blocks and network traffic. We analytically demonstrate that the rate of convergence is inversely proportional to network sizes. Numerical results show that: 1) Compared with the optimal benchmark scheme, the proposed DRL and SARSA-learning algorithms have lower complexity with acceptable accuracy and 2) NOMA-enabled IoT networks outperform the conventional orthogonal multiple access based IoT networks in terms of system throughput

Optimal Power Allocation for Integrated Visible Light Positioning and Communication System with a Single LED-Lamp

In this paper, we investigate an integrated visible light positioning and communication (VLPC) system with a single LED-lamp. First, by leveraging the fact that the VLC channel model is a function of the receiver's location, we propose a system model that estimates the channel state information (CSI) based on the positioning information without transmitting pilot sequences. Second, we derive the Cramer-Rao lower bound (CRLB) on the positioning error variance and a lower bound on the achievable rate with on-off keying modulation. Third, based on the derived performance metrics, we optimize the power allocation to minimize the CRLB, while satisfying the rate outage probability constraint. To tackle this non-convex optimization problem, we apply the worst-case distribution of the Conditional Value-at-Risk (CVaR) and the block coordinate descent (BCD) methods to obtain the feasible solutions. Finally, the effects of critical system parameters, such as outage probability, rate threshold, total power threshold, are revealed by numerical results.Comment: 13 pages, 14 figures, accepted by IEEE Transactions on Communication

An Intelligent Management System for Hybrid Network between Visible Light Communication and Radio Frequency

This thesis investigates the challenges and potential solutions associated with hybrid Visible Light Communication (VLC) and Radio Frequency (RF) systems for indoor network environments. The rapid development of VLC technology, characterized by its high data rates, energy efficiency, and inherent security features, offers promising opportunities to complement RF networks in providing seamless connectivity and improved performance. However, integrating VLC and RF technologies effectively requires addressing a range of research and engineering challenges, including network coexistence, handover mechanisms, resource allocation, localization, and standardization.We begin by conducting a comprehensive literature review encompassing existing research, technologies, and solutions related to hybrid VLC/RF architectures, handover management, indoor localization techniques, and the challenges faced by these systems. This background provides a solid foundation for understanding the current state-of-the-art and identifying research gaps in the field of hybrid VLC/RF networks.Next, we propose a novel hybrid network architecture that integrates VLC and RF communication systems to enhance their strengths while mitigating their weaknesses. We discuss various types of hybrid VLC/RF architectures found in the literature and present our proposed design, which addresses the identified challenges through innovative strategies and mechanisms.To improve system performance in our hybrid system, we develop an enhanced priority feedback channel that optimizes the traffic priority based on user preferences and network conditions. This approach minimizes service disruptions, reduces latency, and maintains user Quality of Experience (QoE)\nomenclature{QoE}{Quality of Experience}.Furthermore, we introduce a novel intelligent management system architecture tailored for hybrid VLC/RF networks. This system employs advanced algorithms and techniques to optimize resource allocation, load balancing, localization, and handover management, ensuring efficient operation and seamless connectivity.We evaluate the performance of our proposed solutions through extensive simulations and testbed experiments, considering different network scenarios and metrics. The results demonstrate significant improvements in terms of data rate, latency, handover success rate, and localization accuracy, validating the effectiveness of our proposed architecture and management system.Lastly, we explore several real-world applications and case studies of our intelligent management system in various indoor environments, such as retail stores, offices, and hospitals. These examples illustrate the practical benefits of our solution in enhancing customer experiences, optimizing operational efficiency, facilitating targeted marketing, and improving energy management.In conclusion, this thesis contributes to the advancement of hybrid VLC/RF networks by proposing an innovative architecture and intelligent management system that address the key challenges faced by these systems in indoor environments. The findings and solutions presented in this work provided the backbone for the future research and development efforts aimed at fully harnessing the potential of VLC technology in combination with RF networks

Reinforcement learning-based intelligent resource allocation for integrated VLCP systems

In this letter, an intelligent resource allocation framework based on model-free reinforcement learning (RL) is first presented for multi-user integrated visible light communication and positioning (VLCP) systems, in order to maximize the sum rate of users while guaranteeing the users' minimum data rates and positioning accuracy constraints. The learning framework can learn the optimal policy under unknown environment's dynamics and the continuous-valued space, and a reward function is proposed to take into account the strict communication and positioning constraints. Moreover, a modified experience replay actor-critic (MERAC) RL approach is proposed to improve the learning efficiency and convergence speed, which efficiently collects the reliable experience and utilizes the most useful knowledge from the memory. Numerical results show that the MERAC approach can effectively learn to satisfy the strict constraints and achieve the fast convergence speed.NRF (Natl Research Foundation, S’pore)Accepted versio

Chemically Engineered Metal Sulfides and Oxides as Electrode Materials for Li-Ion and Photochargeable Batteries

Lithium-ion batteries have dominated battery technologies for portable electronic devices for decades, commanded by intercalation chemistry for electrochemical energy storage while pushing the limits of storage capacity worldwide to the terawatt-hour level. State-of-the-art intercalation cathodes for Li-ion batteries operate within the limits of transition metal oxide electrochemistry. However, conversion-redox processes have rich opportunities for substantially increasing energy densities. Due to the limitations of both the Li content and the extraction of one electron per transition metal, the target energy density of 500 Wh kg-1 of classical layered oxides at the cell level remains elusive. The diversity of compositions that exhibit high reversible capacities following a conversion redox reaction in the solid state has inspired the exploration of new materials for next-generation cathodes for lithium batteries and beyond. While thinking beyond, the electrification of the aviation sector is a game-changer for future transport. Therefore, the requirements on battery technologies must focus even more on high power density, high energy density with fast-charging capability and having low weight and compacted cell design. In addition, battery safety aspects and sustainability of energy materials are key challenges. To address some of the challenges, synthesis and surface engineering of high energy density and fast-charging materials, as well as the development and comprehensive characterization of metal-sulfur batteries for Li-S, Mg-S, and hybrid Li/Mg-S systems, were studied in this thesis. In terms of fast-charging electrode materials, TiNb2O7 was modified by a carbon-coating to improve the charge conduction and specific capacity at high current rates. Further, novel concepts towards photoresponsive cathode materials, such as vanadium pentoxide were investigated in a lithium-ion photo battery. This study reports on the optimization of dual functionality by chemical surface engineering of electrospun vanadium pentoxide fibers as photoresponsive cathodes in lithium-ion batteries. To meet the demand for high energy density in metal-sulfur batteries, lithium-sulfur and mag- nesium-sulfur batteries were explored. For Li-S system, a synthetic approach based on a new molecular precursor [(LiSC2H4)2NCH3] to form lithium sulfide/carbon nano- fibers as cathode was pursued. Suitable cathode materials based on metal sulfides, mainly copper, iron, and copper-iron-sulfides, were investigated in terms of their suitability for rechargeable magnesium batteries due to their high theoretical capacities (Mg: 3,833 mAh/cm3; 2,205 mAh/g) and high abundance in the earth’s crust (23,300 ppm). The influence of their crystal structures, particle morphologies, and nano-sized effects were tested to elucidate and further understand the electrochemical behavior with Mg2+ as the active ion. Introducing a small amount of Li-containing salts to the magnesium electrolyte, resulted in a hybrid electrolyte that showed great potential to improve the electrochemical behavior in combination with metal sulfides following a conversion reaction mechanism