Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first investigate the use of \emph{offline} RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms, including behavior constrained Q-learning (BCQ), conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific RRM problem that aims at maximizing a linear combination {of sum and} 5-percentile rates via user scheduling. We observe that the performance of offline RL for the RRM problem depends critically on the behavior policy used for data collection, and further propose a novel offline RL solution that leverages heterogeneous datasets collected by different behavior policies. We show that with a proper mixture of the datasets, offline RL can produce a near-optimal RL policy even when all involved behavior policies are highly suboptimal.Comment: This paper is the camera ready version for Asilomar 202

Handling Spontaneous Traffic Variations in 5G+ via Offloading onto mmWave-Capable UAV `Bridges'

Unmanned aerial vehicles (UAVs) are increasingly employed for numerous public and civil applications, such as goods delivery, medicine, surveillance, and telecommunications. For the latter, UAVs with onboard communication equipment may help temporarily offload traffic onto the neighboring cells in fifth-generation networks and beyond (5G+). In this paper, we propose and evaluate the use of UAVs traveling over the area of interest to relieve congestion in 5G+ systems under spontaneous traffic fluctuations. To this end, we assess two inherently different offloading schemes, named routed and controlled UAV `bridging'. Using the tools of renewal theory and stochastic geometry, we analytically characterize these schemes in terms of the fraction of traffic demand that can be offloaded onto the UAV `bridge' as our parameter of interest. This framework accounts for the unique features of millimeter-wave (mmWave) radio propagation and city deployment types with potential line-of-sight (LoS) link blockage by buildings. We also introduce enhancements to the proposed schemes that significantly improve the offloading gains. Our findings offer evidence that the UAV `bridges' may be used for efficient traffic offloading in various urban scenarios.Comment: This work has been accepted for publication in the IEEE Transactions on Vehicular Technolog

AMP-Activated Protein Kinase Activation during Cardioplegia-Induced Hypoxia/Reoxygenation Injury Attenuates Cardiomyocytic Apoptosis via Reduction of Endoplasmic Reticulum Stress

Cardioplegic-induced H/R injury results in cardiomyocytic apoptosis. AMPK has been shown to reduce ER stress and the unfolded protein response (UPR). Whether AMPK activation can attenuate cardiomyocytic apoptosis after cardioplegia-induced H/R injury is unknown. Cardiomyocytes were exposed to simulated ischemia by incubation in a hypoxic chamber with intermittent cold cardioplegia solution infusion at 20-minute intervals and subsequently reoxygenated in a normoxic environment. Various doses of AMPK activators (AICAR or metformin) were given 2 days before H/R injury. The cardiomyocytes were harvested after reoxygenation for subsequent examination. With both AMPK activators, the antiapoptotic genes of ER stress and UPR, the subsequent production of proapoptotic proteins was attenuated, and the antiapoptotic proteins were elevated. The activity of the apoptotic effectors of ER stress was also reduced with AMPK activation. Moreover, TUNEL staining showed that AMPK activation significantly reduced the percentage of apoptotic cardiomyocytes after cardioplegia-induced H/R injury. Our results revealed that AMPK activation during cardioplegia-induced H/R injury attenuates cardiomyocytic apoptosis, via enhancement of antiapoptotic and reduction of proapoptotic responses, resulting from lessening ER stress and the UPR. AMPK activation may serve as a future pharmacological target to reduce H/R injury in the clinical setting

Lipoxygenase Pathway Mediates Increases of Airway Resistance and Lung Inflation Induced by Exposure to Nanotitanium Dioxide in Rats

Nanotitanium dioxide particle (nTiO2) inhalation has been reported to induce lung parenchymal injury. After inhalation of nTiO2, we monitored changes in 5-lipoxygenase, endothelial nitric oxide synthase (eNOS), and inducible nitric oxide synthase (iNOS) mRNA in rat lung tissue. Lung function parameters include specific airway resistance (SRaw), peak expiratory flow rate (PEF), functional residual capacity (FRC), and lung compliance (Cchord); blood white blood cell count (WBC), nitric oxide (NO), hydrogen peroxide, and lactic dehydrogenase (LDH); and lung lavage leukotriene C4, interleukin 6 (IL6), tumor necrotic factor α (TNFα), hydroxyl radicals, and NO. Leukotriene receptor antagonist MK571 and 5-lipoxygenase inhibitor MK886 were used for pharmacologic intervention. Compared to control, nTiO2 exposure induced near 5-fold increase in 5-lipoxygenase mRNA expression in lung tissue. iNOS mRNA increased while eNOS mRNA decreased. Lavage leukotriene C4; IL6; TNFα; NO; hydroxyl radicals; and blood WBC, NO, hydrogen peroxide, and LDH levels rose. Obstructive ventilatory insufficiency was observed. MK571 and MK886 both attenuated the systemic inflammation and lung function changes. We conclude that inhaled nTiO2 induces systemic inflammation, cytokine release, and oxidative and nitrosative stress in the lung. The lipoxygenase pathway products, mediated by oxygen radicals and WBC, play a critical role in the obstructive ventilatory insufficiency induced by nTiO2

Inhibition of yes-associated protein suppresses brain metastasis of human lung adenocarcinoma in a murine model.

Yes-associated protein (YAP) is a main mediator of the Hippo pathway and promotes cancer development and progression in human lung cancer. We sought to determine whether inhibition of YAP suppresses metastasis of human lung adenocarcinoma in a murine model. We found that metastatic NSCLC cell lines H2030-BrM3(K-rasG12C mutation) and PC9-BrM3 (EGFRΔexon19 mutation) had a significantly decreased p-YAP(S127)/YAP ratio compared to parental H2030 (K-rasG12C mutation) and PC9 (EGFRΔexon19 mutation) cells (P < .05). H2030-BrM3 cells had significantly increased YAP mRNA and expression of Hippo downstream genes CTGF and CYR61 compared to parental H2030 cells (P < .05). Inhibition of YAP by short hairpin RNA (shRNA) and small interfering RNA (siRNA) significantly decreased mRNA expression in downstream genes CTGF and CYR61 in H2030-BrM3 cells (P < .05). In addition, inhibiting YAP by YAP shRNA significantly decreased migration and invasion abilities of H2030-BrM3 cells (P < .05). We are first to show that mice inoculated with YAP shRNA-transfected H2030-BrM3 cells had significantly decreased metastatic tumour burden and survived longer than control mice (P < .05). Collectively, our results suggest that YAP plays an important role in promoting lung adenocarcinoma brain metastasis and that direct inhibition of YAP by shRNA suppresses H2030-BrM3 cell brain metastasis in a murine model

Aerial Access and Backhaul in mmWave B5G Systems: Performance Dynamics and Optimization

The use of unmanned aerial vehicle (UAV)-based communication in millimeter-wave (mmWave) frequencies to provide on-demand radio access is a promising approach to improve capacity and coverage in beyond-5G (B5G) systems. There are several design aspects to be addressed when optimizing for the deployment of such UAV base stations. As traffic demand of mobile users varies across time and space, dynamic algorithms that correspondingly adjust the UAV locations are essential to maximize performance. In addition to careful tracking of spatio-temporal user/traffic activity, such optimization needs to account for realistic backhaul constraints. In this work, we first review the latest 3GPP activities behind integrated access and backhaul system design, support for UAV base stations, and mmWave radio relaying functionality. We then compare static and mobile UAV-based communication options under practical assumptions on the mmWave system layout, mobility and clusterization of users, antenna array geometry, and dynamic backhauling. We demonstrate that leveraging the UAV mobility to serve moving users may improve the overall system performance even in the presence of backhaul capacity limitations.Comment: 7 pages, 5 figures. This work has been accepted to IEEE Communications Magazine, 201