Access Policy Design for Cognitive Secondary Users under a Primary Type-I HARQ Process

In this paper, an underlay cognitive radio network that consists of an arbitrary number of secondary users (SU) is considered, in which the primary user (PU) employs Type-I Hybrid Automatic Repeat Request (HARQ). Exploiting the redundancy in PU retransmissions, each SU receiver applies forward interference cancelation to remove a successfully decoded PU message in the subsequent PU retransmissions. The knowledge of the PU message state at the SU receivers and the ACK/NACK message from the PU receiver are sent back to the transmitters. With this approach and using a Constrained Markov Decision Process (CMDP) model and Constrained Multi-agent MDP (CMMDP), centralized and decentralized optimum access policies for SUs are proposed to maximize their average sum throughput under a PU throughput constraint. In the decentralized case, the channel access decision of each SU is unknown to the other SU. Numerical results demonstrate the benefits of the proposed policies in terms of sum throughput of SUs. The results also reveal that the centralized access policy design outperforms the decentralized design especially when the PU can tolerate a low average long term throughput. Finally, the difficulties in decentralized access policy design with partial state information are discussed

The Linear Program approach in multi-chain Markov Decision Processes revisited

Linear Programming is known to be an important and useful tool for solving Markov Decision Processes (MDP). Its derivation relies on the Dynamic Programming approach, which also serves to solve MDP. However, for Markov Decision Processes with several constraints the only available methods are based on Linear Programs. The aim of this paper is to investigate some aspects of such Linear Programs, related to multi-chain MDPs. We first present a stochastic interpretation of the decision variables that appear in the Linear Programs available in the literature. We then show for the multi-constrained Markov Decision Process that the Linear Program suggested in [9] can be obtained from an equivalent unconstrained Lagrange formulation of the control problem. This shows the connection between the Linear Program approach and the Lagrange approach, that was previously used only for the case of a single constraint [3,14,15]

Bayesian Learning Strategies in Wireless Networks

This thesis collects the research works I performed as a Ph.D. candidate, where the common thread running through all the works is Bayesian reasoning with applications in wireless networks. The pivotal role in Bayesian reasoning is inference: reasoning about what we don’t know, given what we know. When we make inference about the nature of the world, then we learn new features about the environment within which the agent gains experience, as this is what allows us to benefit from the gathered information, thus adapting to new conditions. As we leverage the gathered information, our belief about the environment should change to reflect our improved knowledge. This thesis focuses on the probabilistic aspects of information processing with applications to the following topics: Machine learning based network analysis using millimeter-wave narrow-band energy traces; Bayesian forecasting and anomaly detection in vehicular monitoring networks; Online power management strategies for energy harvesting mobile networks; Beam training and data transmission optimization in millimeter-wave vehicular networks. In these research works, we deal with pattern recognition aspects in real-world data via supervised/unsupervised learning methods (classification, forecasting and anomaly detection, multi-step ahead prediction via kernel methods). Finally, the mathematical framework of Markov Decision Processes (MDPs), which also serves as the basis for reinforcement learning, is introduced, where Partially Observable MDPs use the notion of belief to make decisions about the state of the world in millimeter-wave vehicular networks. The goal of this thesis is to investigate the considerable potential of inference from insightful perspectives, detailing the mathematical framework and how Bayesian reasoning conveniently adapts to various research domains in wireless networks