Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Spectrum Estimation and Optimal Secondary User Selection in Cognitive Radio Networks

The high-speed development of wireless communication technology has emerged in the surging insistence on optimal spectrum resources. Nevertheless, in consonance to a contemporary study, most of the assigned frequency encounters notable underutilization as far as Cognitive Radio Network (CRN) is concerned. One important issue correlated with spectrum management is how to properly estimate and allocate the spectrum to a Secondary User (SU) for a highly dynamic environment in an optimal manner with minimum sensing delay. In this paper, a Chebyshev Vector Dynamic Spectrum and Kolmogorov-Smirnov Convolutional Network (CVDS-KSCN) method for dynamic spectrum estimation and optimal secondary user selection in CRN is developed. First, it is proposed to tackle the dynamic spectrum access issue with minimum sensing delay in CRN attaining robust spectrum channel throughput with minimum sensing delay. The spectrum estimation is modeled using the Chebyshev distance-based Harmonious Vector Spectrum Estimation model in a dynamic manner. With the dynamic spectrum estimated results, a Kolmogorov-Smirnov Convolutional Neural Network-based Secondary User Selection model is applied to retrieve optimal secondary users in CRN. The performance of CVDS-KSCN is assessed over numerous key aspects, where simulation results confirm the efficiency of the proposed method in achieving high reliable spectrum estimation and Secondary User selection. It is expressive in the simulation results that the proposed CVDS-KSCN method can achieve a good probability of throughput and reduction in sensing delay during Secondary User Selection with low probability of false alarm. The results show that the proposed method outperfroms the DRS and EFAHP algorithms quantitatively in terms of four parameters, namely throughput, sensing delay, false alarm percentage and Secondary User Selection Time