Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Applications of Repeated Games in Wireless Networks: A Survey

A repeated game is an effective tool to model interactions and conflicts for players aiming to achieve their objectives in a long-term basis. Contrary to static noncooperative games that model an interaction among players in only one period, in repeated games, interactions of players repeat for multiple periods; and thus the players become aware of other players' past behaviors and their future benefits, and will adapt their behavior accordingly. In wireless networks, conflicts among wireless nodes can lead to selfish behaviors, resulting in poor network performances and detrimental individual payoffs. In this paper, we survey the applications of repeated games in different wireless networks. The main goal is to demonstrate the use of repeated games to encourage wireless nodes to cooperate, thereby improving network performances and avoiding network disruption due to selfish behaviors. Furthermore, various problems in wireless networks and variations of repeated game models together with the corresponding solutions are discussed in this survey. Finally, we outline some open issues and future research directions.Comment: 32 pages, 15 figures, 5 tables, 168 reference

Learning-based Decision Making in Wireless Communications

Fueled by emerging applications and exponential increase in data traffic, wireless networks have recently grown significantly and become more complex. In such large-scale complex wireless networks, it is challenging and, oftentimes, infeasible for conventional optimization methods to quickly solve critical decision-making problems. With this motivation, in this thesis, machine learning methods are developed and utilized for obtaining optimal/near-optimal solutions for timely decision making in wireless networks. Content caching at the edge nodes is a promising technique to reduce the data traffic in next-generation wireless networks. In this context, we in the first part of the thesis study content caching at the wireless network edge using a deep reinforcement learning framework with Wolpertinger architecture. Initially, we develop a learning-based caching policy for a single base station aiming at maximizing the long-term cache hit rate. Then, we extend this study to a wireless communication network with multiple edge nodes. In particular, we propose deep actor-critic reinforcement learning based policies for both centralized and decentralized content caching. Next, with the purpose of making efficient use of limited spectral resources, we develop a deep actor-critic reinforcement learning based framework for dynamic multichannel access. We consider both a single-user case and a scenario in which multiple users attempt to access channels simultaneously. In the single-user model, in order to evaluate the performance of the proposed channel access policy and the framework\u27s tolerance against uncertainty, we explore different channel switching patterns and different switching probabilities. In the case of multiple users, we analyze the probabilities of each user accessing channels with favorable channel conditions and the probability of collision. Following the analysis of the proposed learning-based dynamic multichannel access policy, we consider adversarial attacks on it. In particular, we propose two adversarial policies, one based on feed-forward neural networks and the other based on deep reinforcement learning policies. Both attack strategies aim at minimizing the accuracy of a deep reinforcement learning based dynamic channel access agent, and we demonstrate and compare their performances. Next, anomaly detection as an active hypothesis test problem is studied. Specifically, we study deep reinforcement learning based active sequential testing for anomaly detection. We assume that there is an unknown number of abnormal processes at a time and the agent can only check with one sensor in each sampling step. To maximize the confidence level of the decision and minimize the stopping time concurrently, we propose a deep actor-critic reinforcement learning framework that can dynamically select the sensor based on the posterior probabilities. Separately, we also regard the detection of threshold crossing as an anomaly detection problem, and analyze it via hierarchical generative adversarial networks (GANs). In the final part of the thesis, to address state estimation and detection problems in the presence of noisy sensor observations and probing costs, we develop a soft actor-critic deep reinforcement learning framework. Moreover, considering Byzantine attacks, we design a GAN-based framework to identify the Byzantine sensors. To evaluate the proposed framework, we measure the performance in terms of detection accuracy, stopping time, and the total probing cost needed for detection

Information reuse in dynamic spectrum access

Dynamic spectrum access (DSA), where the permission to use slices of radio spectrum is dynamically shifted (in time an in different geographical areas) across various communications services and applications, has been an area of interest from technical and public policy perspectives over the last decade. The underlying belief is that this will increase spectrum utilization, especially since many spectrum bands are relatively unused, ultimately leading to the creation of new and innovative services that exploit the increase in spectrum availability. Determining whether a slice of spectrum, allocated or licensed to a primary user, is available for use by a secondary user at a certain time and in a certain geographic area is a challenging task. This requires 'context information' which is critical to the operation of DSA. Such context information can be obtained in several ways, with different costs, and different quality/usefulness of the information. In this paper, we describe the challenges in obtaining this context information, the potential for the integration of various sources of context information, and the potential for reuse of such information for related and unrelated purposes such as localization and enforcement of spectrum sharing. Since some of the infrastructure for obtaining finegrained context information is likely to be expensive, the reuse of this infrastructure/information and integration of information from less expensive sources are likely to be essential for the economical and technological viability of DSA. © 2013 IEEE

Mobility-aware mechanisms for fog node discovery and selection

The recent development of delay-sensitive applications has led to the emergence of the fog computing paradigm. Within this paradigm, computation nodes present at the edge of the network can act as fog nodes (FNs) capable of processing users' tasks, thus resulting in latency reductions compared to the existing cloud-based execution model. In order to realize the full potential of fog computing, new research questions have arised, mainly due to the dynamic and heterogeneous fog computing context. This thesis focuses on the following questions in particular: How can a user detect the presence of a nearby FN? How should a user on the move adapt its FN discovery strategy, according to its changing context? How should an FN be selected , in the case of user mobility and FN mobility? These questions will be addressed throughout the different contributions of this thesis. The first contribution consists in proposing a discovery solution allowing a user to become aware of the existence of a nearby FN. Using our solution, the FN advertizes its presence using custom WiFi beacons, which will be detected by the user via a scan process. An implementation of this approach has been developed and its evaluation results have shown that it results in a non-negligible energy consumption given its use of WiFi. This has led to our second contribution, which aims at improving the WiFi scan performed in our discovery approach, especially in the case of user mobility. At a first stage, this improvement consisted in embedding information about the topology of the FNs in the beacons the user receives from previous FNs. We have shown that by adapting the scan behavior based on this information, considerable energy savings can be achieved, while guaranteeing a high discovery rate. However, as this approach is associated with a restrictive FN topology structure, we proposed a different alternative, at a second stage. This alternative leverages the history of cellular context information as an indicator allowing the user to infer whether an FN may be present in its current location. If so, the scan will be enabled. Otherwise, it is disabled. The simulation results comparing different classification algorithms have shown that a sequence-based model, such as a hidden-Markov model is able to effectively predict the FN presence in the current user location. While the previous approaches have focused on a sparse FN deployment, our third contribution considers a high density of FNs. Consequently, as there are multiple nearby FNs that can process the user's tasks, it is important to derive a suitable FN selection strategy. This strategy should consider the time-varying set of FNs caused by the user's mobility. Besides, it should minimize the number of switches from one FN to another, in order to maintain a good quality of service. With these considerations in mind, we have shown that an adaptive greedy approach, that selects an FN having a good-enough delay estimate, achieves the best results. Finally, unlike the previous contribution, where the focus has been on FN selection when the user is mobile, our final contribution deals with mobile vehicular FNs (VFNs). Given the mobility of such VFNs, it is important to make the most of their resources, since they are only available for a short time at a given area. So, we propose that, in order to select an appropriate VFN for a given task, a reference roadside unit (RSU) responsible for task assignment can use advice from a neighbor RSU. This advice consists in the VFN that will result in the lowest delay for the current task, based on the experience of the neighbor RSU. The results have shown that, using the provided advice, the reference RSU can observe significant delay reductions. All in all, the proposed contributions have addressed various problems that may arise in a fog computing context and the obtained results can be used to guide the development of the building blocks of future fog computing solutions.El recent desenvolupament d'aplicacions IoT ha comportat l'aparició del paradigma de fog computing. Dins d'aquest paradigma, els nodes de càlcul presents a la vora de la xarxa poden actuar com a “fog nodes'' (FN) capaços de processar les tasques dels usuaris, produint així reduccions de latència en comparació amb el model d'execució basat en núvol. Per assolir tot el potencial del fog computing, han sorgit noves qüestions de recerca, principalment a causa del context dinàmic i heterogeni de fog computing. Aquesta tesi se centra especialment en les qüestions següents: Com pot un usuari detectar la presència d'un FN? Com hauria d’adaptar un usuari en moviment la seva estratègia de descobriment de FN, segons el seu context? Com s’ha de seleccionar un FN, en el cas de la mobilitat dels usuaris i la mobilitat FN? Aquestes preguntes s’abordaran al llarg de les diferents aportacions d’aquesta tesi. La primera contribució consisteix a proposar una solució de descobriment que permeti a l'usuari detectar l’existència d’un FN proper. Mitjançant la nostra solució, un FN anuncia la seva presència mitjançant beacons Wi-Fi personalitzats, que seran detectats per l'usuari mitjançant un procés d’exploració. S'ha desenvolupat una implementació d'aquest enfocament i els seus resultats d’avaluació han demostrat que resulta en un consum d'energia menyspreable donat el seu ús del Wi-Fi. Això ha suposat la nostra segona contribució, que té com a objectiu millorar l’exploració Wi-Fi, especialment en el cas de la mobilitat dels usuaris. En una primera fase, aquesta millora va consistir a incorporar informació sobre la topologia dels FN en les beacons que rep l'usuari dels FN anteriors. Hem demostrat que mitjançant l'adaptació del comportament d'escaneig basat en aquesta informació es pot aconseguir un estalvi considerable d’energia, alhora que es garanteix un índex elevat de descobriment. Tanmateix, com aquest enfocament s'associa a una estructura de topologia FN restrictiva, vam proposar una alternativa diferent, en una segona etapa. Aquesta alternativa aprofita la història de la informació del context cel·lular com a indicador que permet a l'usuari deduir si un FN pot estar present en la seva ubicació. En cas afirmatiu, l'exploració estarà habilitada. Els resultats de la simulació comparant diferents algoritmes de classificació han demostrat que un model basat en seqüències, com un model HMM, és capaç de predir eficaçment la presència de FNs a la ubicació actual de l'usuari. Si bé els enfocaments anteriors s’han centrat en un desplegament escàs de FNs, la nostra tercera contribució considera una alta densitat d'FNs. En conseqüència, com que hi ha múltiples FNs propers que poden processar les tasques de l'usuari, és important derivar una estratègia de selecció de FN adequada. Aquesta estratègia hauria de tenir en compte el conjunt variable de temps causat per la mobilitat de l'usuari. A més, hauria de minimitzar el nombre de canvis d'un FN a un altre, per mantenir una bona qualitat del servei. Tenint en compte aquestes consideracions, hem demostrat que un enfocament codiciós adaptatiu, que selecciona un FN amb una estimació de retard suficient, aconsegueix els millors resultats. Finalment, a diferència de l'aportació anterior, on l'atenció s'ha fixat en la selecció d'FN quan l'usuari és mòbil, la nostra contribució final tracta sobre les FNs per a vehicles mòbils (VFNs). Tenint en compte la mobilitat d’aquests VFNs, és important aprofitar al màxim els seus recursos, ja que només estan disponibles per a un temps curt. Així doncs, proposem que, per seleccionar un VFN adequat per a una tasca, una unitat RSU responsable de l'assignació de tasques pot utilitzar consells d'un RSU veí. Aquest consell consisteix en escollir el VFN que suposarà el menor retard de la tasca actual, en funció de l’experiència del RSU veí. Els resultats han demostrat que ..

Learning and Management for Internet-of-Things: Accounting for Adaptivity and Scalability

Internet-of-Things (IoT) envisions an intelligent infrastructure of networked smart devices offering task-specific monitoring and control services. The unique features of IoT include extreme heterogeneity, massive number of devices, and unpredictable dynamics partially due to human interaction. These call for foundational innovations in network design and management. Ideally, it should allow efficient adaptation to changing environments, and low-cost implementation scalable to massive number of devices, subject to stringent latency constraints. To this end, the overarching goal of this paper is to outline a unified framework for online learning and management policies in IoT through joint advances in communication, networking, learning, and optimization. From the network architecture vantage point, the unified framework leverages a promising fog architecture that enables smart devices to have proximity access to cloud functionalities at the network edge, along the cloud-to-things continuum. From the algorithmic perspective, key innovations target online approaches adaptive to different degrees of nonstationarity in IoT dynamics, and their scalable model-free implementation under limited feedback that motivates blind or bandit approaches. The proposed framework aspires to offer a stepping stone that leads to systematic designs and analysis of task-specific learning and management schemes for IoT, along with a host of new research directions to build on.Comment: Submitted on June 15 to Proceeding of IEEE Special Issue on Adaptive and Scalable Communication Network

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig