Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Collaborative beamforming schemes for wireless sensor networks with energy harvesting capabilities

In recent years, wireless sensor networks have attracted considerable attention in the research community. Their development, induced by technological advances in microelectronics, wireless networking and battery fabrication, is mainly motivated by a large number of possible applications such as environmental monitoring, industrial process control, goods tracking, healthcare applications, to name a few. Due to the unattended nature of wireless sensor networks, battery replacement can be either too costly or simply not feasible. In order to cope with this problem and prolong the network lifetime, energy efficient data transmission protocols have to be designed. Motivated by this ultimate goal, this PhD dissertation focuses on the design of collaborative beamforming schemes for wireless sensor networks with energy harvesting capabilities. On the one hand, by resorting to collaborative beamforming, sensors are able to convey a common message to a distant base station, in an energy efficient fashion. On the other, sensor nodes with energy harvesting capabilities promise virtually infinite network lifetime. Nevertheless, in order to realize collaborative beamforming, it is necessary that sensors align their transmitted signals so that they are coherently combined at the destination. Moreover, sensor nodes have to adapt their transmissions according to the amounts of harvested energy over time. First, this dissertation addresses the scenario where two sensor nodes (one of them capable of harvesting ambient energy) collaboratively transmit a common message to a distant base station. In this setting, we show that the optimal power allocation policy at the energy harvesting sensor can be computed independently (i.e., without the knowledge of the optimal policy at the battery operated one). Furthermore, we propose an iterative algorithm that allows us to compute the optimal policy at the battery operated sensor, as well. The insights gained by the aforementioned scenario allow us to generalize the analysis to a system with multiple energy harvesting sensors. In particular, we develop an iterative algorithm which sequentially optimizes the policies for all the sensors until some convergence criterion is satisfied. For the previous scenarios, this PhD dissertation evaluates the impact of total energy harvested, number of sensors and limited energy storage capacity on the system performance. Finally, we consider some practical schemes for carrier synchronization, required in order to implement collaborative beamforming in wireless sensor networks. To that end, we analyze two algorithms for decentralized phase synchronization: (i) the one bit of feedback algorithm previously proposed in the literature; and (ii) a decentralized phase synchronization algorithm that we propose. As for the former, we analyze the impact of additive noise on the beamforming gain and algorithm’s convergence properties, and, subsequently, we propose a variation that performs sidelobe control. As for the latter, the sensors are allowed to choose their respective training timeslots randomly, relieving the base station of the burden associated with centralized coordination. In this context, this PhD dissertation addresses the impact of number of timeslots and additive noise on the achieved received signal strength and throughputEn los últimos años, las redes de sensores inalámbricas han atraído considerable atención en la comunidad investigadora. Su desarrollo, impulsado por recientes avances tecnológicos en microelectrónica y radio comunicaciones, está motivado principalmente por un gran abanico de aplicaciones, tales como: Monitorización ambiental, control de procesos industriales, seguimiento de mercancías, telemedicina, entre otras. En las redes de sensores inalámbricas, es primordial el diseño de protocolos de transmisión energéticamente eficientes ya que no se contempla el reemplazo de baterías debido a su coste y/o complejidad. Motivados por esta problemática, esta tesis doctoral se centra en el diseño de esquemas de conformación de haz distribuidos para redes de sensores, en el que los nodos son capaces de almacenar energía del entorno, lo que en inglés se denomina energy harvesting. En primer lugar, esta tesis doctoral aborda el escenario en el que dos sensores (uno de ellos capaz de almacenar energía del ambiente) transmiten conjuntamente un mensaje a una estación base. En este contexto, se demuestra que la política de asignación de potencia óptima en el sensor con energy harvesting puede ser calculada de forma independiente (es decir, sin el conocimiento de la política óptima del otro sensor). A continuación, se propone un algoritmo iterativo que permite calcular la política óptima en el sensor que funciona con baterías. Este esquema es posteriormente generalizado para el caso de múltiples sensores. En particular, se desarrolla un algoritmo iterativo que optimiza las políticas de todos los sensores secuencialmente. Para los escenarios anteriormente mencionados, esta tesis evalúa el impacto de la energía total cosechada, número de sensores y la capacidad de la batería. Por último, se aborda el problema de sincronización de fase en los sensores con el fin de poder realizar la conformación de haz de forma distribuida. Para ello, se analizan dos algoritmos para la sincronización de fase descentralizados: (i) el algoritmo "one bit of feedback" previamente propuesto en la literatura, y (ii) un algoritmo de sincronización de fase descentralizado que se propone en esta tesis. En el primer caso, se analiza el impacto del ruido aditivo en la ganancia y la convergencia del algoritmo. Además, se propone una variación que realiza el control de lóbulos secundarios. En el segundo esquema, los sensores eligen intervalos de tiempo de forma aleatoria para transmitir y posteriormente reciben información de la estación base para ajustar sus osciladores. En este escenario, esta tesis doctoral aborda el impacto del número de intervalos de tiempo y el ruido aditivo sobre la ganancia de conformación

Reliable Transmission of Short Packets through Queues and Noisy Channels under Latency and Peak-Age Violation Guarantees

This work investigates the probability that the delay and the peak-age of information exceed a desired threshold in a point-to-point communication system with short information packets. The packets are generated according to a stationary memoryless Bernoulli process, placed in a single-server queue and then transmitted over a wireless channel. A variable-length stop-feedback coding scheme---a general strategy that encompasses simple automatic repetition request (ARQ) and more sophisticated hybrid ARQ techniques as special cases---is used by the transmitter to convey the information packets to the receiver. By leveraging finite-blocklength results, the delay violation and the peak-age violation probabilities are characterized without resorting to approximations based on large-deviation theory as in previous literature. Numerical results illuminate the dependence of delay and peak-age violation probability on system parameters such as the frame size and the undetected error probability, and on the chosen packet-management policy. The guidelines provided by our analysis are particularly useful for the design of low-latency ultra-reliable communication systems.Comment: To appear in IEEE journal on selected areas of communication (IEEE JSAC

Distributed Operation of Uncertain Dynamical Cyberphysical Systems

In this thesis we address challenging issues that are faced in the operation of important cyber-physical systems of great current interest. The two particular systems that we address are communication networks and the smart grid. Both systems feature distributed agents making decisions in dynamic uncertain environments. In communication networks, nodes need to decide which packets to transmit, while in the power grid individual generators and loads need to decide how much to pro-duce or consume in a dynamic uncertain environment. The goal in both systems, which also holds for other cyber-physical systems, is to develop distributed policies that perform efficiently in uncertain dynamically changing environments. This thesis proposes an approach of employing duality theory on dynamic stochastic systems in such a way as to develop such distributed operating policies for cyber-physical systems. In the first half of the thesis we examine communication networks. Many cyber-physical systems, e.g., sensor networks, mobile ad-hoc networks, or networked control systems, involve transmitting data over multiple-hops of a communication network. These networks can be unreliable, for example due to the unreliability of the wireless medium. However, real-time applications in cyber-physical systems often require that requisite amounts of data be delivered in a timely manner so that it can be utilized for safely controlling physical processes. Data packets may need to be delivered within their deadlines or at regular intervals without large gaps in packet deliveries when carrying sensor readings. How such packets with deadlines can be scheduled over networks is a major challenge for cyber-physical systems. We develop a framework for routing and scheduling such data packets in a multi-hop network. This framework employs duality theory in such a way that actions of nodes get decoupled, and results in efficient decentralized policies for routing and scheduling such multi-hop communication networks. A key feature of the scheduling policy derived in this work is that the scheduling decisions regarding packets can be made in a fully distributed fashion. A decision regarding the scheduling of an individual packet depend only on the age and location of the packet, and does not require sharing of the queue lengths at various nodes. We examine in more detail a network in which multiple clients stream video packets over shared wireless networks. We are able to derive simple policies of threshold type which maximize the combined QoE of the users. We turn to another important cyber-physical system of great current interest – the emerging smarter grid for electrical power. We address some fundamental problems that arise when attempting to increase the utilization of renewable energy sources. A major challenge is that renewable energy sources are unpredictable in their availability. Utilizing them requires adaptation of demand to their uncertain availability. We address the problem faced by the system operator of coordinating sources of power and loads to balance stochastically time varying supply and demand while maximizing the total utilities of all agents in the system. We develop policies for the system operator that is charged with coordinating such distributed entities through a notion of price. We analyze some models for such systems and employ a combination of duality theory and analysis of stochastic dynamic systems to develop policies that maximize the total utility function of all the agents. We also address the issue of how the size of energy storage facilities should scale with respect to the stochastic behavior of renewables in order to mitigate the unreliability of renewable energy sources