Energy Sharing for Multiple Sensor Nodes with Finite Buffers

We consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. Sensor nodes periodically sense the random field and generate data, which is stored in the corresponding data queues. The EH source harnesses energy from ambient energy sources and the generated energy is stored in an energy buffer. Sensor nodes receive energy for data transmission from the EH source. The EH source has to efficiently share the stored energy among the nodes in order to minimize the long-run average delay in data transmission. We formulate the problem of energy sharing between the nodes in the framework of average cost infinite-horizon Markov decision processes (MDPs). We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the $\epsilon$-greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization in order to find near optimal energy sharing policies. Through simulations, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method.Comment: 38 pages, 10 figure

Feature Search in the Grassmanian in Online Reinforcement Learning

We consider the problem of finding the best features for value function approximation in reinforcement learning and develop an online algorithm to optimize the mean square Bellman error objective. For any given feature value, our algorithm performs gradient search in the parameter space via a residual gradient scheme and, on a slower timescale, also performs gradient search in the Grassman manifold of features. We present a proof of convergence of our algorithm. We show empirical results using our algorithm as well as a similar algorithm that uses temporal difference learning in place of the residual gradient scheme for the faster timescale updates

IEEE Journal of Selected Topics In Signal Processing : Vol. 7, No. 5, October 2013

1. Feature Search in the Grassmanian in Online Reinforcement Learning / Shalabh Bhatnagar, Vivek S. Borkar, Prabuchandran K.J. 2. Deterministic Sequencing of Exploration and Exploitation for Multi-Armed Bandit Problems / Sattar Vakili, Keqin Liu, Qin Zhao 3. Sequentiality and Adaptivity Gains in Active Hypothesis Testing / Mohammad Naghshvar, Tara Javidi 4. Multistage Adaptive Estimation of Sparse Signals / Dennis Wei, Alfred O. Hero 5. Hypothesis Testing in Feedforward Networks with Broadcast Failures / Zhenliang Zhang, et al. 6. Learning-Based Constraint Satisfaction with Sensing Restrictions / Alessandro Checco, Doughlas J. Leith 7. Distributed Energy-Aware Diffusion Least Mean Squares: game-theoretic learning / Omid Namvar Gharehshiran, Vikram Krishnamurthy 8. Distributed Learning and Multiaccess of On-Off Channels / Shiyao Chen, Lang Tong 9. Winning the Lottery: learning perfect coordination with minimal feedback / William Zame, Jie Xu, Michaela van der Schaar 10. Multiagent Reinforcement Learning Based Spectrum Sensing Policies for Cognitive Radio Networks / Jarmo Lunden, et al. 11. Opportunistic Spectrum Access by Exploiting Primary User Feedbacks in Underlay Cognitive Radio Systems: an optimaly analysis / Kehao Wang, Lin Chen, Quan Liu 12. Maximizing Quality of Information from Multiple Sensor Devices: the exploration vs exploitation tradeoff / Ertugul Necdet Ciftcioglu, Aylin Yener, Michael J. Neely 13. Transmit Power Control Policies for Energy Harvesting Sensors with Retransmissions / Anup Aprem, et al. 14. Robust Reputation Protocol Design for Online Communities: a stochastic stability analysis / Yu Zhang, Michaela van der Schaa