Stochastic Lyapunov analysis for consensus algorithms with noisy measurements

Abstract — This paper studies the coordination and consensus of networked agents in an uncertain environment. We consider a group of agents on an undirected graph with fixed topology, but differing from most existing work, each agent has only noisy measurements of its neighbors ’ states. Traditional consensus algorithms in general cannot deal with such a scenario. For consensus seeking, we introduce stochastic approximation type algorithms with a decreasing step size. We present a stochastic Lyaponuv analysis based upon the total mean potential associated with the agents. Subsequently, the so-called direction of invariance is introduced, which combined with the decay property of the stochastic Lyapunov function leads to mean square convergence of the consensus algorithm. I

A bi-level programming approach for trip matrix estimation and traffic control problems with stochastic user equilibrium link flows

This paper deals with two mathematically similar problems in transport network analysis: trip matrix estimation and traffic signal optimisation on congested road networks. These two problems are formulated as bi-level programming problems with stochastic user equilibrium assignment as the second-level programming problem. We differentiate two types of solutions in the combined matrix estimation and stochastic user equilibrium assignment problem (or, the combined signal optimisation and stochastic user equilibrium assignment problem): one is the solution to the bi-level programming problem and the other the mutually consistent solution where the two sub-problems in the combined problem are solved simultaneously. In this paper, we shall concentrate on the bi-level programming approach although we shall also consider mutually consistent solutions so as to contrast the two types of solutions. The purpose of the paper is to present a solution algorithm for the two bi-level programming problems and to test the algorithm on several networks

Multigrid methods for two-player zero-sum stochastic games

We present a fast numerical algorithm for large scale zero-sum stochastic games with perfect information, which combines policy iteration and algebraic multigrid methods. This algorithm can be applied either to a true finite state space zero-sum two player game or to the discretization of an Isaacs equation. We present numerical tests on discretizations of Isaacs equations or variational inequalities. We also present a full multi-level policy iteration, similar to FMG, which allows to improve substantially the computation time for solving some variational inequalities.Comment: 31 page

Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

IGA-based Multi-Index Stochastic Collocation for random PDEs on arbitrary domains

This paper proposes an extension of the Multi-Index Stochastic Collocation (MISC) method for forward uncertainty quantification (UQ) problems in computational domains of shape other than a square or cube, by exploiting isogeometric analysis (IGA) techniques. Introducing IGA solvers to the MISC algorithm is very natural since they are tensor-based PDE solvers, which are precisely what is required by the MISC machinery. Moreover, the combination-technique formulation of MISC allows the straight-forward reuse of existing implementations of IGA solvers. We present numerical results to showcase the effectiveness of the proposed approach.Comment: version 3, version after revisio

Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method can be used to approximate any posterior distribution, provided that it is given in closed form up to the proportionality constant. The approximation can be any distribution in the exponential family or any mixture of such distributions, which means that it can be made arbitrarily precise. Several examples illustrate the speed and accuracy of our approximation method in practice