Robust power series algorithm for epistemic uncertainty propagation in Markov chain models

In this article, we develop a new methodology for integrating epistemic uncertainties into the computation of performance measures of Markov chain models. We developed a power series algorithm that allows for combining perturbation analysis and uncertainty analysis in a joint framework. We characterize statistically several performance measures, given that distribution of the model parameter expressing the uncertainty about the exact parameter value is known. The technical part of the article provides convergence result, bounds for the remainder term of the power series, and bounds for the validity region of the approximation. In the algorithmic part of the article, an efficient implementation of the power series algorithm for propagating epistemic uncertainty in queueing models with breakdowns and repairs is discussed. Several numerical examples are presented to illustrate the performance of the proposed algorithm and are compared with the corresponding Monte Carlo simulations ones

Gradient Estimation for a Class of Systems with Bulk Services: A Problem in Public Transportation

This paper deals with a system where batch arrivals wait in a station until a server (a train) is available, at which moment it services all customers in waiting. This is an example of a bulk server, which has many applications in public transportation, telecommunications, computer resource allocation, and multiple access telecommuncation networks, among others. We consider a subway model and focus on a metro line serving a particular metro station. Denote the planned inter-departure time of this line by theta. The metro station is served by several other lines and passengers change trainsat the station. Traveling times of trains are assumed to be given by fixed times and an additional stochastic noise. We perform a sensitivity analysis of the total delay ofpassengers waiting for the "" line with respect to theta. We establish a smoothed perturbation analysis (SPA), a measure--valued differentiation (MVD), and a score function (SF) estimator. Numerical experiments are performed to compare the ensuing estimators. It turns out that the SPA and MVD estimators are intrinsically different and the model presented in this paper may serve as a counter--example to the widespread belief that SPA and MVD yield similar estimators

Efficient updating of node importance in dynamic real-life networks

The analysis of real-life networks, such as the internet, biometrical networks, and social networks, is challenged by the constantly changing structure of these networks. Typically, such networks consist of multiple weakly connected subcomponents and efficiently updating the importance of network nodes, as captured by the ergodic projector of a random walk on these networks, is a challenging task. In this paper, new approximations are introduced that allow to efficiently update the ergodic projector of Markov multi-chains. Properties such as convergence and error bounds for approximations are established. The numerical applicability is illustrated with a real-life social network example

Weak differentiability of product measures

In this paper, we study cost functions over a finite collection of random variables. For these types of models, a calculus of differentiation is developed that allows us to obtain a closed-form expression for derivatives where "differentiation" has to be understood in the weak sense. The technique for proving the results is new and establishes an interesting link between functional analysis and gradient estimation. The key contribution of this paper is a product rule of weak differentiation. In addition, a product rule of weak analyticity is presented that allows for Taylor series approximations of finite products measures. In particular, from characteristics of the individual probability measures, a lower bound (i.e., domain of convergence) can be established for the set of parameter values for which the Taylor series converges to the true value. Applications of our theory to the ruin problem from insurance mathematics and to stochastic activity networks arising in project evaluation review techniques are provided. © 2010 INFORMS

Naïve learning in social networks with random communication

We study social learning in a social network setting where agents receive independent noisy signals about the truth. Agents naïvely update beliefs by repeatedly taking weighted averages of neighbors’ opinions. The weights are fixed in the sense of representing average frequency and intensity of social interaction. However, the way people communicate is random such that agents do not update their belief in exactly the same way at every point in time. Our findings, based on Theorem 1, Corollary 1 and simulated examples, suggest the following. Even if the social network does not privilege any agent in terms of influence, a large society almost always fails to converge to the truth. We conclude that wisdom of crowds seems an illusive concept and bares the danger of mistaking consensus for truth

Series Expansions for Finite-State Markov Chains

This paper provides series expansions of the stationary distribution of a finite Markov chain. This leads to an efficient numerical algorithm for computing the stationary distribution of a finite Markov chain. Numerical examples are given to illustrate the performance of the algorithm

From Data to Stochastic Modeling and Decision Making:What Can We Do Better?

In the past decades we have witnessed a paradigm-shift from scarcity of data to abundance of data. Big data and data analytics have fundamentally reshaped many areas including operations research. In this paper, we discuss how to integrate data with the model-based analysis in a controlled way. Specifically, we consider techniques to quantify input uncertainty and the decision making under input uncertainty. Numerical experiments demonstrate that different ways in decision making may lead to significantly different outcomes in a maintenance problem

Analysis of Measure-Valued Derivatives in a Reinforcement Learning Actor-Critic Framework

Policy gradient methods are successful for a wide range of reinforcement learning tasks. Traditionally, such methods utilize the score function as stochastic gradient estimator. We investigate the effect of replacing the score function with a measure-valued derivative within an on-policy actor-critic algorithm. The hypothesis is that measure-valued derivatives reduce the need for score function variance reduction techniques that are common in policy gradient algorithms. We adapt the actor-critic to measure-valued derivatives and develop a novel algorithm. This method keeps the computational complexity of the measure-valued derivative within bounds by using a parameterized state-value function approximation. We show empirically that measure-valued derivatives have comparable performance to score functions on the environments Pendulum and MountainCar. The empirical results of this study suggest that measure-valued derivatives can serve as low-variance alternative to score functions in on-policy actor-critic and indeed reduce the need for variance reduction techniques.</p

Maximum likelihood estimation by monte carlo simulation:Toward data-driven stochastic modeling

We propose a gradient-based simulated maximum likelihood estimation to estimate unknown parameters in a stochastic model without assuming that the likelihood function of the observations is available in closed form. A key element is to develop Monte Carlo-based estimators for the density and its derivatives for the output process, using only knowledge about the dynamics of the model. We present the theory of these estimators and demonstrate how our approach can handle various types of model structures. We also support our findings and illustrate the merits of our approach with numerical results

Measure-valued differentiation for stationary Markov chains

http://staff.feweb.vu.nl/bheidergot