AN ANALYTICAL METHOD TO CALCULATE THE ERGODIC AND DIFFERENCE MATRICES OF THE DISCOUNTED MARKOV DECISION PROCESSES

In the paper, a theorem about the existence of the ergodic and difference matrices of the finite state discounted Markov decision processes had been formulated and proved. On the basis of this theorem an analytical method to calculate these matrices is presented. The theorem allows the distribution of the overall value into two parts: the so-called "constant" part, which represents a part of the value related to the ergodic matrix and the "variable" part which represents a sum of the difference matrices. On the basis of the mentioned analytical method a new performance index to the discounted optimal control Markov problem is proposed and some interpretation of the received results is given. The proposed new performance index is formulated as a quotient of the distinguished parts of the overall value. The method is illustrated by two simple examples.

Bounding inferences for large-scale continuous-time Markov chains : a new approach based on lumping and imprecise Markov chains

If the state space of a homogeneous continuous-time Markov chain is too large, making inferences becomes computationally infeasible. Fortunately, the state space of such a chain is usually too detailed for the inferences we are interested in, in the sense that a less detailed—smaller—state space suffices to unambiguously formalise the inference. However, in general this so-called lumped state space inhibits computing exact inferences because the corresponding dynamics are unknown and/or intractable to obtain. We address this issue by considering an imprecise continuous-time Markov chain. In this way, we are able to provide guaranteed lower and upper bounds for the inferences of interest, without suffering from the curse of dimensionality

The ergodic decomposition of asymptotically mean stationary random sources

It is demonstrated how to represent asymptotically mean stationary (AMS) random sources with values in standard spaces as mixtures of ergodic AMS sources. This an extension of the well known decomposition of stationary sources which has facilitated the generalization of prominent source coding theorems to arbitrary, not necessarily ergodic, stationary sources. Asymptotic mean stationarity generalizes the definition of stationarity and covers a much larger variety of real-world examples of random sources of practical interest. It is sketched how to obtain source coding and related theorems for arbitrary, not necessarily ergodic, AMS sources, based on the presented ergodic decomposition.Comment: Submitted to IEEE Transactions on Information Theory, Apr. 200

Asymptotic Expansions for Stationary Distributions of Perturbed Semi-Markov Processes

New algorithms for computing of asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes are presented. The algorithms are based on special techniques of sequential phase space reduction, which can be applied to processes with asymptotically coupled and uncoupled finite phase spaces.Comment: 83 page

A monotone Sinai theorem

Sinai proved that a nonatomic ergodic measure-preserving system has any Bernoulli shift of no greater entropy as a factor. Given a Bernoulli shift, we show that any other Bernoulli shift that is of strictly less entropy and is stochastically dominated by the original measure can be obtained as a monotone factor; that is, the factor map has the property that for each point in the domain, its image under the factor map is coordinatewise smaller than or equal to the original point.Comment: Published at http://dx.doi.org/10.1214/14-AOP968 in the Annals of Probability (http://www.imstat.org/aop/) by the Institute of Mathematical Statistics (http://www.imstat.org

Adaptive Power Allocation and Control in Time-Varying Multi-Carrier MIMO Networks

In this paper, we examine the fundamental trade-off between radiated power and achieved throughput in wireless multi-carrier, multiple-input and multiple-output (MIMO) systems that vary with time in an unpredictable fashion (e.g. due to changes in the wireless medium or the users' QoS requirements). Contrary to the static/stationary channel regime, there is no optimal power allocation profile to target (either static or in the mean), so the system's users must adapt to changes in the environment "on the fly", without being able to predict the system's evolution ahead of time. In this dynamic context, we formulate the users' power/throughput trade-off as an online optimization problem and we provide a matrix exponential learning algorithm that leads to no regret - i.e. the proposed transmit policy is asymptotically optimal in hindsight, irrespective of how the system evolves over time. Furthermore, we also examine the robustness of the proposed algorithm under imperfect channel state information (CSI) and we show that it retains its regret minimization properties under very mild conditions on the measurement noise statistics. As a result, users are able to track the evolution of their individually optimum transmit profiles remarkably well, even under rapidly changing network conditions and high uncertainty. Our theoretical analysis is validated by extensive numerical simulations corresponding to a realistic network deployment and providing further insights in the practical implementation aspects of the proposed algorithm.Comment: 25 pages, 4 figure

New approach to Dynamical Monte Carlo Methods: application to an Epidemic Model

A new approach to Dynamical Monte Carlo Methods is introduced to simulate markovian processes. We apply this approach to formulate and study an epidemic Generalized SIRS model. The results are in excellent agreement with the forth order Runge-Kutta Method in a region of deterministic solution. We also demonstrate that purely local interactions reproduce a poissonian-like process at mesoscopic level. The simulations for this case are checked self-consistently using a stochastic version of the Euler Method.Comment: Written with Scientific WorkPlace 3.51 in REVTex4 format, 11 pages with 2 figures in postscript forma