Uplink dynamic discrete power control in cellular networks

We consider an uplink power control problem where each mobile wishes to maximize its throughput (which depends on the transmission powers of all mobiles) but has a constraint on the average power consumption. A finite number of power levels are available to each mobile. The decision of a mobile to select a particular power level may depend on its channel state. We consider two frameworks concerning the state information of the channels of other mobiles: (i) the case of full state information and (ii) the case of local state information. In each of the two frameworks, we consider both cooperative as well as non-cooperative power control. We manage to characterize the structure of equilibria policies and, more generally, of best-response policies in the non-cooperative case. We present an algorithm to compute equilibria policies in the case of two non-cooperative players. Finally, we study the case where a malicious mobile, which also has average power constraints, tries to jam the communication of another mobile. Our results are illustrated and validated through various numerical examples

Competitive Selection of Ephemeral Relays in Wireless Networks

International audienceWe consider an opportunistic wireless communication setting, in which two nodes (referred to as forwarders) compete to choose a relay node from a set of relays, as they ephemerally become available (e.g., wake up from a sleep state). Each relay, when it becomes available (or arrives), offers a (possibly different) " reward " to each forwarder. Each forwarder's objective is to minimize a combination of the delay incurred in choosing a relay and the reward offered by the chosen relay. As an example, we develop the reward structure for the specific problem of geographical forwarding over a common set of sleep-wake cycling relays. In general, our model can be considered as a game theoretic variant of the asset selling problem studied in the operations research literature. We study two variants of the generic relay selection problem, namely, the completely observable (CO) and the partially observable (PO) cases. These cases are based on whether a forwarder (in addition to observing its reward) can also observe the reward offered to the other forwarder. Formulating both problems as a two person stochastic game, we characterize the solutions in terms of Nash Equilibrium Policy Pairs (NEPPs). For the CO case we provide a general structure of the NEPPs. For the PO case we prove that there exists an NEPP within the class of threshold policy pairs. Through numerical work, for a one-hop forwarding example we compare the cost performance of various NEPPs with a simple forwarding (SF) policy which causes each forwarder to act as if the other is not present. We find that if the forwarders are not very close then the SF policy suffices. Insights gained from this numerical work are then used in an end-to-end simulation of geographical forwarding in a large network, in which we are concerned with delivery of packets from a tagged source to a sink, in the presence of competition from other packet flows destined to the same sink

Mini-batch forward-backward-forward methods for solving stochastic variational inequalities

International audienceWe develop a new stochastic algorithm for solving pseudo-monotone stochastic variational inequalities. Our method builds on Tseng’s forward-backward- forward (FBF) algorithm, which is known in the deterministic literature to be a valuable alternative to Korpelevich’s extragradient method when solving variational inequalities over a convex and closed set governed by pseudo-monotone, Lipschitz continuous operators. The main computational advantage of Tseng’s algorithm is that it relies only on a single projection step and two independent queries of a stochastic oracle. Our algorithm incorporates a mini-batch sampling mechanism and leads to almost sure (a.s.) convergence to an optimal solution. To the best of our knowledge, this is the first stochastic look-ahead algorithm achieving this by using only a single projection at each iteration

Multi-agent Reinforcement Learning as Applied to Autonomous Systems

Multi-agent reinforcement learning (MARL) is a relatively unexplored area. Existing MARL methods and algorithms often scale poorly to the number of agents or suffer from the issue of non-stationary learning. This thesis aims to develop distributed training methods and algorithms for RL of the multi-agent autonomous systems that ensure scalability and stabilized learning.Specifically, we consider three common paradigms of multi-agent interaction: cooperative/ team, strategic/competitive, and leader-follower settings. We first study game-theoretic RL for the last two settings. Game-theoretic models are inherently distributed, and each agent takes other agents’ responses into account in the resulting game-theoretical equilibrium. Therefore, learning based on the game-theoretic equilibrium can effectively address the issue of non-stationary learning. We investigate Nash Q-learning in the strategic/competitive setting and Stackelberg Q-learning in the leader-follower setting, and apply them successfully to several applications that involve simple yet basic multi-agent interactions. We then propose a distributed deep Q-leaning for the cooperative/team setting, where each agent updates the estimate of her Q-value based on her own reward and her neighbors’ Q-values. We analyze the convergence of the proposed algorithm, characterize its performance gap to the centralized Q-learning, and evaluate it with a cooperative multi-agent navigation task

Inertial game dynamics and applications to constrained optimization

Aiming to provide a new class of game dynamics with good long-term rationality properties, we derive a second-order inertial system that builds on the widely studied "heavy ball with friction" optimization method. By exploiting a well-known link between the replicator dynamics and the Shahshahani geometry on the space of mixed strategies, the dynamics are stated in a Riemannian geometric framework where trajectories are accelerated by the players' unilateral payoff gradients and they slow down near Nash equilibria. Surprisingly (and in stark contrast to another second-order variant of the replicator dynamics), the inertial replicator dynamics are not well-posed; on the other hand, it is possible to obtain a well-posed system by endowing the mixed strategy space with a different Hessian-Riemannian (HR) metric structure, and we characterize those HR geometries that do so. In the single-agent version of the dynamics (corresponding to constrained optimization over simplex-like objects), we show that regular maximum points of smooth functions attract all nearby solution orbits with low initial speed. More generally, we establish an inertial variant of the so-called "folk theorem" of evolutionary game theory and we show that strict equilibria are attracting in asymmetric (multi-population) games - provided of course that the dynamics are well-posed. A similar asymptotic stability result is obtained for evolutionarily stable strategies in symmetric (single- population) games.Comment: 30 pages, 4 figures; significantly revised paper structure and added new material on Euclidean embeddings and evolutionarily stable strategie

A Game-Theoretic Framework to Regulate Freeriding in Inter-Provider Spectrum Sharing

Primary-secondary spectrum sharing is limited in terms of design space, and may not be sufficient to meet the ever-increasing demand of connectivity and high signal quality. The next step to increase spectrum sharing efficiency is to design markets where sharing takes place among primary providers rather than leaving it to the limited case where the primary licensee is idle. Attaining contractual spectrum sharing among primary providers, a.k.a. co-primary or inter-provider sharing, involves additional costs for the users, e.g., roaming fee. Co-primary spectrum sharing without additional charge to the users poses two major challenges: a) regulatory approaches must be introduced to incentivize providers to share spectrum resources, and b) small providers in co-primary spectrum sharing markets may freeride on large providers’ networks as the customers of the small providers may be using the spectrum and infrastructure resources of large providers. Such freeriding opportunities must be minimized to realize the benefits of primary-level sharing. We consider a subsidy-based spectrum sharing (SBSS) market to facilitate co-primary spectrum sharing where providers are explicitly incentivized to share spectrum resources. We focus on minimizing freeriding in SBSS markets and introduce a game-theoretic model to regulate the freeriding. We use the model to explore operational regimes with minimal freeriding