Learning Algorithms for Minimizing Queue Length Regret

We consider a system consisting of a single transmitter/receiver pair and $N$ channels over which they may communicate. Packets randomly arrive to the transmitter's queue and wait to be successfully sent to the receiver. The transmitter may attempt a frame transmission on one channel at a time, where each frame includes a packet if one is in the queue. For each channel, an attempted transmission is successful with an unknown probability. The transmitter's objective is to quickly identify the best channel to minimize the number of packets in the queue over $T$ time slots. To analyze system performance, we introduce queue length regret, which is the expected difference between the total queue length of a learning policy and a controller that knows the rates, a priori. One approach to designing a transmission policy would be to apply algorithms from the literature that solve the closely-related stochastic multi-armed bandit problem. These policies would focus on maximizing the number of successful frame transmissions over time. However, we show that these methods have $\Omega(\log{T})$ queue length regret. On the other hand, we show that there exists a set of queue-length based policies that can obtain order optimal $O(1)$ queue length regret. We use our theoretical analysis to devise heuristic methods that are shown to perform well in simulation.Comment: 28 Pages, 11 figure

Scheduling for Optimal Rate Allocation in Ad Hoc Networks With Heterogeneous Delay Constraints

This paper studies the problem of scheduling in single-hop wireless networks with real-time traffic, where every packet arrival has an associated deadline and a minimum fraction of packets must be transmitted before the end of the deadline. Using optimization and stochastic network theory we propose a framework to model the quality of service (QoS) requirements under delay constraints. The model allows for fairly general arrival models with heterogeneous constraints. The framework results in an optimal scheduling algorithm which fairly allocates data rates to all flows while meeting long-term delay demands. We also prove that under a simplified scenario our solution translates into a greedy strategy that makes optimal decisions with low complexity

The power-series algorithm applied to cyclic polling systems

Polling Systems;Queueing Theory;operations research

Stochastic Bandit Models for Delayed Conversions

Online advertising and product recommendation are important domains of applications for multi-armed bandit methods. In these fields, the reward that is immediately available is most often only a proxy for the actual outcome of interest, which we refer to as a conversion. For instance, in web advertising, clicks can be observed within a few seconds after an ad display but the corresponding sale --if any-- will take hours, if not days to happen. This paper proposes and investigates a new stochas-tic multi-armed bandit model in the framework proposed by Chapelle (2014) --based on empirical studies in the field of web advertising-- in which each action may trigger a future reward that will then happen with a stochas-tic delay. We assume that the probability of conversion associated with each action is unknown while the distribution of the conversion delay is known, distinguishing between the (idealized) case where the conversion events may be observed whatever their delay and the more realistic setting in which late conversions are censored. We provide performance lower bounds as well as two simple but efficient algorithms based on the UCB and KLUCB frameworks. The latter algorithm, which is preferable when conversion rates are low, is based on a Poissonization argument, of independent interest in other settings where aggregation of Bernoulli observations with different success probabilities is required.Comment: Conference on Uncertainty in Artificial Intelligence, Aug 2017, Sydney, Australi

TOWARD LAYERLESS COOPERATION AND RATE CONTROL IN WIRELESS MULTI-ACCESS CHANNELS

In wireless networks, a transmitted message may successfully reach multiple nodes simultaneously, which is referred to as the Wireless Multicast Advantage. As such, intermediate nodes have the ability to capture the message and then contribute to the communication toward the ultimate destination by cooperatively relaying the received message. This enables cooperative communication, which has been shown to counteract the effects of fading and attenuation in wireless networks. There has been a great deal of work addressing cooperative methods and their resulting benefits, but most of the work to date has focused on physical-layer techniques and on information-theoretic considerations. While compatible with these, the main thrust of this dissertation is to explore a new approach by implementing cooperation at the network layer. First, we illustrate the idea in a multi-hop multi-access wireless network, in which a set of source users generate packets to deliver to a common destination. An opportunistic and dynamic cooperation protocol is proposed at the network level, where users with a better channel to the destination have the capability and option to relay packets from users that are farther afield. The proposed mode of cooperation protocol is new and relies on MAC/Network-level of relaying, but also takes into account physical-layer parameters that determine successful reception at the destination and/or the relay. We explicitly characterize the stable throughput and average delay performance. Our analysis reveals that cooperation at the network layer leads to substantial performance gains for both performance metrics. Next, on top of the network-layer cooperation, we investigate enhanced cooperative techniques that exploit more sophisticated physical-layer properties. Specifically, we consider dynamic decode-and-forward, superposition coding, and multipacket reception capability, and we quantify the extent to which the enhancement techniques can further improve the stable throughput region. Then we revert back to the two-user multi-access channel with single-packet reception, which has been extensively studied in the case of no cooperation. After cooperation is permitted between the two users, we revisit the relationship between the stability region and the throughput region under both scheduled access and random access schemes. Finally, we shift our focus from the packet-level to bit-level multi-access channels. By exploiting the bit-nature of a packet, we create a bridge between traditional physical-layer-based transmission rates and classical MAC/Network-layer-based throughput rates. We first obtain the closed form of the stability region in bits/slot. Then, as a separate, but related issue, we look at the minimum delivery time policy; for any initial queue size vector, the optimal policy that empties all bits in the system within the shortest time is characterized

Channel-Aware Random Access in the Presence of Channel Estimation Errors

In this work, we consider the random access of nodes adapting their transmission probability based on the local channel state information (CSI) in a decentralized manner, which is called CARA. The CSI is not directly available to each node but estimated with some errors in our scenario. Thus, the impact of imperfect CSI on the performance of CARA is our main concern. Specifically, an exact stability analysis is carried out when a pair of bursty sources are competing for a common receiver and, thereby, have interdependent services. The analysis also takes into account the compound effects of the multipacket reception (MPR) capability at the receiver. The contributions in this paper are twofold: first, we obtain the exact stability region of CARA in the presence of channel estimation errors; such an assessment is necessary as the errors in channel estimation are inevitable in the practical situation. Secondly, we compare the performance of CARA to that achieved by the class of stationary scheduling policies that make decisions in a centralized manner based on the CSI feedback. It is shown that the stability region of CARA is not necessarily a subset of that of centralized schedulers as the MPR capability improves.Comment: The material in this paper was presented in part at the IEEE International Symposium on Information Theory, Cambridge, MA, USA, July 201