3 research outputs found
Q-greedyUCB: a New Exploration Policy for Adaptive and Resource-efficient Scheduling
This paper proposes a learning algorithm to find a scheduling policy that
achieves an optimal delay-power trade-off in communication systems.
Reinforcement learning (RL) is used to minimize the expected latency for a
given energy constraint where the environments such as traffic arrival rates or
channel conditions can change over time. For this purpose, this problem is
formulated as an infinite-horizon Markov Decision Process (MDP) with
constraints. To handle the constrained optimization problem, we adopt the
Lagrangian relaxation technique to solve it. Then, we propose a variant of
Q-learning, Q-greedyUCB that combines Q-learning for \emph{average} reward
algorithm and Upper Confidence Bound (UCB) policy to solve this decision-making
problem. We prove that the Q-greedyUCB algorithm is convergent through
mathematical analysis. Simulation results show that Q-greedyUCB finds an
optimal scheduling strategy, and is more efficient than Q-learning with the
-greedy and Average-payoff RL algorithm in terms of the cumulative
reward (i.e., the weighted sum of delay and energy) and the convergence speed.
We also show that our algorithm can reduce the regret by up to 12% compared to
the Q-learning with the -greedy and Average-payoff RL algorithm
Adaptive Power and Rate Control for Real-time Status Updating over Fading Channels
Age of Information (AoI) has attracted much attention recently due to its
capability of characterizing the freshness of information. To improve
information freshness over fading channels, efficient scheduling methods are
highly desired for wireless transmissions. However, due to the channel
instability and arrival randomness, optimizing AoI is very challenging. In this
paper, we are interested in the AoI-optimal transmissions with rate-adaptive
transmission schemes in a buffer-aware system. More specifically, we utilize a
probabilistic scheduling method to minimize the AoI while satisfying an average
power constraint. By characterizing the probabilistic scheduling policy with a
Constrained Markov Decision Process (CMDP), we formulate a Linear Programming
(LP) problem. Further, a low complexity algorithm is presented to obtain the
optimal scheduling policy, which is proved to belong to a set of
semi-threshold-based policies. Numerical results verify the reduction in
computational complexity and the optimality of semi-threshold-based policy,
which indicates that we can achieve well real-time service with a low
calculating complexity.Comment: Journal versio
Delay-Optimal and Energy-Efficient Communications with Markovian Arrivals
In this paper, delay-optimal and energy-efficient communication is studied
for a single link under Markov random arrivals. We present the optimal tradeoff
between delay and power over Additive White Gaussian Noise (AWGN) channels and
extend the optimal tradeoff for block fading channels. Under time-correlated
traffic arrivals, we develop a cross-layer solution that jointly considers the
arrival rate, the queue length, and the channel state in order to minimize the
average delay subject to a power constraint. For this purpose, we formulate the
average delay and power problem as a Constrained Markov Decision Process
(CMDP). Based on steady-state analysis for the CMDP, a Linear Programming (LP)
problem is formulated to obtain the optimal delay-power tradeoff. We further
show the optimal transmission strategy using a Lagrangian relaxation technique.
Specifically, the optimal adaptive transmission is shown to have a threshold
type of structure, where the thresholds on the queue length are presented for
different transmission rates under the given arrival rates and channel states.
By exploiting the result, we develop a threshold-based algorithm to efficiently
obtain the optimal delay-power tradeoff. We show how a trajectory-sampling
version of the proposed algorithm can be developed without the prior need of
arrival statistics