Online Learning for Combinatorial Network Optimization with Restless Markovian Rewards

Combinatorial network optimization algorithms that compute optimal structures taking into account edge weights form the foundation for many network protocols. Examples include shortest path routing, minimal spanning tree computation, maximum weighted matching on bipartite graphs, etc. We present CLRMR, the first online learning algorithm that efficiently solves the stochastic version of these problems where the underlying edge weights vary as independent Markov chains with unknown dynamics. The performance of an online learning algorithm is characterized in terms of regret, defined as the cumulative difference in rewards between a suitably-defined genie, and that obtained by the given algorithm. We prove that, compared to a genie that knows the Markov transition matrices and uses the single-best structure at all times, CLRMR yields regret that is polynomial in the number of edges and nearly-logarithmic in time

IEEE 802.15.4.e TSCH-Based Scheduling for Throughput Optimization: A Combinatorial Multi-Armed Bandit Approach

In TSCH, which is a MAC mechanism set of the IEEE 802.15.4e amendment, calculation, construction, and maintenance of the packet transmission schedules are not defined. Moreover, to ensure optimal throughput, most of the existing scheduling methods are based on the assumption that instantaneous and accurate Channel State Information (CSI) is available. However, due to the inevitable errors in the channel estimation process, this assumption cannot be materialized in many practical scenarios. In this paper, we propose two alternative and realistic approaches. In our first approach, we assume that only the statistical knowledge of CSI is available a priori. Armed with this knowledge, the average packet rate on each link is computed and then, using the results, the throughput-optimal schedule for the assignment of (slot-frame) cells to links can be formulated as a max-weight bipartite matching problem, which can be solved efficiently using the well-known Hungarian algorithm. In the second approach, we assume that no CSI knowledge (even statistical) is available at the design stage. For this zero-knowledge setting, we introduce a machine learning-based algorithm by formally modeling the scheduling problem in terms of a combinatorial multi-armed bandit (CMAB) process. Our CMAB-based scheme is widely applicable to many real operational environments, thanks to its reduced reliance on design-time knowledge. Simulation results show that the average throughput obtained by the statistical CSI-based method is within the margin of 15% from the theoretical upper bound associated with perfect instantaneous CSI. The aforesaid margin is around 18% for our learning-theoretic solution