RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systems

With the rapid advance of information technology, network systems have become increasingly complex and hence the underlying system dynamics are often unknown or difficult to characterize. Finding a good network control policy is of significant importance to achieve desirable network performance (e.g., high throughput or low delay). In this work, we consider using model-based reinforcement learning (RL) to learn the optimal control policy for queueing networks so that the average job delay (or equivalently the average queue backlog) is minimized. Traditional approaches in RL, however, cannot handle the unbounded state spaces of the network control problem. To overcome this difficulty, we propose a new algorithm, called Reinforcement Learning for Queueing Networks (RL-QN), which applies model-based RL methods over a finite subset of the state space, while applying a known stabilizing policy for the rest of the states. We establish that the average queue backlog under RL-QN with an appropriately constructed subset can be arbitrarily close to the optimal result. We evaluate RL-QN in dynamic server allocation, routing and switching problems. Simulation results show that RL-QN minimizes the average queue backlog effectively

Most Complex Non-Returning Regular Languages

A regular language $L$ is non-returning if in the minimal deterministic finite automaton accepting it there are no transitions into the initial state. Eom, Han and Jir\'askov\'a derived upper bounds on the state complexity of boolean operations and Kleene star, and proved that these bounds are tight using two different binary witnesses. They derived upper bounds for concatenation and reversal using three different ternary witnesses. These five witnesses use a total of six different transformations. We show that for each $n\ge 4$ there exists a ternary witness of state complexity $n$ that meets the bound for reversal and that at least three letters are needed to meet this bound. Moreover, the restrictions of this witness to binary alphabets meet the bounds for product, star, and boolean operations. We also derive tight upper bounds on the state complexity of binary operations that take arguments with different alphabets. We prove that the maximal syntactic semigroup of a non-returning language has $(n-1)^n$ elements and requires at least $\binom{n}{2}$ generators. We find the maximal state complexities of atoms of non-returning languages. Finally, we show that there exists a most complex non-returning language that meets the bounds for all these complexity measures.Comment: 22 pages, 6 figure

Badly approximable numbers over imaginary quadratic fields

We recall the notion of nearest integer continued fractions over the Euclidean imaginary quadratic fields $K$ and characterize the "badly approximable" numbers, ($z$ such that there is a $C(z)>0$ with $|z-p/q|\geq C/|q|^2$ for all $p/q\in K$), by boundedness of the partial quotients in the continued fraction expansion of $z$. Applying this algorithm to "tagged" indefinite integral binary Hermitian forms demonstrates the existence of entire circles in $\mathbb{C}$ whose points are badly approximable over $K$, with effective constants. By other methods (the Dani correspondence), we prove the existence of circles of badly approximable numbers over any imaginary quadratic field, with loss of effectivity. Among these badly approximable numbers are algebraic numbers of every even degree over $\mathbb{Q}$, which we characterize. All of the examples we consider are associated with cocompact Fuchsian subgroups of the Bianchi groups $SL_2(\mathcal{O})$, where $\mathcal{O}$ is the ring of integers in an imaginary quadratic field.Comment: v3: Improved exposition (hopefully), especially in the second half of the pape

Virasoro character identities from the Andrews--Bailey construction

We prove $q$-series identities between bosonic and fermionic representations of certain Virasoro characters. These identities include some of the conjectures made by the Stony Brook group as special cases. Our method is a direct application of Andrews' extensions of Bailey's lemma to recently obtained polynomial identities.Comment: 22 pages. Expanded version with new result

Spiraling of approximations and spherical averages of Siegel transforms

We consider the question of how approximations satisfying Dirichlet's theorem spiral around vectors in $\mathbb{R}^d$. We give pointwise almost everywhere results (using only the Birkhoff ergodic theorem on the space of lattices). In addition, we show that for $\textit{every}$ unimodular lattice, on average, the directions of approximates spiral in a uniformly distributed fashion on the $d-1$ dimensional unit sphere. For this second result, we adapt a very recent proof of Marklof and Str\"ombergsson \cite{MS3} to show a spherical average result for Siegel transforms on $\operatorname{SL}_{d+1}(\mathbb{R})/\operatorname{SL}_{d+1}(\mathbb{Z})$. Our techniques are elementary. Results like this date back to the work of Eskin-Margulis-Mozes \cite{EMM} and Kleinbock-Margulis \cite{KM} and have wide-ranging applications. We also explicitly construct examples in which the directions are not uniformly distributed.Comment: 20 pages, 1 figure. Noteworthy changes from the previous version: New title. New result added (Theorem 1.1). Strengthening of Theorem 1.