Online Reinforcement Learning for Dynamic Multimedia Systems

In our previous work, we proposed a systematic cross-layer framework for dynamic multimedia systems, which allows each layer to make autonomous and foresighted decisions that maximize the system's long-term performance, while meeting the application's real-time delay constraints. The proposed solution solved the cross-layer optimization offline, under the assumption that the multimedia system's probabilistic dynamics were known a priori. In practice, however, these dynamics are unknown a priori and therefore must be learned online. In this paper, we address this problem by allowing the multimedia system layers to learn, through repeated interactions with each other, to autonomously optimize the system's long-term performance at run-time. We propose two reinforcement learning algorithms for optimizing the system under different design constraints: the first algorithm solves the cross-layer optimization in a centralized manner, and the second solves it in a decentralized manner. We analyze both algorithms in terms of their required computation, memory, and inter-layer communication overheads. After noting that the proposed reinforcement learning algorithms learn too slowly, we introduce a complementary accelerated learning algorithm that exploits partial knowledge about the system's dynamics in order to dramatically improve the system's performance. In our experiments, we demonstrate that decentralized learning can perform as well as centralized learning, while enabling the layers to act autonomously. Additionally, we show that existing application-independent reinforcement learning algorithms, and existing myopic learning algorithms deployed in multimedia systems, perform significantly worse than our proposed application-aware and foresighted learning methods.Comment: 35 pages, 11 figures, 10 table

Analysis of Power-Down Systems with Five States

We consider a device, which has states ON, OFF and fixed number of intermediate states.In the ON state the device uses full power whereas in the OFF state the device consumes no energy but a constant cost is associated with switching back to ON. Intermediate states use some fraction of energy proportional to the usage time but switching back to the ON state has a constant setup cost depending on the current state. Such systems are widely used to conserve energy, for example to speed scale CPUs, to control data centers, or to manage renewable energy. We analyze such a system in terms of competitive analysis and give a heuristic for finding optimal online algorithms. We then use our approach to discuss five-state systems which are widely used in practice

Competitive Power Down Methods in Green Computing

For the power-down problem one considers a device which has states OFF, ON, and a number of intermediate states. The state of the device can be switched at any time. In the OFF state the device consumes zero energy and in the ON state it works at its full power consumption. The intermediate states consume only some fraction of energy proportional to the usage time but switching back to the ON state has has different constant setup cost depending on the current state. Requests for service (i.e. for when the device has to be in the ON state) are not known in advance, thus power-down problems are studied in the framework of online algorithms, where a system has to react without knowledge of future requests. Online algorithms are analyzed in terms of competitiveness, a measure of performance that compares the solution obtained online with the optimal online solution for the same problem, where the lowest possible competitiveness is best. Power-down mechanisms are widely used to save energy and were one of the first problems to be studied in green computing. They can be used to optimize energy usage in cloud computing, or for scheduling energy supply in the smart grid. However, many approaches are simplistic, and do not work well in practice nor do they have a good theoretical underpinning. In fact, it is surprising that only very few algorithmic techniques exist. This thesis widens the algorithmic base for such problems in a number of ways. We study systems with few states which are especially relevant in real wold applications. We give exact ratios for systems with three and five states. We then introduce a new technique, called “decrease and reset”, where the algorithm automatically attunes itself to the frequency of requests, and gives a better performance for real world inputs than currently existing algorithms. We further refine this approach by a budget-based methods which keeps a tally of gains and losses as requests are processed. We also analyze systems with infinite states and devise several strategies to transition between states. The thesis gives results both in terms of theoretical analysis as well as a result of extensive simulation