Learning Scheduling Algorithms for Data Processing Clusters

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective such as minimizing average job completion time. Off-the-shelf RL techniques, however, cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves the average job completion time over hand-tuned scheduling heuristics by at least 21%, achieving up to 2x improvement during periods of high cluster load

Static assignment of complex stochastic tasks using stochastic majorization

We consider the problem of statically assigning many tasks to a (smaller) system of homogeneous processors, where a task's structure is modeled as a branching process, and all tasks are assumed to have identical behavior. We show how the theory of majorization can be used to obtain a partial order among possible task assignments. Our results show that if the vector of numbers of tasks assigned to each processor under one mapping is majorized by that of another mapping, then the former mapping is better than the latter with respect to a large number of objective functions. In particular, we show how measurements of finishing time, resource utilization, and reliability are all captured by the theory. We also show how the theory may be applied to the problem of partitioning a pool of processors for distribution among parallelizable tasks

Measurements, Models, Systems and Design

531 s.

Evolution of adaptation mechanisms: adaptation energy, stress, and oscillating death

In 1938, H. Selye proposed the notion of adaptation energy and published "Experimental evidence supporting the conception of adaptation energy". Adaptation of an animal to different factors appears as the spending of one resource. Adaptation energy is a hypothetical extensive quantity spent for adaptation. This term causes much debate when one takes it literally, as a physical quantity, i.e. a sort of energy. The controversial points of view impede the systematic use of the notion of adaptation energy despite experimental evidence. Nevertheless, the response to many harmful factors often has general non-specific form and we suggest that the mechanisms of physiological adaptation admit a very general and nonspecific description. We aim to demonstrate that Selye's adaptation energy is the cornerstone of the top-down approach to modelling of non-specific adaptation processes. We analyse Selye's axioms of adaptation energy together with Goldstone's modifications and propose a series of models for interpretation of these axioms. {\em Adaptation energy is considered as an internal coordinate on the `dominant path' in the model of adaptation}. The phenomena of `oscillating death' and `oscillating remission' are predicted on the base of the dynamical models of adaptation. Natural selection plays a key role in the evolution of mechanisms of physiological adaptation. We use the fitness optimization approach to study of the distribution of resources for neutralization of harmful factors, during adaptation to a multifactor environment, and analyse the optimal strategies for different systems of factors

The multiple-function multi-input/multi-output digital controller system for the AFW wind-tunnel model

A real time multiple-function digital controller system was developed for the Active Flexible Wing (AFW) Program. The digital controller system (DCS) allowed simultaneous execution of two control laws: flutter suppression and either roll trim or a rolling maneuver load control. The DCS operated within, but independently of, a slower host operating system environment, at regulated speeds up to 200 Hz. It also coordinated the acquisition, storage, and transfer of data for near real time controller performance evaluation and both open- and closed-loop plant estimation. It synchronized the operation of four different processing units, allowing flexibility in the number, form, functionality, and order of control laws, and variability in the selection of the sensors and actuators employed. Most importantly, the DCS allowed for the successful demonstration of active flutter suppression to conditions approximately 26 percent (in dynamic pressure) above the open-loop boundary in cases when the model was fixed in roll and up to 23 percent when it was free to roll. Aggressive roll maneuvers with load control were achieved above the flutter boundary. The purpose here is to present the development, validation, and wind tunnel testing of this multiple-function digital controller system