Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search

Increasingly complex and diverse deep neural network (DNN) models necessitate distributing the execution across multiple devices for training and inference tasks, and also require carefully planned schedules for performance. However, existing practices often rely on predefined schedules that may not fully exploit the benefits of emerging diverse model-aware operator placement strategies. Handcrafting high-efficiency schedules can be challenging due to the large and varying schedule space. This paper presents Tessel, an automated system that searches for efficient schedules for distributed DNN training and inference for diverse operator placement strategies. To reduce search costs, Tessel leverages the insight that the most efficient schedules often exhibit repetitive pattern (repetend) across different data inputs. This leads to a two-phase approach: repetend construction and schedule completion. By exploring schedules for various operator placement strategies, Tessel significantly improves both training and inference performance. Experiments with representative DNN models demonstrate that Tessel achieves up to 5.5x training performance speedup and up to 38% inference latency reduction.Comment: The paper is accepted by HPCA 202

Scheduling task dependence graphs with variable task execution times onto heterogeneous multiprocessors

ABSTRACT We present a statistical optimization approach for scheduling a task dependence graph with variable task execution times onto a heterogeneous multiprocessor system. Scheduling methods in the presence of variations typically rely on worst-case timing estimates for hard real-time applications, or average-case analysis for other applications. However, a large class of soft real-time applications require only statistical guarantees on latency and throughput. We present a general statistical model that captures the probability distributions of task execution times as well as the correlations of execution times of different tasks. We use a Monte Carlo based technique to perform makespan analysis of different schedules based on this model. This approach can be used to analyze the variability present in a variety of soft real-time applications, including a H.264 video processing application. We present two scheduling algorithms based on statistical makespan analysis. The first is a heuristic based on a critical path analysis of the task dependence graph. The other is a simulated annealing algorithm using incremental timing analysis. Both algorithms take as input the required statistical guarantee, and can thus be easily re-used for different required guarantees. We show that optimization methods based on statistical analysis show a 25-30% improvement in makespan over methods based on static worst-case analysis

Statistical modelling and estimation of solar radiation.

M.Sc. University of KwaZulu-Natal, Durban 2014.Solar radiation is a primary driving force behind a number of solar energy applications such as photovoltaic systems for electricity generation amongst others. Hence, the accurate modelling and prediction of the solar flux incident at a particular location, is essential for the design and performance prediction of solar energy conversion systems. In this regard, literature shows that time series models such as the Box-Jenkins Seasonal/Non-seasonal Autoregressive Integrated Moving Average (S/ARIMA) stochastic models have considerable efficacy to describe, monitor and forecast solar radiation data series at various sites on the earths surface (see e.g. Reikard, 2009). This success is attributable to their ability to capture the stochastic component of the irradiance series due to the effects of the ever-changing atmospheric conditions. On the other hand at the top of the atmosphere, there are no such conditions and deterministic models which have been used successfully to model extra-terrestrial solar radiation. One such modelling procedure is the use of a sinusoidal predictor at determined harmonic (Fourier) frequencies to capture the inherent periodicities (seasonalities) due to the diurnal cycle. We combine this deterministic model component and SARIMA models to construct harmonically coupled SARIMA (HCSARIMA) models to model the resulting mixture of stochastic and deterministic components of solar radiation recorded at the earths surface. A comparative study of these two classes of models is undertaken for the horizontal global solar irradiance incident on the solar panels at UKZN Howard College (UKZN HC), located at 29.9º South, 30.98º East with elevation, 151.3m. The results indicated that both SARIMA and HCSARIMA models are good in describing the underlying data generating processes for all data series with respect to different diagnostics. In terms of the predictive ability, the HCSARIMA models generally had a competitive edge over the SARIMA models in most cases. Also, a tentative study of long range dependence (long memory) shows this phenomenon to be inherent in high frequency data series. Therefore autoregressive fractionally integrated moving average (ARFIMA) models are recommended for further studies on high frequency irradiance.Please refer to page xii of thesis for abbreviations that appear in the abstract