Machine learning-based performance models are increasingly being used to
build critical job scheduling and application optimization decisions.
Traditionally, these models assume that data distribution does not change as
more samples are collected over time. However, owing to the complexity and
heterogeneity of production HPC systems, they are susceptible to hardware
degradation, replacement, and/or software patches, which can lead to drift in
the data distribution that can adversely affect the performance models. To this
end, we develop continually learning performance models that account for the
distribution drift, alleviate catastrophic forgetting, and improve
generalizability. Our best model was able to retain accuracy, regardless of
having to learn the new distribution of data inflicted by system changes, while
demonstrating a 2x improvement in the prediction accuracy of the whole data
sequence in comparison to the naive approach.Comment: Presented at Workshop on Machine Learning for Systems at 36th
Conference on Neural Information Processing Systems (NeurIPS 2022