Deep neural networks (DNNs), are widely used in many industries such as image
recognition, supply chain, medical diagnosis, and autonomous driving. However,
prior work has shown the high accuracy of a DNN model does not imply high
robustness (i.e., consistent performances on new and future datasets) because
the input data and external environment (e.g., software and model
configurations) for a deployed model are constantly changing. Hence, ensuring
the robustness of deep learning is not an option but a priority to enhance
business and consumer confidence. Previous studies mostly focus on the data
aspect of model variance. In this article, we systematically summarize DNN
robustness issues and formulate them in a holistic view through two important
aspects, i.e., data and software configuration variances in DNNs. We also
provide a predictive framework to generate representative variances
(counterexamples) by considering both data and configurations for robust
learning through the lens of search-based optimization