An Estimator for the Sensitivity to Perturbations of Deep Neural Networks

For Deep Neural Networks (DNNs) to become useful in safety-critical applications, such as self-driving cars and disease diagnosis, they must be stable to perturbations in input and model parameters. Characterizing the sensitivity of a DNN to perturbations is necessary to determine minimal bit-width precision that may be used to safely represent the network. However, no general result exists that is capable of predicting the sensitivity of a given DNN to round-off error, noise, or other perturbations in input. This paper derives an estimator that can predict such quantities. The estimator is derived via inequalities and matrix norms, and the resulting quantity is roughly analogous to a condition number for the entire neural network. An approximation of the estimator is tested on two Convolutional Neural Networks, AlexNet and VGG-19, using the ImageNet dataset. For each of these networks, the tightness of the estimator is explored via random perturbations and adversarial attacks.Comment: Actual work and paper concluded in January 201

Estimating the minimum bit-width precision for stable deep neural networks utilizing numerical linear algebra

Understanding the bit-width precision is critical in compact representation of a Deep Neural Network (DNN) model with minimal degradation in the inference accuracy. While DNNs are resilient to small errors and noise as pointed out by many prior sources, there is a need to develop a generic mathematical framework for evaluating a given DNN’s sensitivity to input bit-width precision. In this work, we derive a bit-width precision estimator which incorporates the sensitivity of DNN inference accuracy to round-off errors, noise, or other perturbations in inputs. We use the tools of numerical linear algebra, particularly stability analysis, to establish the general bounds that can be imposed on the precision. Random perturbations and ‘worst-case’ perturbations, via adversarial attacks, are applied to determine the tightness of the proposed estimator. The experimental results on AlexNet and VGG-19 showed that minimum 11 bits of input bit-width precision is required for these networks to remain stable. The proposed bit-width precision estimator can enable compact yet highly accurate DNN implementationsElectrical and Computer Engineerin