With the increased usage of AI accelerators on mobile and edge devices,
on-device machine learning (ML) is gaining popularity. Consequently, thousands
of proprietary ML models are being deployed on billions of untrusted devices.
This raises serious security concerns about model privacy. However, protecting
the model privacy without losing access to the AI accelerators is a challenging
problem. In this paper, we present a novel on-device model inference system,
ShadowNet. ShadowNet protects the model privacy with Trusted Execution
Environment (TEE) while securely outsourcing the heavy linear layers of the
model to the untrusted hardware accelerators. ShadowNet achieves this by
transforming the weights of the linear layers before outsourcing them and
restoring the results inside the TEE. The nonlinear layers are also kept secure
inside the TEE. The transformation of the weights and the restoration of the
results are designed in a way that can be implemented efficiently. We have
built a ShadowNet prototype based on TensorFlow Lite and applied it on four
popular CNNs, namely, MobileNets, ResNet-44, AlexNet and MiniVGG. Our
evaluation shows that ShadowNet achieves strong security guarantees with
reasonable performance, offering a practical solution for secure on-device
model inference.Comment: single column, 21 pages (29 pages include appendix), 12 figure