Achieving efficient and robust multi-channel data learning is a challenging
task in data science. By exploiting low-rankness in the transformed domain,
i.e., transformed low-rankness, tensor Singular Value Decomposition (t-SVD) has
achieved extensive success in multi-channel data representation and has
recently been extended to function representation such as Neural Networks with
t-product layers (t-NNs). However, it still remains unclear how t-SVD
theoretically affects the learning behavior of t-NNs. This paper is the first
to answer this question by deriving the upper bounds of the generalization
error of both standard and adversarially trained t-NNs. It reveals that the
t-NNs compressed by exact transformed low-rank parameterization can achieve a
sharper adversarial generalization bound. In practice, although t-NNs rarely
have exactly transformed low-rank weights, our analysis further shows that by
adversarial training with gradient flow (GF), the over-parameterized t-NNs with
ReLU activations are trained with implicit regularization towards transformed
low-rank parameterization under certain conditions. We also establish
adversarial generalization bounds for t-NNs with approximately transformed
low-rank weights. Our analysis indicates that the transformed low-rank
parameterization can promisingly enhance robust generalization for t-NNs.Comment: 46 pages, accepted to NeurIPS 2023. We have corrected several typos
in the first version (arXiv:2303.00196