Deep Neural Networks (DNNs) have been a large driver and enabler for AI
breakthroughs in recent years. These models have been getting larger in their
attempt to become more accurate and tackle new upcoming use-cases, including
AR/VR and intelligent assistants. However, the training process of such large
models is a costly and time-consuming process, which typically yields a single
model to fit all targets. To mitigate this, various techniques have been
proposed in the literature, including pruning, sparsification or quantization
of the model weights and updates. While able to achieve high compression rates,
they often incur computational overheads or accuracy penalties. Alternatively,
factorization methods have been leveraged to incorporate low-rank compression
in the training process. Similarly, such techniques (e.g.,~SVD) frequently rely
on the computationally expensive decomposition of layers and are potentially
sub-optimal for non-linear models, such as DNNs. In this work, we take a
further step in designing efficient low-rank models and propose Maestro, a
framework for trainable low-rank layers. Instead of regularly applying a priori
decompositions such as SVD, the low-rank structure is built into the training
process through a generalized variant of Ordered Dropout. This method imposes
an importance ordering via sampling on the decomposed DNN structure. Our
theoretical analysis demonstrates that our method recovers the SVD
decomposition of linear mapping on uniformly distributed data and PCA for
linear autoencoders. We further apply our technique on DNNs and empirically
illustrate that Maestro enables the extraction of lower footprint models that
preserve model performance while allowing for graceful accuracy-latency
tradeoff for the deployment to devices of different capabilities.Comment: Under revie