11,045 research outputs found
Towards Practical Control of Singular Values of Convolutional Layers
In general, convolutional neural networks (CNNs) are easy to train, but their
essential properties, such as generalization error and adversarial robustness,
are hard to control. Recent research demonstrated that singular values of
convolutional layers significantly affect such elusive properties and offered
several methods for controlling them. Nevertheless, these methods present an
intractable computational challenge or resort to coarse approximations. In this
paper, we offer a principled approach to alleviating constraints of the prior
art at the expense of an insignificant reduction in layer expressivity. Our
method is based on the tensor-train decomposition; it retains control over the
actual singular values of convolutional mappings while providing structurally
sparse and hardware-friendly representation. We demonstrate the improved
properties of modern CNNs with our method and analyze its impact on the model
performance, calibration, and adversarial robustness. The source code is
available at: https://github.com/WhiteTeaDragon/practical_svd_convComment: Published as a conference paper at NeurIPS 202
SVD-DIP: Overcoming the Overfitting Problem in DIP-based CT Reconstruction
The deep image prior (DIP) is a well-established unsupervised deep learning
method for image reconstruction; yet it is far from being flawless. The DIP
overfits to noise if not early stopped, or optimized via a regularized
objective. We build on the regularized fine-tuning of a pretrained DIP, by
adopting a novel strategy that restricts the learning to the adaptation of
singular values. The proposed SVD-DIP uses ad hoc convolutional layers whose
pretrained parameters are decomposed via the singular value decomposition.
Optimizing the DIP then solely consists in the fine-tuning of the singular
values, while keeping the left and right singular vectors fixed. We thoroughly
validate the proposed method on real-measured CT data of a lotus root as
well as two medical datasets (LoDoPaB and Mayo). We report significantly
improved stability of the DIP optimization, by overcoming the overfitting to
noise
- …