14 research outputs found
MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier
We offer a method for one-shot mask-guided image synthesis that allows
controlling manipulations of a single image by inverting a quasi-robust
classifier equipped with strong regularizers. Our proposed method, entitled
MAGIC, leverages structured gradients from a pre-trained quasi-robust
classifier to better preserve the input semantics while preserving its
classification accuracy, thereby guaranteeing credibility in the synthesis.
Unlike current methods that use complex primitives to supervise the process or
use attention maps as a weak supervisory signal, MAGIC aggregates gradients
over the input, driven by a guide binary mask that enforces a strong, spatial
prior. MAGIC implements a series of manipulations with a single framework
achieving shape and location control, intense non-rigid shape deformations, and
copy/move operations in the presence of repeating objects and gives users firm
control over the synthesis by requiring to simply specify binary guide masks.
Our study and findings are supported by various qualitative comparisons with
the state-of-the-art on the same images sampled from ImageNet and quantitative
analysis using machine perception along with a user survey of 100+ participants
that endorse our synthesis quality. Project page at
https://mozhdehrouhsedaghat.github.io/magic.html. Code is available at
https://github.com/mozhdehrouhsedaghat/magicComment: Accepted to the Thirty-Seventh Conference on Artificial Intelligence
(AAAI) 2023 - 12 pages, 9 figure
Quantifying the Preferential Direction of the Model Gradient in Adversarial Training With Projected Gradient Descent
Adversarial training, especially projected gradient descent (PGD), has been
the most successful approach for improving robustness against adversarial
attacks. After adversarial training, gradients of models with respect to their
inputs have a preferential direction. However, the direction of alignment is
not mathematically well established, making it difficult to evaluate
quantitatively. We propose a novel definition of this direction as the
direction of the vector pointing toward the closest point of the support of the
closest inaccurate class in decision space. To evaluate the alignment with this
direction after adversarial training, we apply a metric that uses generative
adversarial networks to produce the smallest residual needed to change the
class present in the image. We show that PGD-trained models have a higher
alignment than the baseline according to our definition, that our metric
presents higher alignment values than a competing metric formulation, and that
enforcing this alignment increases the robustness of models.Comment: Updates for second version: added methods/analysis for multiclass
datasets; added new references found since last submission; removed claims
about interpretability; overall editin
Inverting Adversarially Robust Networks for Image Synthesis
Recent research in adversarially robust classifiers suggests their
representations tend to be aligned with human perception, which makes them
attractive for image synthesis and restoration applications. Despite favorable
empirical results on a few downstream tasks, their advantages are limited to
slow and sensitive optimization-based techniques. Moreover, their use on
generative models remains unexplored. This work proposes the use of robust
representations as a perceptual primitive for feature inversion models, and
show its benefits with respect to standard non-robust image features. We
empirically show that adopting robust representations as an image prior
significantly improves the reconstruction accuracy of CNN-based feature
inversion models. Furthermore, it allows reconstructing images at multiple
scales out-of-the-box. Following these findings, we propose an
encoding-decoding network based on robust representations and show its
advantages for applications such as anomaly detection, style transfer and image
denoising