589 research outputs found
UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity
Image reconstruction and captioning from brain activity evoked by visual
stimuli allow researchers to further understand the connection between the
human brain and the visual perception system. While deep generative models have
recently been employed in this field, reconstructing realistic captions and
images with both low-level details and high semantic fidelity is still a
challenging problem. In this work, we propose UniBrain: Unify Image
Reconstruction and Captioning All in One Diffusion Model from Human Brain
Activity. For the first time, we unify image reconstruction and captioning from
visual-evoked functional magnetic resonance imaging (fMRI) through a latent
diffusion model termed Versatile Diffusion. Specifically, we transform fMRI
voxels into text and image latent for low-level information and guide the
backward diffusion process through fMRI-based image and text conditions derived
from CLIP to generate realistic captions and images. UniBrain outperforms
current methods both qualitatively and quantitatively in terms of image
reconstruction and reports image captioning results for the first time on the
Natural Scenes Dataset (NSD) dataset. Moreover, the ablation experiments and
functional region-of-interest (ROI) analysis further exhibit the superiority of
UniBrain and provide comprehensive insight for visual-evoked brain decoding
Enhancing Adversarial Robustness via Score-Based Optimization
Adversarial attacks have the potential to mislead deep neural network
classifiers by introducing slight perturbations. Developing algorithms that can
mitigate the effects of these attacks is crucial for ensuring the safe use of
artificial intelligence. Recent studies have suggested that score-based
diffusion models are effective in adversarial defenses. However, existing
diffusion-based defenses rely on the sequential simulation of the reversed
stochastic differential equations of diffusion models, which are
computationally inefficient and yield suboptimal results. In this paper, we
introduce a novel adversarial defense scheme named ScoreOpt, which optimizes
adversarial samples at test-time, towards original clean data in the direction
guided by score-based priors. We conduct comprehensive experiments on multiple
datasets, including CIFAR10, CIFAR100 and ImageNet. Our experimental results
demonstrate that our approach outperforms existing adversarial defenses in
terms of both robustness performance and inference speed
Purify++: Improving Diffusion-Purification with Advanced Diffusion Models and Control of Randomness
Adversarial attacks can mislead neural network classifiers. The defense
against adversarial attacks is important for AI safety. Adversarial
purification is a family of approaches that defend adversarial attacks with
suitable pre-processing. Diffusion models have been shown to be effective for
adversarial purification. Despite their success, many aspects of diffusion
purification still remain unexplored. In this paper, we investigate and improve
upon three limiting designs of diffusion purification: the use of an improved
diffusion model, advanced numerical simulation techniques, and optimal control
of randomness. Based on our findings, we propose Purify++, a new diffusion
purification algorithm that is now the state-of-the-art purification method
against several adversarial attacks. Our work presents a systematic exploration
of the limits of diffusion purification methods
- …