SC2GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space

Generative Adversarial Networks (GANs) can synthesize realistic images, with the learned latent space shown to encode rich semantic information with various interpretable directions. However, due to the unstructured nature of the learned latent space, it inherits the bias from the training data where specific groups of visual attributes that are not causally related tend to appear together, a phenomenon also known as spurious correlations, e.g., age and eyeglasses or women and lipsticks. Consequently, the learned distribution often lacks the proper modelling of the missing examples. The interpolation following editing directions for one attribute could result in entangled changes with other attributes. To address this problem, previous works typically adjust the learned directions to minimize the changes in other attributes, yet they still fail on strongly correlated features. In this work, we study the entanglement issue in both the training data and the learned latent space for the StyleGAN2-FFHQ model. We propose a novel framework SC$^2$GAN that achieves disentanglement by re-projecting low-density latent code samples in the original latent space and correcting the editing directions based on both the high-density and low-density regions. By leveraging the original meaningful directions and semantic region-specific layers, our framework interpolates the original latent codes to generate images with attribute combination that appears infrequently, then inverts these samples back to the original latent space. We apply our framework to pre-existing methods that learn meaningful latent directions and showcase its strong capability to disentangle the attributes with small amounts of low-density region samples added.Comment: Accepted to the Out Of Distribution Generalization in Computer Vision workshop at ICCV202

Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers

Vision Transformers (ViT) have shown their competitive advantages performance-wise compared to convolutional neural networks (CNNs) though they often come with high computational costs. To this end, previous methods explore different attention patterns by limiting a fixed number of spatially nearby tokens to accelerate the ViT's multi-head self-attention (MHSA) operations. However, such structured attention patterns limit the token-to-token connections to their spatial relevance, which disregards learned semantic connections from a full attention mask. In this work, we propose a novel approach to learn instance-dependent attention patterns, by devising a lightweight connectivity predictor module to estimate the connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. As each token only attends to a small number of other tokens, the binarized connectivity masks are often very sparse by nature and therefore provide the opportunity to accelerate the network via sparse computations. Equipped with the learned unstructured attention pattern, sparse attention ViT (Sparsifiner) produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet compared to token sparsity. Our method reduces 48% to 69% FLOPs of MHSA while the accuracy drop is within 0.4%. We also show that combining attention and token sparsity reduces ViT FLOPs by over 60%.Comment: Accepted at CVPR 202

Revealing Ultrafast Two-Electron Transfer Over Tryptophan with Mass Spectrometry

Electron transfer crucial to bioenergetics is ubiquitously present in biological systems but most of them escape from direct observations. By using tryptophan and its derivatives with 1-CH3, 2-CH3, 5-CH3 and 5-OH substitutions as model molecules, we have unambiguously demonstrated successive two-electron transfer to tryptophan as well as electronic and vibrational excited molecular dissociation with mass spectrometry. The ultra-short time delay between two electrons down to sub-attosecond over a distance less than 10 Å was found to cause the strong coupling of electronic and vibrational excitations that was validated by the observation of radical-radical coupling. Intramolecular H migrations along with two-electron transfer was demonstrated with H/D exchange and 13C stable isotope labeling. This proposed technique allows us to observe the ultrafast electron transfer from tryptophan to the heme group in myoglobin proteins. It bridges electron transfer to energy transfer that has been revealed in FRET alone. FeII (porph•‐) and FeI (porph•‐) resulting from one- and two-electron transfer, respectively, have been unambiguously identified<br /

Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers

Augmented reality applications have rapidly spread across online platforms, allowing consumers to virtually try-on a variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by graphics artists to automatically create realistic rendering from a reference product image

Laser Activated Electron Tunneling Based Mass Spectrometric Imaging of Molecular Architectures of Mouse Brain Revealing Regional Specific Lipids

A comprehensive description of overall brain architecture at the molecular level is essential for understanding behavioral and cognitive processes in health and diseases. Although fluorescent labeling of target proteins has been successfully established to visualize a brain connectome, the molecular basis for diverse neurophysiological phenomena remains largely unknown. Here we report a brain-wide, molecular-level, and microscale imaging of endogenous metabolites, in particular, lipids of mouse brain by using laser activated electron tunneling (LAET) and mass spectrometry. In this approach, atomic electron emission along with finely tuned laser beam size provides high resolution that can be down to the sub-micrometer level to display spatial distribution of lipids in mouse brain slices. Electron-directed soft ionization has been achieved through exothermal capture of tunneling photoelectrons as well as unpaired electron-initiated chemical bond cleavages. Regionally specific lipids including saturated, mono-unsaturated, and poly-unsaturated fatty acids as well as other lipids, which may be implicated in neurological signaling pathways, have been discovered by using this laser activated electron tunneling based mass spectrometric imaging (LAET-MSI) technique

Dietary High Zinc Oxide Modulates the Microbiome of Ileum and Colon in Weaned Piglets

Dietary zinc oxide (ZnO) at pharmacological level has been widely used to prevent and treat diarrhea in weaning piglets. Despite its importance for promoting animal health and performance, the influence of microbiome profiles in intestinal tracts by ZnO needs to be comprehensively investigated. In this study, we conducted a comparative microbial community analysis in the ileum and colon of piglets fed by either control diet, high ZnO (3,000 mg/kg) supplement or antibiotics (300 mg/kg chlortetracycline and 60 mg/kg colistin sulfate) supplement. Our results showed that both high dietary ZnO and in-feed antibiotics supplementations significantly increased 5 phyla of Spirochaetes, Tenericutes, Euryarchaeota, Verrucomicrobia, TM7, and reduced 1 phyla of Chlamydiae in ileal digesta. The relative abundance of opportunistic pathogens Campylobacterales were decreased while Enterobacteriales were increased in ZnO or antibiotics-supplemented group when compared to the control. In the colon, the phyla Euryarchaeota, the genus Methanobrevibacter, and the species Methanobrevibacter smithii were drastically increased by high dietary ZnO supplementation when compared with other groups. The microbial functional prediction analysis showed that high dietary ZnO and in-feed antibiotics had a higher abundance of transporter pathway enrichment in the ileum when compared with the control. While in the colon high dietary ZnO had a higher abundant enrichment of methane metabolism involving energy supply when compared with other groups. Both high dietary ZnO and antibiotics increased the microbiota diversity of ileal digesta while they decreased the microbiota diversity of the colonic digesta. Collectively, these results suggested that dietary ZnO and in-feed antibiotics supplementations presented similar effect on ileal microbiota, and mainly affected the non-predominant microbiota

Fabrication and Characterization of an Optimized Low-Loss Two-Mode Fiber for Optoacoustic Sensing

An optimized multi-step index (MSI) 2-LP-mode fiber is proposed and fabricated with low propagation loss of 0.179 dB/km, low intermodal crosstalk and excellent bend resistance. We experimentally clarified the characteristics of backward Brillouin scattering (BBS) and forward Brillouin scattering (FBS) induced by radial acoustic modes (R0,m) in the fabricated MSI 2-LP-mode fiber, respectively. Via the use of this two-mode fiber, we demonstrated a novel discriminative measurement method of temperature and acoustic impedance based on BBS and FBS, achieving improved experimental measurement uncertainties of 0.2 °C and 0.019 kg/(s·mm2) for optoacoustic chemical sensing. The low propagation loss of the sensing fiber and the new measurement method based on both BBS and FBS may pave the way for long-distance and high spatial resolution distributed fiber sensors