Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks

Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. We make several new observations about the top eigenspace of layer-wise Hessian: top eigenspaces for different models have surprisingly high overlap, and top eigenvectors form low rank matrices when they are reshaped into the same shape as the corresponding weight matrix. Towards formally explaining such structures of the Hessian, we show that the new eigenspace structure can be explained by approximating the Hessian using Kronecker factorization; we also prove the low rank structure for random data at random initialization for over-parametrized two-layer neural nets. Our new understanding can explain why some of these structures become weaker when the network is trained with batch normalization. The Kronecker factorization also leads to better explicit generalization bounds.Comment: 60 pages, 30 figures. Main text: 10 pages, 7 figures. First two authors have equal contribution and are in alphabetical orde

C. elegans fatty acid two-hydroxylase regulates intestinal homeostasis by affecting heptadecenoic acid production

Background/Aims: The hydroxylation of fatty acids at the C-2 position is the first step of fatty acid α-oxidation and generates sphingolipids containing 2-hydroxy fatty acyl moieties. Fatty acid 2-hydroxylation is catalyzed by Fatty acid 2-hydroxylase (FA2H) enzyme. However, the precise roles of FA2H and fatty acid 2-hydroxylation in whole cell homeostasis still remain unclear. Methods: Here we utilize Caenorhabditis elegans as the model and systemically investigate the physiological functions of FATH-1/C25A1.5, the highly conserved worm homolog for mammalian FA2H enzyme. Immunostaining, dye-staining and translational fusion reporters were used to visualize FATH-1 protein and a variety of subcellular structures. The “click chemistry” method was employed to label 2-OH fatty acid in vivo. Global and tissue-specific RNAi knockdown experiments were performed to inactivate FATH-1 function. Lipid analysis of the fath-1 deficient mutants was achieved by mass spectrometry. Results: C. elegans FATH-1 is expressed at most developmental stages and in most tissues. Loss of fath-1 expression results in severe growth retardation and shortened lifespan. FATH-1 function is crucially required in the intestine but not the epidermis with stereospecificity. The “click chemistry” labeling technique showed that the FATH-1 metabolites are mainly enriched in membrane structures preferable to the apical side of the intestinal cells. At the subcellular level, we found that loss of fath-1 expression inhibits lipid droplets formation, as well as selectively disrupts peroxisomes and apical endosomes. Lipid analysis of the fath-1 deficient animals revealed a significant reduction in the content of heptadecenoic acid, while other major FAs remain unaffected. Feeding of exogenous heptadecenoic acid (C17: 1), but not oleic acid (C18: 1), rescues the global and subcellular defects of fath-1 knockdown worms. Conclusion: Our study revealed that FATH-1 and its catalytic products are highly specific in the context of chirality, C-chain length, spatial distribution, as well as the types of cellular organelles they affect. Such an unexpected degree of specificity for the synthesis and functions of hydroxylated FAs helps to regulate protein transport and fat metabolism, therefore maintaining the cellular homeostasis of the intestinal cells. These findings may help our understanding of FA2H functions across species, and offer potential therapeutical targets for treating FA2H-related diseases

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., $7.5$). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., $512\times512$) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: https://ml.cs.tsinghua.edu.cn/prolificdreamer/Comment: NeurIPS 2023 (Spotlight

Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition

Face recognition is a prevailing authentication solution in numerous biometric applications. Physical adversarial attacks, as an important surrogate, can identify the weaknesses of face recognition systems and evaluate their robustness before deployed. However, most existing physical attacks are either detectable readily or ineffective against commercial recognition systems. The goal of this work is to develop a more reliable technique that can carry out an end-to-end evaluation of adversarial robustness for commercial systems. It requires that this technique can simultaneously deceive black-box recognition models and evade defensive mechanisms. To fulfill this, we design adversarial textured 3D meshes (AT3D) with an elaborate topology on a human face, which can be 3D-printed and pasted on the attacker's face to evade the defenses. However, the mesh-based optimization regime calculates gradients in high-dimensional mesh space, and can be trapped into local optima with unsatisfactory transferability. To deviate from the mesh-based space, we propose to perturb the low-dimensional coefficient space based on 3D Morphable Model, which significantly improves black-box transferability meanwhile enjoying faster search efficiency and better visual quality. Extensive experiments in digital and physical scenarios show that our method effectively explores the security vulnerabilities of multiple popular commercial services, including three recognition APIs, four anti-spoofing APIs, two prevailing mobile phones and two automated access control systems

MusicAOG: an Energy-Based Model for Learning and Sampling a Hierarchical Representation of Symbolic Music

In addressing the challenge of interpretability and generalizability of artificial music intelligence, this paper introduces a novel symbolic representation that amalgamates both explicit and implicit musical information across diverse traditions and granularities. Utilizing a hierarchical and-or graph representation, the model employs nodes and edges to encapsulate a broad spectrum of musical elements, including structures, textures, rhythms, and harmonies. This hierarchical approach expands the representability across various scales of music. This representation serves as the foundation for an energy-based model, uniquely tailored to learn musical concepts through a flexible algorithm framework relying on the minimax entropy principle. Utilizing an adapted Metropolis-Hastings sampling technique, the model enables fine-grained control over music generation. A comprehensive empirical evaluation, contrasting this novel approach with existing methodologies, manifests considerable advancements in interpretability and controllability. This study marks a substantial contribution to the fields of music analysis, composition, and computational musicology

Beyond the Obvious: Evaluating the Reasoning Ability In Real-life Scenarios of Language Models on Life Scapes Reasoning Benchmark~(LSR-Benchmark)

This paper introduces the Life Scapes Reasoning Benchmark (LSR-Benchmark), a novel dataset targeting real-life scenario reasoning, aiming to close the gap in artificial neural networks' ability to reason in everyday contexts. In contrast to domain knowledge reasoning datasets, LSR-Benchmark comprises free-text formatted questions with rich information on real-life scenarios, human behaviors, and character roles. The dataset consists of 2,162 questions collected from open-source online sources and is manually annotated to improve its quality. Experiments are conducted using state-of-the-art language models, such as gpt3.5-turbo and instruction fine-tuned llama models, to test the performance in LSR-Benchmark. The results reveal that humans outperform these models significantly, indicating a persisting challenge for machine learning models in comprehending daily human life

Exploring the performance and mass-transfer characteristics of porous zinc anodes for membraneless hybrid-flow batteries

Zinc-based hybrid-flow batteries are considered as a promising alternative to conventional electrochemical energy-storage systems for medium- to large-scale applications due to their high energy densities, safety, and abundance. However, the performance of these batteries has been limited by issues such as dendritic growth and passivation of zinc anodes during charge–discharge cycling. To address this challenge, a variety of two- and three-dimensional zinc anodes have been investigated. While two-dimensional zinc anodes have been extensively studied, there has been limited investigation into three-dimensional zinc anodes for hybrid-flow batteries. This study highlights the potential of three-dimensional zinc anodes to mitigate overpotentials and improve the mass transport of active species to promote negative electrode reactions. The performance of a membraneless flow battery based on low-cost zinc and organic quinone was herein evaluated using experimental and numerical approaches. Specifically, the use of zinc fiber was shown to yield an average coulombic efficiency of approximately 90% and an average voltage efficiency of approximately 82% over the course of 100 cycles at a current density of 30 mA cm−2. These results indicate the viability of using zinc fiber anodes to improve the performance of existing hybrid-flow batteries