IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization

Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks. Recent advance in representation learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings can significantly improve performance on downstream tasks with faster convergence and better generalization. The isotropy of the pre-trained embeddings in PTLMs, however, is relatively under-explored. In this paper, we analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with straightforward visualization, and point out two major issues: high variance in their standard deviation, and high correlation between different dimensions. We also propose a new network regularization method, isotropic batch normalization (IsoBN) to address the issues, towards learning more isotropic representations in fine-tuning by dynamically penalizing dominating principal components. This simple yet effective fine-tuning method yields about 1.0 absolute increment on the average of seven NLU tasks.Comment: AAAI 202

Quantum Pseudoentanglement

Quantum pseudorandom states are efficiently constructable states which nevertheless masquerade as Haar-random states to poly-time observers. First defined by Ji, Liu and Song, such states have found a number of applications ranging from cryptography to the AdS/CFT correspondence. A fundamental question is exactly how much entanglement is required to create such states. Haar-random states, as well as $t$-designs for $t\geq 2$, exhibit near-maximal entanglement. Here we provide the first construction of pseudorandom states with only polylogarithmic entanglement entropy across an equipartition of the qubits, which is the minimum possible. Our construction can be based on any one-way function secure against quantum attack. We additionally show that the entanglement in our construction is fully "tunable", in the sense that one can have pseudorandom states with entanglement $\Theta(f(n))$ for any desired function $\omega(\log n) \leq f(n) \leq O(n)$. More fundamentally, our work calls into question to what extent entanglement is a "feelable" quantity of quantum systems. Inspired by recent work of Gheorghiu and Hoban, we define a new notion which we call "pseudoentanglement", which are ensembles of efficiently constructable quantum states which hide their entanglement entropy. We show such states exist in the strongest form possible while simultaneously being pseudorandom states. We also describe diverse applications of our result from entanglement distillation to property testing to quantum gravity.Comment: 32 page

Multiple-Change-Point Modeling and Exact Bayesian Inference of Degradation Signal for Prognostic Improvement

Prognostics play an increasingly important role in modern engineering systems for smart maintenance decision-making. In parametric regression-based approaches, the parametric models are often too rigid to model degradation signals in many applications. In this paper, we propose a Bayesian multiple-change-point (CP) modeling framework to better capture the degradation path and improve the prognostics. At the offline modeling stage, a novel stochastic process is proposed to model the joint prior of CPs and positions. All hyperparameters are estimated through an empirical two-stage process. At the online monitoring and remaining useful life (RUL) prediction stage, a recursive updating algorithm is developed to exactly calculate the posterior distribution and RUL prediction sequentially. To control the computational cost, a fixed-support-size strategy in the online model updating and a partial Monte Carlo strategy in the RUL prediction are proposed. The effectiveness and advantages of the proposed method are demonstrated through thorough simulation and real case studies

A signal cascade originated from epidermis defines apical-basal patterning of Arabidopsis shoot apical meristems

In multicellular organisms, a long-standing question is how spatial patterns of distinct cell types are initiated and maintained during continuous cell division and proliferation. Along the vertical axis of plant shoot apical meristems (SAMs), stem cells are located at the top while cells specifying the stem cells are located more basally, forming a robust apical-basal pattern. We previously found that in Arabidopsis SAMs, the HAIRY MERISTEM (HAM) family transcription factors form a concentration gradient from the epidermis to the interior cell layers, and this gradient is essential for the stem cell specification and the apical-basal patterning of the SAMs. Here, we uncover that epidermis specific transcription factors, ARABIDOPSIS THALIANA MERISTEM LAYER 1 (ATML1) and its close homolog, define the concentration gradient of HAM in the SAM through activating a group of microRNAs. This study provides a molecular framework linking the epidermis-derived signal to the stem cell homeostasis in plants

Public-key pseudoentanglement and the hardness of learning ground state entanglement structure

Given a local Hamiltonian, how difficult is it to determine the entanglement structure of its ground state? We show that this problem is computationally intractable even if one is only trying to decide if the ground state is volume-law vs near area-law entangled. We prove this by constructing strong forms of pseudoentanglement in a public-key setting, where the circuits used to prepare the states are public knowledge. In particular, we construct two families of quantum circuits which produce volume-law vs near area-law entangled states, but nonetheless the classical descriptions of the circuits are indistinguishable under the Learning with Errors (LWE) assumption. Indistinguishability of the circuits then allows us to translate our construction to Hamiltonians. Our work opens new directions in Hamiltonian complexity, for example whether it is difficult to learn certain phases of matter.Comment: 58 page

A universal high energy anomaly in angle resolved photoemission spectra of high temperature superconductors - possible evidence of spinon and holon branches

A universal high energy anomaly in the single particle spectral function is reported in three different families of high temperature superconductors by using angle-resolved photoemission spectroscopy. As we follow the dispersing peak of the spectral function from the Fermi energy to the valence band complex, we find dispersion anomalies marked by two distinctive high energy scales, E_1=~ 0.38 eV and E_2=~0.8 eV. E_1 marks the energy above which the dispersion splits into two branches. One is a continuation of the near parabolic dispersion, albeit with reduced spectral weight, and reaches the bottom of the band at the gamma point at ~0.5 eV. The other is given by a peak in the momentum space, nearly independent of energy between E_1 and E_2. Above E_2, a band-like dispersion re-emerges. We conjecture that these two energies mark the disintegration of the low energy quasiparticles into a spinon and holon branch in the high T_c cuprates.Comment: accepted for publication in Phys. Rev. Let