Controlling Text-to-Image Diffusion by Orthogonal Finetuning

Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.Comment: NeurIPS 2023 (43 pages, 34 figures, project page: https://oft.wyliu.com/

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language.Comment: Technical Report (33 pages, 18 figures

Impact of lockdown on the growth of children in China aged 3-6 years during the COVID-19 pandemic

BackgroundLockdowns in COVID-19 pandemic led to less physical activity and more intake of unhealthy food in children. The aim of this study was to investigate the negative impact of major lockdowns on the growth of children aged 3-6 years during COVID-19 pandemic period.MethodsPhysical examination results in 2019 to 2022 from 5834 eligible children (2972 males and 2862 females) from Southwestern China who were 3 years old in 2019 were retrospectively collected. Height and weight data points were extracted from the results, and percentiles of height (height%), weight (weight%), and BMI (BMI%), and rates of overweight and obesity were calculated and compared between different years during the pandemic.ResultsAfter analyzing the 15404 growth data points from 5834 children, a slowly increasing trend of height% from 2019 to 2022 was observed. Weight%, BMI%, overweight rate, obesity rate, and combined overweight and obesity rate had two peaks in 2020 and 2022 when major lockdowns were adopted and a drop in between (year 2021), except for obesity rate which did not drop in 2021. Similar results were shown after stratification by gender.ConclusionThe lockdowns in COVID-19 pandemic promoted obesity of kindergarten children, but did not show any negative impact on their height growth possibly due to over-nutrition of children during lockdowns. More efforts need to be made to limit the increase of obesity rate in kindergarten children during possible future lockdowns

Charge redistribution, charge order and plasmon in La2−x_{2-x}Srx_{x}CuO4_{4}/La2_{2}CuO4_{4} superlattices

Interfacial superconductors have the potential to revolutionize electronics, quantum computing, and fundamental physics due to their enhanced superconducting properties and ability to create new types of superconductors. The emergence of superconductivity at the interface of La$_{2-x}$Sr$_{x}$CuO$_{4}$/La$_{2}$CuO$_{4}$ (LSCO/LCO), with a T$_c$ enhancement of $\sim$ 10 K compared to the La$_{2-x}$Sr$_{x}$CuO$_{4}$ bulk single crystals, provides an exciting opportunity to study quantum phenomena in reduced dimensions. To investigate the carrier distribution and excitations in interfacial superconductors, we combine O K-edge resonant inelastic X-ray scattering and atomic-resolved scanning transmission electron microscopy measurements to study La$_{2-x}$Sr$_{x}$CuO$_{4}$/La$_{2}$CuO$_{4}$ superlattices (x=0.15, 0.45) and bulk La$_{1.55}$Sr$_{0.45}$CuO$_{4}$ films. We find direct evidence of charge redistribution, charge order and plasmon in LSCO/LCO superlattices. Notably, the observed behaviors of charge order and plasmon deviate from the anticipated properties of individual constituents or the average doping level of the superlattice. Instead, they conform harmoniously to the effective doping, a critical parameter governed by the T$_c$ of interfacial superconductors.Comment: 8 pages, 5 figure

Prion-like Aggregation of Mitochondrial Antiviral Signaling Protein in Lupus Patients Is Associated With Increased Levels of Type I Interferon: MAVS AGGREGATION AND TYPE I IFN IN LUPUS

Increased levels of Type I interferon (IFN-I) and IFN-I-regulated genes are found in patients with systemic lupus erythematosus (SLE) and may be central to its pathogenesis. The mitochondrial adaptor protein MAVS is a key regulator of IFN-I that undergoes a dramatic prion-like aggregation and self-propagates the activation signal from viral RNA to amplify downstream IFN production. We wondered if such MAVS aggregates might play a role in the sustained increased production of IFN-I in SLE

Comparison of the chemical evolution and characteristics of 495 biomass burning plumes intercepted by the NASA DC-8 aircraft during the ARCTAS/CARB-2008 field campaign

This paper compares measurements of gaseous and particulate emissions from a wide range of biomass-burning plumes intercepted by the NASA DC-8 research aircraft during the three phases of the ARCTAS-2008 experiment: ARCTAS-A, based out of Fairbanks, Alaska USA (3 April to 19 April 2008); ARCTAS-B based out of Cold Lake, Alberta, Canada (29 June to 13 July 2008); and ARCTAS-CARB, based out of Palmdale, California, USA (18 June to 24 June 2008). Extensive investigations of boreal fire plume evolution were undertaken during ARCTAS-B, where four distinct fire plumes that were intercepted by the aircraft over a range of down-wind distances (0.1 to 16 hr transport times) were studied in detail. Based on these analyses, there was no evidence for ozone production and a box model simulation of the data confirmed that net ozone production was slow (on average 1 ppbv h−1 in the first 3 h and much lower afterwards) due to limited NOx. Peroxyacetyl nitrate concentrations (PAN) increased with plume age and the box model estimated an average production rate of ~80 pptv h−1 in the first 3 h. Like ozone, there was also no evidence for net secondary inorganic or organic aerosol formation. There was no apparent increase in aerosol mass concentrations in the boreal fire plumes due to secondary organic aerosol (SOA) formation; however, there were indications of chemical processing of the organic aerosols. In addition to the detailed studies of boreal fire plume evolution, about 500 smoke plumes intercepted by the NASA DC-8 aircraft were segregated by fire source region. The normalized excess mixing ratios (i.e. ΔX/ΔCO) of gaseous (carbon dioxide, acetonitrile, hydrogen cyanide, toluene, benzene, methane, oxides of nitrogen (NOx), ozone, PAN) and fine aerosol particulate components (nitrate, sulfate, ammonium, chloride, organic aerosols and water soluble organic carbon) of these plumes were compared