BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/

LAVIS: A Library for Language-Vision Intelligence

We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. LAVIS aims to serve as a one-stop comprehensive library that brings recent advancements in the language-vision field accessible for researchers and practitioners, as well as fertilizing future research and development. It features a unified interface to easily access state-of-the-art image-language, video-language models and common datasets. LAVIS supports training, evaluation and benchmarking on a rich variety of tasks, including multimodal classification, retrieval, captioning, visual question answering, dialogue and pre-training. In the meantime, the library is also highly extensible and configurable, facilitating future development and customization. In this technical report, we describe design principles, key components and functionalities of the library, and also present benchmarking results across common language-vision tasks. The library is available at: https://github.com/salesforce/LAVIS.Comment: Preprint of LAVIS technical repor

Increased Mobility of Metal Oxide Nanoparticles Due to Photo and Thermal Induced Disagglomeration

Significant advances have been made on our understanding of the fate and transport of engineered nanomaterials. One unexplored aspect of nanoparticle aggregation is how environmental stimuli such as light exposure and temperature variations affect the mobility of engineered nanoparticles. In this study, TiO2, ZnO, and CeO2 were chosen as model materials for investigating the mobility of nanoparticles under three external stimuli: heat, light and sonication. Sunlight and high power sonication were able to partially disagglomerate metal oxide clusters, but primary particles bonded by solid state necks were left intact. A cycle of temperature increase from 25°C to 65°C and then decrease back was found to disagglomerate the compact clusters in the heating phase and reagglomerate them as more open fractal structures during the cooling phase. A fractal model summing the pair-wise DLVO interactions between primary particles within two fractal agglomerates predicts weak attractions on the order of a few kT. Our study shows that common environmental stimuli such as light exposure or temperature variation can disagglomerate nanoparticle clusters and enhance their mobility in open waters. This phenomenon warrants attention since it is likely that metal oxide nanoparticles will experience these natural stimuli during their transport in the environment

Specific Inhibition of Phosphodiesterase-4B Results in Anxiolysis and Facilitates Memory Acquisition

Cognitive dysfunction is a core feature of dementia and a prominent feature in psychiatric disease. As non-redundant regulators of intracellular cAMP gradients, phosphodiesterases (PDE) mediate fundamental aspects of brain function relevant to learning, memory, and higher cognitive functions. Phosphodiesterase-4B (PDE4B) is an important phosphodiesterase in the hippocampal formation, is a major Disrupted in Schizophrenia 1 (DISC1) binding partner and is itself a risk gene for psychiatric illness. To define the effects of specific inhibition of the PDE4B subtype, we generated mice with a catalytic domain mutant form of PDE4B (Y358C) that has decreased ability to hydrolyze cAMP. Structural modelling predictions of decreased function and impaired binding with DISC1 were confirmed in cell assays. Phenotypic characterization of the PDE4BY358C mice revealed facilitated phosphorylation of CREB, decreased binding to DISC1, and upregulation of DISC1 and β-Arrestin in hippocampus and amygdala. In behavioural assays, PDE4BY358C mice displayed decreased anxiety and increased exploration, as well as cognitive enhancement across several tests of learning and memory, consistent with synaptic changes including enhanced long-term potentiation and impaired depotentiation ex vivo. PDE4BY358C mice also demonstrated enhanced neurogenesis. Contextual fear memory, though intact at 24 hours, was decreased at 7 days in PDE4BY358C mice, an effect replicated pharmacologically with a non-selective PDE4 inhibitor, implicating cAMP signalling by PDE4B in a very late phase of consolidation. No effect of the PDE4BY358C mutation was observed in the pre-pulse inhibition and forced swim tests. Our data establish specific inhibition of PDE4B as a promising therapeutic approach for disorders of cognition and anxiety, and a putative target for pathological fear memory

Long-wavelength infrared (~10-15 mm) electroluminescence from Sb-based interband cascade devices

Electroluminescence in the long-wavelength infrared (10-15 mm) spectrum region was observed from Sb-based type-II interband cascade quantum well structures. The device structure was grown by molecular beam epitaxy on a GaSb substrate and comprises 10 repeated periods of active regions separated by digitally graded multilayer injection regions. The devices have been operated at 300 K and 77 K, with an output optical power up to 50 nW. The emission wavelength, the longest observed in any compound semiconductor material at room temperature, results from tailoring the heterostructure, demonstrating a unique capability of this Sb-family type-II material system.NRC publication: Ye

Extravascular modified lipoproteins: a role in the propagation of diabetic retinopathy in a mouse model of type 1 diabetes

AIMS/HYPOTHESIS: We aimed to determine whether plasma lipoproteins, after leakage into the retina and modification by glycation and oxidation, contribute to the development of diabetic retinopathy in a mouse model of type 1 diabetes. METHODS: To simulate permeation of plasma lipoproteins into retinal tissues, streptozotocin-induced mouse models of diabetes and non-diabetic mice were challenged with intravitreal injection of human ‘highly-oxidised glycated’ low-density lipoprotein (HOG-LDL), native- (N-) LDL, or the vehicle PBS. Retinal histology, electroretinography (ERG) and biochemical markers were assessed over the subsequent 14 days. RESULTS: Intravitreal administration of N-LDL and PBS had no effect on retinal structure or function in either diabetic or non-diabetic animals. In non-diabetic mice, HOG-LDL elicited a transient inflammatory response without altering retinal function, but in diabetic mice it caused severe, progressive retinal injury, with abnormal morphology, ERG changes, vascular leakage, vascular endothelial growth factor overexpression, gliosis, endoplasmic reticulum stress, and propensity to apoptosis. CONCLUSIONS/INTERPRETATION: Diabetes confers susceptibility to retinal injury imposed by intravitreal injection of modified LDL. The data add to the existing evidence that extravasated, modified plasma lipoproteins contribute to the propagation of diabetic retinopathy. Intravitreal delivery of HOG-LDL simulates a stress known to be present, in addition to hyperglycaemia, in human diabetic retinopathy once blood-retinal barriers are compromised. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s00125-016-4012-6) contains peer-reviewed but unedited supplementary material, which is available to authorised users

Imaging the Pair-Correlated HNCO Photodissociation: The NH(a(1)Delta) + CO(X-1 Sigma(+)) Channel

The NH(a(1)Delta) + CO((XE+)-E-1) product channel for the photodissociation of HNCO at 201 nm was investigated using the sliced velocity map ion imaging technique with the detection of NH(a(1)Delta) products via (2 + 1) resonance enhanced multiphoton ionization (REMPI). Images were measured for the NH(a(1)Delta) rotational states in the ground and vibrational excited states (v = 0 and 1). Correlation between the NH(a(1)Delta) and CO rovibrational state distributions were determined from these images. Experimental results show that the vibrational distribution of the CO fragment in the NH(a(1)Delta) + CO(X-1 Sigma(+)) channel peaks at v = 1. The negative anisotropy parameter measured for the NH(a(1)Delta) (v = 0 and 1lj) products indicates a direct dissociation process for the N-C bond cleavage in the 5, state. A bimodal CO rotational distribution was observed, suggesting that HNCO dissociates in the S-1 state in two distinctive pathways