SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models

Diffusion models, which have emerged to become popular text-to-image generation models, can produce high-quality and content-rich images guided by textual prompts. However, there are limitations to semantic understanding and commonsense reasoning in existing models when the input prompts are concise narrative, resulting in low-quality image generation. To improve the capacities for narrative prompts, we propose a simple-yet-effective parameter-efficient fine-tuning approach called the Semantic Understanding and Reasoning adapter (SUR-adapter) for pre-trained diffusion models. To reach this goal, we first collect and annotate a new dataset SURD which consists of more than 57,000 semantically corrected multi-modal samples. Each sample contains a simple narrative prompt, a complex keyword-based prompt, and a high-quality image. Then, we align the semantic representation of narrative prompts to the complex prompts and transfer knowledge of large language models (LLMs) to our SUR-adapter via knowledge distillation so that it can acquire the powerful semantic understanding and reasoning capabilities to build a high-quality textual semantic representation for text-to-image generation. We conduct experiments by integrating multiple LLMs and popular pre-trained diffusion models to show the effectiveness of our approach in enabling diffusion models to understand and reason concise natural language without image quality degradation. Our approach can make text-to-image diffusion models easier to use with better user experience, which demonstrates our approach has the potential for further advancing the development of user-friendly text-to-image generation models by bridging the semantic gap between simple narrative prompts and complex keyword-based prompts. The code is released at https://github.com/Qrange-group/SUR-adapter.Comment: accepted by ACM MM 202

ASR: Attention-alike Structural Re-parameterization

The structural re-parameterization (SRP) technique is a novel deep learning technique that achieves interconversion between different network architectures through equivalent parameter transformations. This technique enables the mitigation of the extra costs for performance improvement during training, such as parameter size and inference time, through these transformations during inference, and therefore SRP has great potential for industrial and practical applications. The existing SRP methods have successfully considered many commonly used architectures, such as normalizations, pooling methods, multi-branch convolution. However, the widely used self-attention modules cannot be directly implemented by SRP due to these modules usually act on the backbone network in a multiplicative manner and the modules' output is input-dependent during inference, which limits the application scenarios of SRP. In this paper, we conduct extensive experiments from a statistical perspective and discover an interesting phenomenon Stripe Observation, which reveals that channel attention values quickly approach some constant vectors during training. This observation inspires us to propose a simple-yet-effective attention-alike structural re-parameterization (ASR) that allows us to achieve SRP for a given network while enjoying the effectiveness of the self-attention mechanism. Extensive experiments conducted on several standard benchmarks demonstrate the effectiveness of ASR in generally improving the performance of existing backbone networks, self-attention modules, and SRP methods without any elaborated model crafting. We also analyze the limitations and provide experimental or theoretical evidence for the strong robustness of the proposed ASR.Comment: Technical repor

RA2: predicting simulation execution time for cloud-based design space explorations

Design space exploration refers to the evaluation of implementation alternatives for many engineering and design problems. A popular exploration approach is to run a large number of simulations of the actual system with varying sets of configuration parameters to search for the optimal ones. Due to the potentially huge resource requirements, cloud-based simulation execution strategies should be considered in many cases. In this paper, we look at the issue of running large-scale simulation-based design space exploration problems on commercial Infrastructure-as-a-Service clouds, namely Amazon EC2, Microsoft Azure and Google Compute Engine. To efficiently manage cloud resources used for execution, the key problem would be to accurately predict the running time for each simulation instance in advance. This is not trivial due to the currently wide range of cloud resource types which offer varying levels of performance. In addition, the widespread use of virtualization techniques in most cloud providers often introduces unpredictable performance interference. In this paper, we propose a resource and application-aware (RA2) prediction approach to combat performance variability on clouds. In particular, we employ neural network based techniques coupled with non-intrusive monitoring of resource availability to obtain more accurate predictions. We conducted extensive experiments on commercial cloud platforms using an evacuation planning design problem over a month-long period. The results demonstrate that it is possible to predict simulation execution times in most cases with high accuracy. The experiments also provide some interesting insights on how we should run similar simulation problems on various commercially available clouds

Substitution of Ni for Fe in superconducting Fe0.98_{0.98}Te0.5_{0.5}Se0.5_{0.5} depresses the normal-state conductivity but not the magnetic spectral weight

We have performed systematic resistivity and inelastic neutron scattering measurements on Fe$_{0.98-z}$Ni$_z$Te$_{0.5}$Se$_{0.5}$ samples to study the impact of Ni substitution on the transport properties and the low-energy ($\le$ 12 meV) magnetic excitations. It is found that, with increasing Ni doping, both the conductivity and superconductivity are gradually suppressed; in contrast, the low-energy magnetic spectral weight changes little. Comparing with the impact of Co and Cu substitution, we find that the effects on conductivity and superconductivity for the same degree of substitution grow systematically as the atomic number of the substituent deviates from that of Fe. The impact of the substituents as scattering centers appears to be greater than any contribution to carrier concentration. The fact that low-energy magnetic spectral weight is not reduced by increased electron scattering indicates that the existence of antiferromagnetic correlations does not depend on electronic states close to the Fermi energy.Comment: 6 pages, 5 figure

Guide them through: an automatic crowd control framework using multi-objective genetic programming

We propose an automatic crowd control framework based on multi-objective optimisa- tion of strategy space using genetic programming. In particular, based on the sensed local crowd densities at different segments, our framework is capable of generating control strategies that guide the individuals on when and where to slow down for opti- mal overall crowd flow in realtime, quantitatively measured by multiple objectives such as shorter travel time and less congestion along the path. The resulting Pareto-front al- lows selection of resilient and efficient crowd control strategies in different situations. We first chose a benchmark scenario as used in [1] to test the proposed method. Results show that our method is capable of finding control strategies that are not only quanti- tatively measured better, but also well aligned with domain experts’ recommendations on effective crowd control such as “slower is faster” and “asymmetric control”. We further applied the proposed framework in actual event planning with approximately 400 participants navigating through a multi-story building. In comparison with the baseline crowd models that do no employ control strategies or just use some hard-coded rules, the proposed framework achieves a shorter travel time and a significantly lower (20%) congestion along critical segments of the path

Functional Analysis of General Odorant Binding Protein 2 from the Meadow Moth, Loxostege sticticalis L. (Lepidoptera: Pyralidae)

Odorant binding proteins play a crucial role in transporting semiochemicals across the sensillum lymph to olfactory receptors within the insect antennal sensilla. In this study, the general odorant binding protein 2 gene was cloned from the antennae of Loxostege sticticalis, using reverse transcription PCR and rapid amplification of cDNA ends. Recombinant LstiGOBP2 was expressed in Escherichia coli and purified by Ni ion affinity chromatography. Real-time PCR assays indicated that LstiGOBP2 mRNA is expressed mainly in adult antennae, with expression levels differing with developmental age. Ligand-binding experiments using N-phenyl-naphthylamine (1-NPN) as a fluorescent probe demonstrated that the LstiGOBP2 protein has binding affinity to a broad range of odorants. Most importantly, trans-11-tetradecen-1-yl acetate, the pheromone component of Loxostege sticticalis, and trans-2-hexenal and cis-3-hexen-1-ol, the most abundant plant volatiles in essential oils extracted from host plants, had high binding affinities to LstiGOBP2 and elicited strong electrophysiological responses from the antennae of adults

Genome-wide Analyses Identify KIF5A as a Novel ALS Gene

To identify novel genes associated with ALS, we undertook two lines of investigation. We carried out a genome-wide association study comparing 20,806 ALS cases and 59,804 controls. Independently, we performed a rare variant burden analysis comparing 1,138 index familial ALS cases and 19,494 controls. Through both approaches, we identified kinesin family member 5A (KIF5A) as a novel gene associated with ALS. Interestingly, mutations predominantly in the N-terminal motor domain of KIF5A are causative for two neurodegenerative diseases: hereditary spastic paraplegia (SPG10) and Charcot-Marie-Tooth type 2 (CMT2). In contrast, ALS-associated mutations are primarily located at the C-terminal cargo-binding tail domain and patients harboring loss-of-function mutations displayed an extended survival relative to typical ALS cases. Taken together, these results broaden the phenotype spectrum resulting from mutations in KIF5A and strengthen the role of cytoskeletal defects in the pathogenesis of ALS.Peer reviewe

Direct Evidence of Ice Crystallization Inhibition by Dielectric Relaxation of Hydrated Ions

In this paper, the inhibition effect of an alternative current (AC) electric field on ice crystallization in 0.9 wt % NaCl aqueous solution was confirmed thermodynamically with characterization. An innovative experimental and analytical method, combining differential scanning calorimeter (DSC) measurement with an externally applied electric field was created by implanting microelectrodes in a sample crucible. It was found that the ice crystallization, including pure ice and salty ice, was obviously inhibited after field cooling with an external AC electric field in a frequency range of 100 k–10 MHz, and the crystallization ratio was related to frequency. Compared with non-field cooling, the crystallization ratio of ice crystals was reduced to less than 20% when E = 57.8 kV/m and f = 1 MHz. The dielectric spectrum results show that this inhibition effect of an alternating electric field on ice crystal growth is closely related to the dielectric relaxation process of hydrated ions