First-Principles Theoretical Study of Non-equilibrium Electron Dynamics and Electronic Excitation

Exploring non-equilibrium electron dynamics is pivotal for developing predictive insights into molecular and material behavior. Understanding electronic excitations is essential to decipher non-equilibrium processes such as optical absorption, electron transport, and relaxation mechanisms. Quantum mechanics-based first-principles methods are highly effective in modeling the relationship between atomic structure and electron dynamics, eliminating the need for empirical fitting and ensuring quantitative accuracy in properties related to excited states.This dissertation delves into first-principles simulations of non-equilibrium electron dynamics and electronic excitation within condensed matter systems from a chemistry standpoint. The first portion of my dissertation presents nonequilibrium electron dynamics studies via real-time, time-dependent density functional theory (RT-TDDFT). Firstly, we extend natural transition orbitals within RT-TDDFT, so called, “dynamical transition orbital” offering a particle–hole perspective for non-equilibrium electron dynamics simulations. And then, we delve into the nonequilibrium phenomenon of "Floquet topological pumping" within condensed matter physics. We demonstrate the nonadiabatic Thouless pumping of electrons in trans-polyacetylene through Floquet engineering, utilizing first-principles theory. Our approach employs time-dependent maximally localized Wannier functions in real-time density functional theory simulations, linking the winding number—a topological invariant—to an molecular-level understanding of quantized pumping. Also, we identify a single dynamical transition orbital as crucial for quantized pumping, which transitions from π bonding to resonance and antibonding character during the drive cycle. Furthermore, we examine how molecular-level alterations impact the Floquet topological phase of trans-polyacetylene, particularly focusing on how chemical substitutions influence electronic structure properties, including mesmeric, inductive, and electron conjugation effects. Last, we also examine the robustness of this quantum phenomenon at ambient conditions, factoring in the dynamical electron-ion coupling and thermal fluctuations.The final portion of this dissertation comprises advanced method developments of technical methodologies for calculating electronic excited states in extended systems. We implement the Bethe–Salpeter equation in the formulism of all-electron numeric atomic orbital for periodic systems. This advancement in methodology eliminates the uncertainties stemming from the use of non-local pseudopotentials, enabling quantum chemists to leverage recent progress in Green’s function theory methods. Doctor of Philosoph

Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concept provided by the user input image is still a challenging task. To address this problem, researchers have been exploring various methods for customizing pre-trained text-to-image generation models. Currently, most existing methods for customizing pre-trained text-to-image generation models involve the use of regularization techniques to prevent over-fitting. While regularization will ease the challenge of customization and leads to successful content creation with respect to text guidance, it may restrict the model capability, resulting in the loss of detailed information and inferior performance. In this work, we propose a novel framework for customized text-to-image generation without the use of regularization. Specifically, our proposed framework consists of an encoder network and a novel sampling method which can tackle the over-fitting problem without the use of regularization. With the proposed framework, we are able to customize a large-scale text-to-image generation model within half a minute on single GPU, with only one image provided by the user. We demonstrate in experiments that our proposed framework outperforms existing methods, and preserves more fine-grained details

Theory of Moment Propagation for Quantum Dynamics in Single-Particle Description

We present a novel theoretical formulation for performing quantum dynamics in terms of moments within the single-particle description. By expressing the quantum dynamics in terms of increasing orders of moments, instead of single-particle wave functions as generally done in time-dependent density functional theory, we describe an approach for reducing the high computational cost of simulating the quantum dynamics. The equation of motion is given for the moments by deriving analytical expressions for the first-order and second-order time derivatives of the moments, and a numerical scheme is developed for performing quantum dynamics by expanding the moments in the Taylor series as done in classical molecular dynamics simulation. We propose a few numerical approaches using this theoretical formalism on a simple one-dimensional model system, for which an analytically exact solution can be derived. Application of the approaches to an anharmonic system is also discussed to illustrate their generality. We also discuss the use of an artificial neural network model to circumvent the numerical evaluation of the second-order time derivatives of the moments, as analogously done in the context of classical molecular dynamics simulations

Learning Navigational Visual Representations with Semantic Map Supervision

Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot. However, most existing works only employ visual backbones pre-trained either with independent images for classification or with self-supervised learning methods to adapt to the indoor navigation domain, neglecting the spatial relationships that are essential to the learning of navigation. Inspired by the behavior that humans naturally build semantically and spatially meaningful cognitive maps in their brains during navigation, in this paper, we propose a novel navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps (Ego$^2$-Map). We apply the visual transformer as the backbone encoder and train the model with data collected from the large-scale Habitat-Matterport3D environments. Ego$^2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation. Experiments show that agents using our learned representations on object-goal navigation outperform recent visual pre-training methods. Moreover, our representations significantly improve vision-and-language navigation in continuous environments for both high-level and low-level action spaces, achieving new state-of-the-art results of 47% SR and 41% SPL on the test server

Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions

Human-motion generation is a long-standing challenging task due to the requirement of accurately modeling complex and diverse dynamic patterns. Most existing methods adopt sequence models such as RNN to directly model transitions in the original action space. Due to high dimensionality and potential noise, such modeling of action transitions is particularly challenging. In this paper, we focus on skeleton-based action generation and propose to model smooth and diverse transitions on a latent space of action sequences with much lower dimensionality. Conditioned on a latent sequence, actions are generated by a frame-wise decoder shared by all latent action-poses. Specifically, an implicit RNN is defined to model smooth latent sequences, whose randomness (diversity) is controlled by noise from the input. Different from standard action-prediction methods, our model can generate action sequences from pure noise without any conditional action poses. Remarkably, it can also generate unseen actions from mixed classes during training. Our model is learned with a bi-directional generative-adversarial-net framework, which not only can generate diverse action sequences of a particular class or mix classes, but also learns to classify action sequences within the same model. Experimental results show the superiority of our method in both diverse action-sequence generation and classification, relative to existing methods.Comment: AAAI 202

First-Principles Approach for Coupled Quantum Dynamics of Electrons and Protons in Heterogeneous Systems

The coupled quantum dynamics of electrons and protons is ubiquitous in many dynamical processes involving light-matter interaction, such as solar energy conversion in chemical systems and photosynthesis. A first-principles description of such nuclear-electronic quantum dynamics requires not only the time-dependent treatment of nonequilibrium electron dynamics but also that of quantum protons. Quantum mechanical correlation between electrons and protons adds further complexity to such coupled dynamics. Here we extend real-time nuclear-electronic orbital time-dependent density functional theory (RT-NEO-TDDFT) to periodic systems and perform first-principles simulations of coupled quantum dynamics of electrons and protons in complex heterogeneous systems. The process studied is electronically excited state intramolecular proton transfer of o-hydroxybenzaldehyde in water and at a silicon (111) semiconductor-molecule interface. These simulations illustrate how environments such as hydrogen-bonding water molecules and an extended material surface impact the dynamical process on the atomistic level. Depending on how the molecule is chemisorbed on the surface, excited state electron transfer from the molecule to the semiconductor surface can inhibit ultrafast proton transfer within the molecule. This work elucidates how heterogeneous environments influence the balance between the quantum mechanical proton transfer and excited electron dynamics. The periodic RT-NEO-TDDFT approach is applicable to a wide range of other photoinduced heterogeneous processes

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.Comment: Preprint. Work in progres

Towards Building the Federated GPT: Federated Instruction Tuning

While "instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amounts of diverse and high-quality instruction data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data, especially when it comes to human-written data, can pose significant challenges both in terms of cost and accessibility. Moreover, concerns related to privacy can further limit access to such data, making the process of obtaining it a complex and nuanced undertaking. Consequently, this hinders the generality of the tuned models and may restrict their effectiveness in certain contexts. To tackle this issue, our study introduces a new approach called Federated Instruction Tuning (FedIT), which leverages federated learning (FL) as the learning framework for the instruction tuning of LLMs. This marks the first exploration of FL-based instruction tuning for LLMs. This is especially important since text data is predominantly generated by end users. Therefore, it is imperative to design and adapt FL approaches to effectively leverage these users' diverse instructions stored on local devices, while preserving privacy and ensuring data security. In the current paper, by conducting widely used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with the proposed framework FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions. Further, in this paper, we developed a Github repository named Shepherd. This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.Comment: Project page: https://github.com/JayZhang42/FederatedGPT-Shepher

Bone Mesenchymal Stem Cell-Derived Extracellular Vesicles Promote Recovery Following Spinal Cord Injury via Improvement of the Integrity of the Blood-Spinal Cord Barrier

Mesenchymal stem cell (MSC) transplantation has been shown to represent a potential treatment for traumatic spinal cord injury (SCI). However, there are several obstacles that need to be overcome before MSCs can be considered for clinical application, such as failure of MSCs to reach the spinal cord lesion core and possible tumor formation. Recent studies have suggested that MSC treatment is beneficial owing to paracrine-secreted factors. Extracellular vesicles are considered to be some of the most valuable paracrine molecules. However, the therapeutic mechanism of extracellular vesicles on spinal cord injury has not been studied clearly. Therefore, our study investigated the effect of systemic administration of extracellular vesicles on the loss of motor function after SCI and examined the potential mechanisms underlying their effects. Disruption of the blood-spinal cord barrier (BSCB) is a crucial factor that can be detrimental to motor function recovery. Pericytes are an important component of the neurovascular unit, and play a pivotal role in maintaining the structural integrity of the BSCB. Our study demonstrated that administration of bone mesenchymal stem cell-derived extracellular vesicles (BMSC-EV) reduced brain cell death, enhanced neuronal survival and regeneration, and improved motor function compared with the administration of BMSC-EV free culture media (EV-free CM). Besides, the BSCB was attenuated and pericyte coverage was significantly decreased in vivo. Furthermore, we found that exosomes reduced pericyte migration via downregulation of NF-κB p65 signaling, with a consequent decrease in the permeability of the BSCB. In summary, we identified that extracellular vesicles treatment suppressed the migration of pericytes and further improved the integrity of the BSCB via NF-κB p65 signaling in pericytes. Our data suggest that extracellular vesicles may serve as a promising treatment strategy for SCI