Multi-Entity Dependence Learning with Rich Context via Conditional Variational Auto-encoder

Multi-Entity Dependence Learning (MEDL) explores conditional correlations among multiple entities. The availability of rich contextual information requires a nimble learning scheme that tightly integrates with deep neural networks and has the ability to capture correlation structures among exponentially many outcomes. We propose MEDL_CVAE, which encodes a conditional multivariate distribution as a generating process. As a result, the variational lower bound of the joint likelihood can be optimized via a conditional variational auto-encoder and trained end-to-end on GPUs. Our MEDL_CVAE was motivated by two real-world applications in computational sustainability: one studies the spatial correlation among multiple bird species using the eBird data and the other models multi-dimensional landscape composition and human footprint in the Amazon rainforest with satellite images. We show that MEDL_CVAE captures rich dependency structures, scales better than previous methods, and further improves on the joint likelihood taking advantage of very large datasets that are beyond the capacity of previous methods.Comment: The first two authors contribute equall

Experimental and numerical investigation in CO2 sequestrations in chemical looping combustion

Chemical looping combustion (CLC) process is an emerging alternative to traditional CO2 mitigation technology in many industrial applications since it could produce high pure CO2 gas stream with relatively low cost. The flow occurring in the CLC is intrinsically a gas-solid two-phase flow coupled with heterogeneous reactions whilst the performance of CLC is significantly affected by the efficiency of the combustion taking place in the fuel reactor. This PhD research project investigates the application of chemical looping combustion technology for CO2 sequestrations, focusing on the hydrodynamics and chemical kinetics of the flows of the CLC in the fuel reactor. As bypass fluidised bubbles in dense phase regions of the fuel reactor remarkably affect the efficiency of combustion in the CLC, the phenomena of bubble motion are experimentally and numerically investigated first. Chapter 2 proposes a new analytical approach coupled with the adoption of auto-correlated wavelet transform to experimentally study the correlations between the detected pressure fluctuation signals obtained from a model fuel reactor in which the chemical reaction has been redundant and the occurrence of bubbles. The sub-signals of pressure fluctuations obtained can be used as the indicator to identify the occurrence of bubbles, which has been validated by the snapshots of the fluidisation patterns. Experimental results clearly show that the formed bubbles in the dense phase regions behave two distinct types, small bubbles with the characteristics of high fluctuation frequency and large bubbles with lower fluctuation frequency. The characteristic frequencies of these detected bubbles can be also identified through the analysis of the pressure fluctuation signals. In parallel to the experimental study, the applications of Computational fluid dynamics (CFD) numerical modelling to study the flow dynamic behaviour of CLC in the fuel reactor were attempted. Eulerian-Eulerian two fluid model and Eulerian-Lagrangian approach, represented by Computational fluid dynamics/Discrete element method (CFD-DEM) in the present study, were employed, respectively, to study the hydrodynamics in the fuel reactor of CLC. Chapter 3 presents the work which CFD-DEM modelling was employed to investigate the bubble hydrodynamics in the dense region of fluidised bed fuel reactor under the different inlet conditions. Correlations between the local dynamic parameters such as the pressure fluctuation, local solid volume fraction fluctuation and instantaneous velocities are introduced to detect the occurrence of the bubbles, where the bubble has been defined in terms of the volumetrically averaged local void fraction. The simulations demonstrated that these bubbles are highly correlated with the local large eddies embedded in the flow. It was also revealed that small bubbles with high by-passing frequency mainly occur in the bottom region of the fuel reactor while large bubbles with relatively lower frequency are found in the region close to the free board surface. This finding affirms that the size of bubble is highly correlated with the local dynamic field. A modified Darton’s model that uses local Reynolds number and dimensionless height ratio was thus proposed for prediction of the equivalent diameters of the formed bubbles at the given height position. In Chapters 4 and 5, Eulerian-Eulerian two-fluid CFD modelling is employed to study the hydrodynamics of the CLC coupled with the heterogeneous reaction in the fuel reactors with different configurations. Based on the simulation results, the correlation parameters that correlate the local volume fractions with the local dynamic parameters such as the pressure, velocity and temperature fluctuations were proposed, aiming at indicating the bubble occurrence in the fuel reactor where the heterogeneous reaction takes place simultaneously. The frequency of bubble occurrence at the given height position is also identified quantitatively through monitoring the time-dependant pressure fluctuations obtained from the CFD modelling. As the CLC involves heterogeneous reaction among the reactants in the fuel reactor where the oxides are reduced to the metal particles before refeeding back to the air reactor, most of the previously documented studies using CFD modelling for prediction of hydrodynamics in the fuel reactor adopted shrinking core model proposed by Szekely’s et al. (1973) but the effects of the irregularity geometry of the oxygen carriers and product-layer diffusion on the simulation have been overlooked. Thus, an improved shrinking core model that takes effects of both the irregularity geometry of the oxygen carriers and product-layer diffusion into account was proposed. Compared with the predictions using the original shrinking core model, e.g. García-Labiano et al. (2004) and Zafar et al. (2007a), the simulation results obtained by using the improved model can significantly improve the accuracy for prediction of the conversion rates. The simulations also indicate that the effect of product-layer diffusion becomes more notable with an increase in the completeness of conversion. An empirical relation is thereby proposed to describe the variations of the effect of product-layer diffusion on the oxygen carrier conversion. In summary, this dissertation contributes to the knowledge and understanding of the CLC in several aspects, in particular hydrodynamics and chemical kinetics of the flow in the fuel reactor. Firstly, a new analytical method coupled the auto-correlated wavelet transform was proposed to study the bubble formation in the dense bed region by analysing the pressure fluctuation signals. Secondly, the correlation parameters that correlate the local volume fractions with those dynamic parameters such as the pressure and velocity were introduced to predict the occurrence of bubbles at the given height position of the fuel reactor. Thirdly, the conventional shrinking core model has been improved by taking the effects of irregularity of solid particle and the product-layer diffusion into account

Time Variation of Fine-Structure Constant Constrained by [O III] Emission-Lines at 1.1<z<3.7

[O III]$\lambda\lambda$4960,5008 doublet are often the strongest narrow emission lines in starburst galaxies and quasi-stellar objects (QSOs), and thus are a promising probe to possible variation of the fine-structure constant $\alpha$ over cosmic time. Previous such studies using QSOs optical spectra were limited to $z<1$. In this work, we constructed a sample of 40 spectra of Ly$\alpha$ emitting galaxies (LAEs) and a sample of 46 spectra of QSOs at $1.09<z<3.73$ using the VLT/X-Shooter near-infrared spectra publicly available. We measured the wavelength ratios of the two components of the spin-orbit doublet and accordingly calculated $\alpha(z)$ using two methods. Analysis on all of the 86 spectra yielded $\Delta\alpha/\alpha=(-3\pm6)\times10^{-5}$ with respect to the laboratory $\alpha$ measurements, consistent with no variation over the explored time interval. If assuming a uniform variation rate, we obtained $\alpha^{-1}{\rm d}\alpha/{\rm d}t = (-3\pm6)\times10^{-15}$ yr$^{-1}$ within the last 12 Gyrs. Extensive tests indicate that $\alpha$ variation could be better constrained using starburst galaxies' spectra than using QSO spectra in future studies.Comment: 24 pages, 22 figures. Accepted for publication in MNRA

LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear

OTOV2: Automatic, Generic, User-Friendly

The existing model compression methods via structured pruning typically require complicated multi-stage procedures. Each individual stage necessitates numerous engineering efforts and domain-knowledge from the end-users which prevent their wider applications onto broader scenarios. We propose the second generation of Only-Train-Once (OTOv2), which first automatically trains and compresses a general DNN only once from scratch to produce a more compact model with competitive performance without fine-tuning. OTOv2 is automatic and pluggable into various deep learning applications, and requires almost minimal engineering efforts from the users. Methodologically, OTOv2 proposes two major improvements: (i) Autonomy: automatically exploits the dependency of general DNNs, partitions the trainable variables into Zero-Invariant Groups (ZIGs), and constructs the compressed model; and (ii) Dual Half-Space Projected Gradient (DHSPG): a novel optimizer to more reliably solve structured-sparsity problems. Numerically, we demonstrate the generality and autonomy of OTOv2 on a variety of model architectures such as VGG, ResNet, CARN, ConvNeXt, DenseNet and StackedUnets, the majority of which cannot be handled by other methods without extensive handcrafting efforts. Together with benchmark datasets including CIFAR10/100, DIV2K, Fashion-MNIST, SVNH and ImageNet, its effectiveness is validated by performing competitively or even better than the state-of-the-arts. The source code is available at https://github.com/tianyic/only_train_once.Comment: Published on ICLR 2023. Remark here that a few images of dependency graphs can not be included in arXiv due to exceeding size limi