FP8-BERT: Post-Training Quantization for Transformer

Transformer-based models, such as BERT, have been widely applied in a wide range of natural language processing tasks. However, one inevitable side effect is that they require massive memory storage and inference cost when deployed in production. Quantization is one of the popularized ways to alleviate the cost. However, the previous 8-bit quantization strategy based on INT8 data format either suffers from the degradation of accuracy in a Post-Training Quantization (PTQ) fashion or requires an expensive Quantization-Aware Training (QAT) process. Recently, a new numeric format FP8 (i.e. floating-point of 8-bits) has been proposed and supported in commercial AI computing platforms such as H100. In this paper, we empirically validate the effectiveness of FP8 as a way to do Post-Training Quantization without significant loss of accuracy, with a simple calibration and format conversion process. We adopt the FP8 standard proposed by NVIDIA Corp. (2022) in our extensive experiments of BERT variants on GLUE and SQuAD v1.1 datasets, and show that PTQ with FP8 can significantly improve the accuracy upon that with INT8, to the extent of the full-precision model

Deep Learning to Improve the Sensitivity of Di-Higgs Searches in the 4b4b Channel

The study of di-Higgs events, both resonant and non-resonant, plays a crucial role in understanding the fundamental interactions of the Higgs boson. In this work we consider di-Higgs events decaying into four $b$-quarks and propose to improve the experimental sensitivity by utilizing a novel machine learning algorithm known as Symmetry Preserving Attention Network (\textsc{Spa-Net}) -- a neural network structure whose architecture is designed to incorporate the inherent symmetries in particle reconstruction tasks. We demonstrate that the \textsc{Spa-Net} can enhance the experimental reach over baseline methods such as the cut-based and the Deep Neural Networks (DNN)-based analyses. At the Large Hadron Collider, with a 14-TeV centre-of-mass energy and an integrated luminosity of 300 fb$^{-1}$, the \textsc{Spa-Net} allows us to establish 95\% C.L. upper limits in resonant production cross-sections that are 10\% to 45\% stronger than baseline methods. For non-resonant di-Higgs production, \textsc{Spa-Net} enables us to constrain the self-coupling that is 9\% more stringent than the baseline method

In vitro human haematopoietic stem cell expansion and differentiation

The haematopoietic system plays an essential role in our health and survival. It is comprised of a range of mature blood and immune cell types, including oxygen-carrying erythrocytes, platelet-producing megakaryocytes and infection-fighting myeloid and lymphoid cells. Self-renewing multipotent haematopoietic stem cells (HSCs) and a range of intermediate haematopoietic progenitor cell types differentiate into these mature cell types to continuously support haematopoietic system homeostasis throughout life. This process of haematopoiesis is tightly regulated in vivo and primarily takes place in the bone marrow. Over the years, a range of in vitro culture systems have been developed, either to expand haematopoietic stem and progenitor cells or to differentiate them into the various haematopoietic lineages, based on the use of recombinant cytokines, co-culture systems and/or small molecules. These approaches provide important tractable models to study human haematopoiesis in vitro. Additionally, haematopoietic cell culture systems are being developed and clinical tested as a source of cell products for transplantation and transfusion medicine. This review discusses the in vitro culture protocols for human HSC expansion and differentiation, and summarises the key factors involved in these biological processes

Computational fluid dynamic modeling of 100ml and scaled-down 10ml stirred suspension bioreactors enables prediction of embryonic stem cell characteristics

There is a growing necessity for cell cultivation using bioreactors to translate laboratory based culture protocols into reproducible, scalable, and robust bioprocesses. Stirred suspension bioreactors offer several advantages over planar static cultures, including: reduced labour and operating costs, reduced space requirements, greater cellular homogeneity, and increased cell density per volume [1]. An important consideration when using stirred suspension bioreactors is mechanical stimulation. Fluid shear at the fluid-cell interface triggers cellular responses through mechanotransduction and can modulate stem cell proliferation and differentiation. However, if the shear stress caused by the impeller exceeds the tolerance limit of the cells, it causes cell damage and death, resulting in a lower quality and yield of cells. The shear rate distribution depends on bioreactor geometry, impeller agitation rate, cell density, and cell media viscosity [2]. Current scale-up protocols to predict agitation rates rely on maximum values of hydrodynamic variables, which occur only at the impeller tip. The volume averaged shear stress and maximum shear stress differ greatly, and cells dispersed within the liquid experience different local and global forces. This makes it difficult to predict how cells will respond to changes in bioreactor geometries and sizes. Profiling distributed and average forces in the bioreactor is critical to ensure quality and yield in cell manufacturing. Hydrodynamics, specifically velocities, shear rates, and energy dissipation rates, can be studied using computational fluid dynamic (CFD) modeling. Please click Additional Files below to see the full abstract