Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models

Large generative models such as large language models (LLMs) and diffusion models have revolutionized the fields of NLP and computer vision respectively. However, their slow inference, high computation and memory requirement makes it challenging to deploy them on edge devices. In this study, we propose a light-weight quantization aware fine tuning technique using knowledge distillation (KD-QAT) to improve the performance of 4-bit weight quantized LLMs using commonly available datasets to realize a popular language use case, on device chat applications. To improve this paradigm of finetuning, as main contributions, we provide insights into stability of KD-QAT by empirically studying the gradient propagation during training to better understand the vulnerabilities of KD-QAT based approaches to low-bit quantization errors. Based on our insights, we propose ov-freeze, a simple technique to stabilize the KD-QAT process. Finally, we experiment with the popular 7B LLaMAv2-Chat model at 4-bit quantization level and demonstrate that ov-freeze results in near floating point precision performance, i.e., less than 0.7% loss of accuracy on Commonsense Reasoning benchmarks.Comment: Accepted at Practical ML for Low Resource Settings Workshop at ICLR 202

Seasonal evolution of the Yellow Sea Cold Water Mass and its interactions with ambient hydrodynamic system

The Yellow Sea Cold Water Mass (YSCWM) is an important component of the hydrodynamic system in the South Yellow Sea (SYS). However, its intricate interactions with the ambient flows over long time scales are not fully understood. This paper presents the analysis of the data set obtained from a seabed‐mounted Acoustic Doppler Current Profiler (ADCP) deployed for nearly 1 year in the western SYS. It allowed us to study the evolution of YSCWM, including the seasonal changes of tidal currents, near‐inertial oscillations (NIOs), and the wind‐driven currents due to typhoons and winter storms. Strong NIOs were found near the bottom of mixed layer and in the pycnocline with nearly opposite current directions, with maximum velocity of nearly 20 cm·s−1 in summer. The YSCWM can also inhibit the direct downward energy transport in the water column due to typhoons. Conversely, the hydrodynamic system also feeds back to influence the change of YSCWM. A large current shear (S) of 20 cm·s−1·m−1 is generated near the top of pycnocline. Generally, the intensity and depth of the pycnocline determine S's magnitude and vertical location, respectively. Based on the monthly averaged density profile data, the Richardson number and wavelet analysis, the NIOs are considered to be capable of inducing predominant shear instability around the pycnocline. However, the NIOs are not strong enough to influence the lower YSCWM. In addition, in autumn, each fortnightly spring tide corresponds with a bottom temperature increase of nearly 2°C, indicating that tidal currents are the leading hydrodynamic driving force to decline the YSCWM

Hydrodynamic condition and suspended sediment diffusion in the Yellow Sea and East China Sea

Based on monthly averaged current, temperature, and salinity, we analyzed the changes of suspended sediment concentration (SSC) and the relationship with the warm current, coastal current, and cold water mass (CWM) in the East China Seas (ECSs). The result shows that the coastal current and surface diluted water are the route for transporting suspended sediment. The Kuroshio and its derived warm current branches play the important role of the continental shelf circulation system and control the diffusion of suspended sediment. High SSC has been mainly concentrated in coastal current and CWM. Two sedimentary dynamic patterns have been identified. The winter‐half‐year pattern lasts almost 7 months. The coastal currents off the Shandong Peninsula, northern Jiangsu, Zhejiang‐Fujian coast are the main routes for diffusion and deposition of the suspended sediment from the Yellow River and Changjiang River. The summer‐half‐year pattern is characterized by the well‐developed CWM. All CWMs have a unique function to trap suspended sediment under the thermocline due to weakening tidal current and residual current there. These CWMs in the Yellow Sea (YS) and north ECS are connected together. The layer above the thermocline is characterized by diluted water with low salinity, high temperature. Suspended sediment can be transported into the Okinawa Trough and the South Korea coast during this period. A strong eddy always occur nearby the Kuroshio bend at northeast Taiwan, which has promoted the exchange between the ECS shelf and Okinawa Trough, and the development of the shelf edge current and Taiwan warm current (TWC)

Dissociating Markers of Senescence and Protective Ability in Memory T Cells

No unique transcription factor or biomarker has been identified to reliably distinguish effector from memory T cells. Instead a set of surface markers including IL-7Rα and KLRG1 is commonly used to predict the potential of CD8 effector T cells to differentiate into memory cells. Similarly, these surface markers together with the tumor necrosis factor family member CD27 are frequently used to predict a memory T cell's ability to mount a recall response. Expression of these markers changes every time a memory cell is stimulated and repeated stimulation can lead to T cell senescence and loss of memory T cell responsiveness. This is a concern for prime–boost vaccine strategies which repeatedly stimulate T cells with the aim of increasing memory T cell frequency. The molecular cues that cause senescence are still unknown, but cell division history is likely to play a major role. We sought to dissect the roles of inflammation and cell division history in developing T cell senescence and their impact on the expression pattern of commonly used markers of senescence. We developed a system that allows priming of CD8 T cells with minimal inflammation and without acquisition of maximal effector function, such as granzyme expression, but a cell division history similar to priming with systemic inflammation. Memory cells derived from minimal effector T cells are fully functional upon rechallenge, have full access to non-lymphoid tissue and appear to be less senescent by phenotype upon rechallenge. However, we report here that these currently used biomarkers to measure senescence do not predict proliferative potential or protective ability, but merely reflect initial priming conditions

An asymmetric upwind flow, Yellow Sea Warm Current : 2. Arrested topographic waves in response to the northwesterly wind

Author Posting. © American Geophysical Union, 2011. This article is posted here by permission of American Geophysical Union for personal use, not for redistribution. The definitive version was published in Journal of Geophysical Research 116 (2011): C04027, doi:10.1029/2010JC006514.A warm and salty water mass exists along the Yellow Sea Trough (YST) in winter. This oceanic water mass is distinct from the ambient shelf water and is distributed on the western side of the YST. It has long been reasoned that a Yellow Sea Warm Current (YSWC) must exist. A recent observational study indeed supports the existence of the YSWC and shows that its position moved progressively westward as the warm water intrudes further shoreward toward the northwest. In this paper, we explain mechanisms for sustaining the YSWC and for its westward displacement. The northwesterly monsoonal wind prevails in the winter and is directed against the YSWC. The cross-trough scale is small compared with the spatial scale of monsoonal variation, so one can assume, to the first order, that the wind stress is uniform across the trough. The curl of depth-averaged wind stress has opposite signs on the two sides of the trough. Consequently, two oppositely rotating gyres develop initially and they converge along the trough giving rise to a barotropic upwind flow. But this upwind flow lasts only for a few days as the two gyres evolve and propagate as topographic waves. For a northerly wind, both gyres move westward since the positive (negative) potential vorticity flux on the western (eastern) side of the trough pushes the water toward shore (trough). If the bottom friction is negligible, the steady response becomes a large anticyclonic gyre over the trough and the upwind current is squeezed toward the shore line. In this case, no YSWC is sustained along or near the trough. This runaway warm current can be arrested by a moderate bottom friction. We therefore propose that the YSWC is actually arrested topographic waves in response to local wind stress forcing.X.L. has been supported by China’s National Basic Research Priorities Programmer (2007CB411804 and 2005CB422303), the Ministry of Education’s 111 Project (B07036), the Program for New Century Excellent Talents in University (NECT‐07‐ 0781), and the China National Science Foundation (40976004, 40921004, and 40930844). J.Y. has been supported by the U.S. National Science Foundation and the Woods Hole Oceanographic Institution’s Coastal Ocean Institute

The Kuroshio Extension : a leading mechanism for the seasonal sea-level variability along the west coast of Japan

Author Posting. © The Author(s), 2009. This is the author's version of the work. It is posted here by permission of Springer for personal use, not for redistribution. The definitive version was published in Ocean Dynamics 60 (2010): 667-672, doi:10.1007/s10236-009-0239-9.Sea level changes coherently along the two coasts of Japan on the seasonal time scale. AVISO satellite altimetry data and OFES (OGCM for the Earth Simulator) results indicate that the variation propagates clockwise from Japan's east coast through the Tsushima Strait into the Japan/East Sea (JES) and then northward along the west coast. In this study, we hypothesize and test numerically that the sea level variability along the west coast of Japan is remotely forced by the Kuroshio Extension (KE) off the east coast. Topographic Rossby waves and boundary Kelvin waves facilitate the connection. Our 3-d POM model when forced by observed wind stress reproduces well the seasonal changes in the vicinity of JES. Two additional experiments were conducted to examine the relative roles of remote forcing and local forcing. The sea level variability inside the JES was dramatically reduced when the Tsushima Strait is blocked in one experiment. The removal of the local forcing, in another experiment, has little effect on the JES variability. Both experiments support our hypothesis that the open-ocean forcing, possibly through the KE variability, is the leading forcing mechanism for sea level change along the west coast of Japan.This work was conducted when Chao Ma was a visiting graduate student at WHOI. His visit has been supported by China Scholarship Council and WHOI Academics Office. This study has been supported by WHOI’s Coastal Ocean Institute, the National Basic Research Program of China 2005CB422303 and 2007CB481804), the International Science and Technology Cooperation Program of China (2006DFB21250), the Natural Science Foundation of China (40706006) , and the Ministry of Education’s 111 Project (B07036). Lin was supported by the Program for New Century Excellent Talents in University (NECT-07-0781)

An asymmetric upwind flow, Yellow Sea Warm Current : 1. New observations in the western Yellow Sea

Author Posting. © American Geophysical Union, 2011. This article is posted here by permission of American Geophysical Union for personal use, not for redistribution. The definitive version was published in Journal of Geophysical Research 116 (2011): C04026, doi:10.1029/2010JC006513.The winter water mass along the Yellow Sea Trough (YST), especially on the western side of the trough, is considerably warmer and saltier than the ambient shelf water mass. This observed tongue-shape hydrographic feature implies the existence of a winter along-trough and onshore current, often referred to as the Yellow Sea Warm Current (YSWC). However, the YSWC has not been confirmed by direct current measurements and therefore skepticism remains regarding its existence. Some studies suggest that the presence of the warm water could be due to frontal instability, eddies, or synoptic scale wind bursts. It is noted that in situ observations used in most previous studies were from the central and eastern sides of the YST even though it is known that the warm water core is more pronounced along the western side. Data from the western side have been scarce. Here we present a set of newly available Chinese observations, including some from a coordinated effort involving three Chinese vessels in the western YST during the 2006–2007 winter. The data show unambiguously the existence of the warm current on the western side of YST. Both the current and hydrography observations indicate a dominant barotropic structure of YSWC. The westward deviation of YSWC axis is particularly obvious to the south of 35°N and is clearly associated with an onshore movement of warm water. To the north of 35°N, the YSWC flows along the bathymetry with slightly downslope movement. We conclude that the barotropic current is mainly responsible for the warm water intrusion, while the Ekman and baroclinic currents play an important but secondary role. These observations help fill an observational gap and establish a more complete view of the YSWC.The authors have been supported by China’s National Basic Research Priorities Programmer (2007CB411804 and 2005CB422303), the Ministry of Education’s 111 Project (B07036), the Program for New Century Excellent Talents in University (NECT‐07‐0781), and the China National Science Fundation (40976004, 40921004. and 40930844). J.Y. is supported by the U.S. National Science Foundation and the Woods Hole Oceanographic Institution’s Coastal Ocean Institute

Omalizumab may decrease IgE synthesis by targeting membrane IgE+ human B cells

Omalizumab, is a humanized anti-IgE monoclonal antibody used to treat allergic asthma. Decreased serum IgE levels, lower eosinophil and B cell counts have been noted as a result of treatment. In vitro studies and animal models support the hypothesis that omalizumab inhibits IgE synthesis by B cells and causes elimination of IgE-expressing cells either by induction of apoptosis or induction of anergy or tolerance. METHODS: We examined the influence of omalizumab on human tonsillar B cell survival and on the genes involved in IgE synthesis. Tonsillar B cells were stimulated with IL-4 plus anti-CD40 antibody to induce class switch recombination to IgE production in the presence or absence of omalizumab. Cell viability was assessed and RNA extracted to examine specific genes involved in IgE synthesis. CONCLUSIONS: We found that omalizumab reduced viable cell numbers but this was not through induction of apoptosis. IL-4R and germline Cϵ mRNA levels were decreased as well as the number of membrane IgE+ cells in B cells treated with omalizumab. These data suggest that omalizumab may decrease IgE synthesis by human B cells by specifically targeting membrane IgE-bearing B cells and inducing a state of anergy

Using Multiple Microenvironments to Find Similar Ligand-Binding Sites: Application to Kinase Inhibitor Binding

The recognition of cryptic small-molecular binding sites in protein structures is important for understanding off-target side effects and for recognizing potential new indications for existing drugs. Current methods focus on the geometry and detailed chemical interactions within putative binding pockets, but may not recognize distant similarities where dynamics or modified interactions allow one ligand to bind apparently divergent binding pockets. In this paper, we introduce an algorithm that seeks similar microenvironments within two binding sites, and assesses overall binding site similarity by the presence of multiple shared microenvironments. The method has relatively weak geometric requirements (to allow for conformational change or dynamics in both the ligand and the pocket) and uses multiple biophysical and biochemical measures to characterize the microenvironments (to allow for diverse modes of ligand binding). We term the algorithm PocketFEATURE, since it focuses on pockets using the FEATURE system for characterizing microenvironments. We validate PocketFEATURE first by showing that it can better discriminate sites that bind similar ligands from those that do not, and by showing that we can recognize FAD-binding sites on a proteome scale with Area Under the Curve (AUC) of 92%. We then apply PocketFEATURE to evolutionarily distant kinases, for which the method recognizes several proven distant relationships, and predicts unexpected shared ligand binding. Using experimental data from ChEMBL and Ambit, we show that at high significance level, 40 kinase pairs are predicted to share ligands. Some of these pairs offer new opportunities for inhibiting two proteins in a single pathway

Spectroscopic scanning tunneling microscopy insights into Fe-based superconductors

In the first three years since the discovery of Fe-based high Tc superconductors, scanning tunneling microscopy (STM) and spectroscopy have shed light on three important questions. First, STM has demonstrated the complexity of the pairing symmetry in Fe-based materials. Phase-sensitive quasiparticle interference (QPI) imaging and low temperature spectroscopy have shown that the pairing order parameter varies from nodal to nodeless s\pm within a single family, FeTe1-xSex. Second, STM has imaged C4 -> C2 symmetry breaking in the electronic states of both parent and superconducting materials. As a local probe, STM is in a strong position to understand the interactions between these broken symmetry states and superconductivity. Finally, STM has been used to image the vortex state, giving insights into the technical problem of vortex pinning, and the fundamental problem of the competing states introduced when superconductivity is locally quenched by a magnetic field. Here we give a pedagogical introduction to STM and QPI imaging, discuss the specific challenges associated with extracting bulk properties from the study of surfaces, and report on progress made in understanding Fe-based superconductors using STM techniques.Comment: 36 pages, 23 figures, 229 reference