Measures of Information Reflect Memorization Patterns

Neural networks are known to exploit spurious artifacts (or shortcuts) that co-occur with a target label, exhibiting heuristic memorization. On the other hand, networks have been shown to memorize training examples, resulting in example-level memorization. These kinds of memorization impede generalization of networks beyond their training distributions. Detecting such memorization could be challenging, often requiring researchers to curate tailored test sets. In this work, we hypothesize -- and subsequently show -- that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization. We quantify the diversity in the neural activations through information-theoretic measures and find support for our hypothesis on experiments spanning several natural language and vision tasks. Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabelled in-distribution examples. Lastly, we demonstrate the utility of our findings for the problem of model selection. The associated code and other resources for this work are available at https://rachitbansal.github.io/information-measures.Comment: 22 pages; NeurIPS 2022. Code and data at https://rachitbansal.github.io/information-measure

Linear Connectivity Reveals Generalization Strategies

It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained. Under some circumstances, including transfer learning from pretrained models, these paths are presumed to be linear. In contrast to existing results, we find that among text classifiers (trained on MNLI, QQP, and CoLA), some pairs of finetuned models have large barriers of increasing loss on the linear paths between them. On each task, we find distinct clusters of models which are linearly connected on the test loss surface, but are disconnected from models outside the cluster -- models that occupy separate basins on the surface. By measuring performance on specially-crafted diagnostic datasets, we find that these clusters correspond to different generalization strategies: one cluster behaves like a bag of words model under domain shift, while another cluster uses syntactic heuristics. Our work demonstrates how the geometry of the loss surface can guide models towards different heuristic functions.Comment: Publushed as a conference paper at ICLR 202

LLM Augmented LLMs: Expanding Capabilities through Composition

Foundational models with billions of parameters which have been trained on large corpora of data have demonstrated non-trivial skills in a variety of domains. However, due to their monolithic structure, it is challenging and expensive to augment them or impart new skills. On the other hand, due to their adaptation abilities, several new instances of these models are being trained towards new domains and tasks. In this work, we study the problem of efficient and practical composition of existing foundation models with more specific models to enable newer capabilities. To this end, we propose CALM -- Composition to Augment Language Models -- which introduces cross-attention between models to compose their representations and enable new capabilities. Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using' existing LLMs along with a few additional parameters and data, (ii) Existing model weights are kept intact, and hence preserves existing capabilities, and (iii) Applies to diverse domains and settings. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13\% on tasks like translation into English and arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40\% over the base model for code generation and explanation tasks -- on-par with fully fine-tuned counterparts.Comment: 17 pages, 2 figures, 8 table

The Effect of Oligomerization on A Solid-Binding Peptide Binding to Silica-Based Materials

The bifunctional linker-protein G (LPG) fusion protein comprises a peptide (linker) sequence and a truncated form of Streptococcus strain G148 protein G (protein G). The linker represents a multimeric solid-binding peptide (SBP) comprising 4 × 21-amino acid sequence repeats that display high binding affinity towards silica-based materials. In this study, several truncated derivatives were investigated to determine the effect of the SBP oligomerization on the silica binding function of LPG (for the sake of clarity, LPG will be referred from here on as 4 × LPG). Various biophysical characterization techniques were used to quantify and compare the truncated derivatives against 4 × LPG and protein G without linker (PG). The derivative containing two sequence repeats (2 × LPG) showed minimal binding to silica, while the truncated derivative with only a single sequence (1 × LPG) displayed no binding. The derivative containing three sequence repeats (3 × LPG) was able to bind to silica with a binding affinity of KD = 53.23 ± 4.5 nM, which is 1.5 times lower than that obtained for 4 × LPG under similar experimental conditions. Circular dichroism (CD) spectroscopy and fluorescence spectroscopy studies indicated that the SBP degree of oligomerization has only a small effect on the secondary structure (the linker unravels the beginning of the protein G sequence) and chemical stability of the parent protein G. However, based on quartz crystal microbalance with dissipation monitoring (QCM-D), oligomerization is an important parameter for a strong and stable binding to silica. The replacement of three sequence repeats by a (GGGGS)12 glycine-rich spacer indicated that the overall length rather than the SBP oligomerization mediated the effective binding to silica

Probing the binding mechanism of a solid-binding peptide

Theoretical thesis.Includes bibliographic records.Chapter 1. Experimental and theoretical tools to elucidate the binding mechanisms of solid-binding peptides -- Chapter 2. Elucidating the binding mechanism of a novel silica-binding peptide -- Chapter 3. Effect of solid-binding peptide multimerization on its binding to silica-based materials -- Chapter 4. Solid-binding peptide-based biosensor for efficient detection of HER2 -- Chapter 5. Summary and future perspectives -- AppendixThe interactions between biomolecules and solid surfaces play an important role in designing new materials and applications which mimic nature. Recently, solid-binding peptides (SBPs) have emerged as potential molecular building blocks in nanobiotechnology. SBPs are short amino acid sequences (7-12 amino acids) that have the distinctive ability to recognize and bind to the surfaces of specific solid materials, such as metals and metal oxides, semiconductors, carbon-based materials, and polymers. These peptides act as 'molecular linkers' that mediate the simple and controlled attachment of biomolecules onto solid surfaces to confer biological functionality. SBP-solid interactions rely primarily on the peptide adopting structural conformations that maximize the contact between the reactive side chains of its amino acid residues and the solid surface. However, due to the high level of complexity of the SBP-solid interface within the surrounding solution, the exact mechanism that determines the SBP recognition, selectivity, and strong binding affinity remain ill-defined. We have engineered a bifunctional fusion protein composed of a solid-binding peptide (referred to as the 'linker'), which is a tetra-repeat of 21 amino acid sequence with unique binding affinity to silica-based materials, and a Streptococcus protein G, which binds antibodies. Linker protein G (LPG) acts as an anchor for the rapid and oriented immobilization of antibodies onto silica surfaces without using any complex conjugation chemistry. Although we have used this linker technology in biotechnology and biomedicine applications, there is still a lack of knowledge and understanding about the interaction mechanism which facilitates the binding of this SBP to the surface of silica. In this work, different biophysical characterization techniques, namely quartz crystal microbalance for dissipation monitoring (QCM-D), surface plasmon resonance (SPR), circular dichroism spectrometry (CD) and fluorescence spectrometry, were used to study the binding of LPG to silica surfaces and compared to protein G (PG) without the linker. LPG displayed high binding affinity to silica surface (KD=34.77±11.8 nM) with a standing-up orientation in comparison to parent PG, which exhibited no measurable binding affinity. Incorporation of the linker in the fusion protein LPG had no effect on the antibody binding function of PG, which retained its secondary structure and displayed no alteration of its chemical stability. We also engineered several truncated derivatives (1xLPG, 2xLPG, 3xLPG) from LPG to determine the effect of SBP multimerization on the silica binding function of LPG. The quantitative binding analysis for the different truncated derivatives were compared to that of LPG and PG (without linker) using various biophysical characterization techniques. Out of these truncated derivatives, 1xLPG (single linker sequence) displayed no binding to silica surface while the 2xLPG (two linker sequences) displayed minimal binding. Although the three-repeat derivative (3xLPG) binds to silica with a binding affinity (KD) of 53.23 ± 4.5 nM, it was 1.5 times lower than that of the four-repeat sequence (LPG). Spectroscopic techniques like circular dichroism (CD) spectroscopy and fluorescence spectroscopy studies indicated that the SBP degree of multimerization has no effect on the secondary structure and chemical stability of the parent protein G. However, the data from quartz crystal microbalance with dissipation monitoring (QCM-D) showed that multimerization was an important parameter for a strong and stable silica binding. The effect of peptide length on silica binding was evaluated by replacing the 3 sequence repeats by a (GGGGS)12 glycine-rich spacer. The results indicated that the overall length rather than the SBP multimerization mediated the effective binding to silica. A preliminary investigation was performed to assess the linker-protein G (LPG) as a suitable system for potential use in nanomedicine applications. We took advantage of the ability of the LPG bifunctional fusion protein to bind in an end-on-end orientation on the silica surface. The sensitive, selective and efficient detection of the HER2 biomarker was also investigated in the presence of biological fluids. This preliminary work showed that LPG mediates the functionalization of silica-coated nanoparticles with the anti-HER2 antibody trastuzumab easily and under mild conditions. The system was able to detect the HER2 biomarker in the presence of biological fluids demonstrating its potential suitability in nanomedicine applications e.g., nanoparticle-based drug delivery systems and efficient detection of disease biomarkers. This work provides the first insights into the binding mechanism of the aforementioned silica binding SBP. It also highlights the advantage of the LPG as a milder, facile and faster affinity immobilization technique of biomolecules to inorganic surfaces compared to traditional chemical coupling techniques --abstract.Mode of access: Internet.1 online resource (xi, 110 pages

Elucidating the Binding Mechanism of a Novel Silica-Binding Peptide

Linker-protein G (LPG) is a bifunctional fusion protein composed of a solid-binding peptide (SBP, referred as the “linker”) with high affinity to silica-based compounds and a Streptococcus protein G (PG), which binds antibodies. The binding mechanisms of LPG to silica-based materials was studied using different biophysical techniques and compared to that of PG without the linker. LPG displayed high binding affinity to a silica surface (KD = 34.77 ± 11.8 nM), with a vertical orientation, in comparison to parent PG, which exhibited no measurable binding affinity. Incorporation of the linker in the fusion protein, LPG, had no effect on the antibody-binding function of PG, which retained its secondary structure and displayed no alteration of its chemical stability. The LPG system provided a milder, easier, and faster affinity-driven immobilization of antibodies to inorganic surfaces when compared to traditional chemical coupling techniques

Comparison of aspartate aminotransferase platelet ratio index score and insulin resistance in type 2 diabetes mellitus with non-alcoholic fatty liver disease

Objective. Nonalcoholic fatty liver disease (NAFLD) is a spectrum of liver diseases characterized by the presence of ectopic fat in the liver and steatosis, which cannot be explained by alcohol consumption. The association between NAFLD and type 2 diabetes mellitus (T2DM) is well established. As liver fibrosis progresses in a patient with NAFLD, insulin resistance (IR) increases and may worsen diabetes control. The aspartate aminotransferase platelet ratio index (APRI) score is a simple and inexpensive bedside marker that can detect liver fibrosis and cirrhosis. Several studies have shown an association between APRI and NAFLD. However, there is a gap in correlation with IR in patients with diabetes. In this study, we sought to correlate IR and NAFLD in diabetes using the APRI score

Draft genome sequence of chickpea (Cicer arietinum) provides a resource for trait improvement

Chickpea (Cicer arietinum) is the second most widely grown legume crop after soybean, accounting for a substantial proportion of human dietary nitrogen intake and playing a crucial role in food security in developing countries. We report the ∼738-Mb draft whole genome shotgun sequence of CDC Frontier, a kabuli chickpea variety, which contains an estimated 28,269 genes. Resequencing and analysis of 90 cultivated and wild genotypes from ten countries identifies targets of both breeding-associated genetic sweeps and breeding-associated balancing selection. Candidate genes for disease resistance and agronomic traits are highlighted, including traits that distinguish the two main market classes of cultivated chickpea - desi and kabuli. These data comprise a resource for chickpea improvement through molecular breeding and provide insights into both genome diversity and domestication. Copyrigh