Fine-Tuning Pre-trained Transformers into Decaying Fast Weights

Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based methods to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. We explore these approaches and find that they are unnecessarily complex, and propose a simple alternative - decaying fast weights - that runs fast on GPU, outperforms prior methods, and retains 99% of attention's performance for GPT-2. We also show competitive performance on WikiText-103 against more complex attention substitutes

Pagoamide A, a Cyclic Depsipeptide Isolated from a Cultured Marine Chlorophyte, Derbesia sp., Using MS/MS-Based Molecular Networking

A thiazole-containing cyclic depsipeptide with 11 amino acid residues, named pagoamide A (1), was isolated from laboratory cultures of a marine Chlorophyte, Derbesia sp. This green algal sample was collected from America Samoa, and pagoamide A was isolated using guidance by MS/MS-based molecular networking. Cultures were grown in a light- and temperature-controlled environment and harvested after several months of growth. The planar structure of pagoamide A (1) was characterized by detailed 1D and 2D NMR experiments along with MS and UV analysis. The absolute configurations of its amino acid residues were determined by advanced Marfey's analysis following chemical hydrolysis and hydrazinolysis reactions. Two of the residues in pagoamide A (1), phenylalanine and serine, each occurred twice in the molecule, once in the d- and once in the l-configuration. The biosynthetic origin of pagoamide A (1) was considered in light of other natural products investigations with coenocytic green algae

A Convolutional Neural Network-Based Approach for the Rapid Annotation of Molecularly Diverse Natural Products

This report describes the first application of the novel NMR-based machine learning tool "Small Molecule Accurate Recognition Technology" (SMART 2.0) for mixture analysis and subsequent accelerated discovery and characterization of new natural products. The concept was applied to the extract of a filamentous marine cyanobacterium known to be a prolific producer of cytotoxic natural products. This environmental Symploca extract was roughly fractionated, and then prioritized and guided by cancer cell cytotoxicity, NMR-based SMART 2.0, and MS2-based molecular networking. This led to the isolation and rapid identification of a new chimeric swinholide-like macrolide, symplocolide A, as well as the annotation of swinholide A, samholides A-I, and several new derivatives. The planar structure of symplocolide A was confirmed to be a structural hybrid between swinholide A and luminaolide B by 1D/2D NMR and LC-MS2 analysis. A second example applies SMART 2.0 to the characterization of structurally novel cyclic peptides, and compares this approach to the recently appearing "atomic sort" method. This study exemplifies the revolutionary potential of combined traditional and deep learning-assisted analytical approaches to overcome longstanding challenges in natural products drug discovery