Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture. Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks.Comment: Accepted at AAAI24(https://aaai.org/aaai-conference/

Ecosystem Consequences of Plant Genetic Divergence with Colonization of New Habitat

When plants colonize new habitats altered by natural or anthropogenic disturbances, those individuals may encounter biotic and abiotic conditions novel to the species, which can cause plant functional trait divergence. Over time, site-driven adaptation can give rise to population-level genetic variation, with consequences for plant community dynamics and ecosystem processes. We used a series of 3000-yr-old, lava-created forest fragments on the Island of Hawai`i to examine whether disturbance and subsequent colonization can lead to genetically differentiated populations, and where differentiation occurs, if there are ecosystem consequences of trait-driven changes. These fragments are dominated by a single tree species, Metrosideros polymorpha (Myrtaceae) or ʻohiʻa, which have been actively colonizing the surrounding lava flow created in 1858. To test our ideas about differentiation of genetically determined traits, we (1) created rooted cuttings of ʻohiʻa individuals sampled from fragment interiors and open lava sites, raised these individuals in a greenhouse, and then used these cuttings to create a common garden where plant growth was monitored for three years; and (2) assessed genetic variation and made QST/FST comparisons using microsatellite repeat markers. Results from the greenhouse showed quantitative trait divergence in plant height and pubescence across plants sampled from fragment interior and matrix sites. Results from the subsequent common garden study confirmed that the matrix environment can select for individuals with 9.1% less shoot production and 17.3% higher leaf pubescence. We found no difference in molecular genetic variation indicating gene flow among the populations. The strongest QST level was greater than the FST estimate, indicating sympatric genetic divergence in growth traits. Tree height was correlated with ecosystem properties such as soil carbon and nitrogen storage, soil carbon turnover rates, and soil phosphatase activity, indicating that selection for growth traits will influence structure, function, and dynamics of developing ecosystems. These data show that divergence can occur on centennial timescales of early colonization

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract)

This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these ”attentionless Transformers” to rival the performance of the original architecture. Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks

Multiple mutations in the para-sodium channel gene are associated with pyrethroid resistance in Rhipicephalus microplus from the United States and Mexico.

Acaricide resistant Rhipicephalus microplus populations have become a major problem for many cattle producing areas of the world. Pyrethroid resistance in arthropods is typically associated with mutations in domains I, II, III, and IV of voltage-gated sodium channel genes. In R. microplus, known resistance mutations include a domain II change (C190A) in populations from Australia, Africa, and South America and a domain III mutation (T2134A) that only occurs in Mexico and the U.S. We investigated pyrethroid resistance in cattle fever ticks from Texas and Mexico by estimating resistance levels in field-collected ticks using larval packet discriminating dose (DD) assays and identifying single nucleotide polymorphisms (SNPs) in the para-sodium channel gene that associated with resistance. We then developed qPCR assays for three SNPs and screened a larger set of 1,488 R. microplus ticks, representing 77 field collections and four laboratory strains, for SNP frequency. We detected resistance SNPs in 21 of 68 U.S. field collections and six of nine Mexico field collections. We expected to identify the domain III SNP (T2134A) at a high frequency; however, we only found it in three U.S. collections. A much more common SNP in the U.S. (detected in 19 of 21 field collections) was the C190A domain II mutation, which has never before been reported from North America. We also discovered a novel domain II SNP (T170C) in ten U.S. and two Mexico field collections. The T170C transition mutation has previously been associated with extreme levels of resistance (super-knockdown resistance) in insects. We found a significant correlation (r = 0.81) between the proportion of individuals in field collections that carried any two resistance SNPs and the percent survivorship of F1 larvae from these collections in DD assays. This relationship is accurately predicted by a simple linear regression model (R2 = 0.6635). These findings demonstrate that multiple mutations in the para-sodium channel gene independently associate with pyrethroid resistance in R. microplus ticks, which is likely a consequence of human-induced selection

Identification of Susceptibility Genes for Cancer in a Genome-wide Scan: Results from the Colon Neoplasia Sibling Study

Colorectal cancer (CRC) is the third most commonly diagnosed cancer in Americans and is the second leading cause of cancer mortality. Only a minority (∼5%) of familial CRC can be explained by known genetic variants. To identify susceptibility genes for familial colorectal neoplasia, the colon neoplasia sibling study conducted a comprehensive, genome-wide linkage scan of 194 kindreds. Clinical information (histopathology, size and number of polyps, and other primary cancers) was used in conjunction with age at onset and family history for classification of the families into five phenotypic subgroups (severe histopathology, oligopolyposis, young, colon/breast, and multiple cancer) prior to analysis. By expanding the traditional affected-sib-pair design to include unaffected and discordant sib pairs, analytical power and robustness to type I error were increased. Sib-pair linkage statistics and Haseman-Elston regression identified 19 linkage peaks, with interesting results for chromosomes 1p31.1, 15q14-q22, 17p13.3, and 21. At marker D1S1665 (1p31.1), there was strong evidence for linkage in the multiple-cancer subgroup (p = 0.00007). For chromosome 15q14-q22, a linkage peak was identified in the full-sample (p = 0.018), oligopolyposis (p = 0.003), and young (p = 0.0009) phenotypes. This region includes the HMPS/CRAC1 locus associated with hereditary mixed polyposis syndrome (HMPS) in families of Ashkenazi descent. We provide compelling evidence linking this region in families of European descent with oligopolyposis and/or young age at onset (≤51) phenotypes. We found linkage to BRCA2 in the colon/breast phenotypic subgroup and identified a second locus in the region of D21S1437 segregating with, but distinct from, BRCA2. Linkage to 17p13.3 at marker D17S1308 in the breast/colon subgroup identified HIC1 as a candidate gene. We demonstrated that using clinical information, unaffected siblings, and family history can increase the analytical power of a linkage study

More than 50% of <i>Clostridium difficile</i> Isolates from Pet Dogs in Flagstaff, USA, Carry Toxigenic Genotypes

<div><p>Nosocomial acquisition of <i>Clostridium difficile</i> is well documented, yet recent studies have highlighted the importance of community acquired infections and identified community associated reservoirs for this pathogen. Multiple studies have implicated companion pets and farm animals as possible sources of community acquired <i>C</i>. <i>difficile</i> infections in humans. To explore the potential role of pet dogs in human <i>C</i>. <i>difficile</i> infections we systematically collected canine fecal samples (<i>n</i> = 197) in Flagstaff, AZ. Additionally, nineteen fecal samples were collected at a local veterinary clinic from diarrheic dogs. We used these combined samples to investigate important questions regarding <i>C</i>. <i>difficile</i> colonization in pet canines: 1) What is the prevalence and diversity of <i>C</i>. <i>difficile</i> in this companion pet population, and 2) Do <i>C</i>. <i>difficile</i> isolates collected from canines genetically overlap with isolates that cause disease in humans? We used a two-step sequence typing approach, including multilocus sequence typing to determine the overall genetic diversity of <i>C</i>. <i>difficile</i> present in Flagstaff canines, and whole-genome sequencing to assess the fine-scale diversity patterns within identical multilocus sequence types from isolates obtained within and among multiple canine hosts. We detected <i>C</i>. <i>difficile</i> in 17% of the canine fecal samples with 10% containing toxigenic strains that are known to cause human disease. Sequencing analyses revealed similar genotypes in dogs and humans. These findings suggest that companion pets are a potential source of community acquired <i>C</i>. <i>difficile</i> infections in humans.</p></div