Advancing Regular Language Reasoning in Linear Recurrent Neural Networks

In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language modeling and long-range modeling while offering rapid parallel training and constant inference costs. With the resurged interest in LRNNs, we study whether they can learn the hidden rules in training sequences, such as the grammatical structures of regular language. We theoretically analyze some existing LRNNs and discover their limitations on regular language. Motivated by the analysis, we propose a new LRNN equipped with a block-diagonal and input-dependent transition matrix. Experiments suggest that the proposed model is the only LRNN that can perform length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic.Comment: The first two authors contributed equally to this wor

Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation

An ideal length-extrapolatable Transformer language model can handle sequences longer than the training length without any fine-tuning. Such long-context utilization capability relies heavily on a flexible positional embedding design. Upon investigating the flexibility of existing large pre-trained Transformer language models, we find that the T5 family deserves a closer look, as its positional embeddings capture rich and flexible attention patterns. However, T5 suffers from the dispersed attention issue: the longer the input sequence, the flatter the attention distribution. To alleviate the issue, we propose two attention alignment strategies via temperature scaling. Our findings show improvement on the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. This suggests that a flexible positional embedding design and attention alignment can go a long way toward Transformer length extrapolation

Disordered Fe vacancies and superconductivity in potassium-intercalated iron selenide (K2-xFe4+ySe5)

The parent compound of an unconventional superconductor must contain unusual correlated electronic and magnetic properties of its own. In the high-Tc potassium intercalated FeSe, there has been significant debate regarding what the exact parent compound is. Our studies unambiguously show that the Fe-vacancy ordered K2Fe4Se5 is the magnetic, Mott insulating parent compound of the superconducting state. Non-superconducting K2Fe4Se5 becomes a superconductor after high temperature annealing, and the overall picture indicates that superconductivity in K2-xFe4+ySe5 originates from the Fe-vacancy order to disorder transition. Thus, the long pending question whether magnetic and superconducting state are competing or cooperating for cuprate superconductors may also apply to the Fe-chalcogenide superconductors. It is believed that the iron selenides and related compounds will provide essential information to understand the origin of superconductivity in the iron-based superconductors, and possibly to the superconducting cuprates

Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.Comment: Accepted by ACL 202

Label-free quantitative proteomics of CD133-positive liver cancer stem cells

Abstract Background CD133-positive liver cancer stem cells, which are characterized by their resistance to conventional chemotherapy and their tumor initiation ability at limited dilutions, have been recognized as a critical target in liver cancer therapeutics. In the current work, we developed a label-free quantitative method to investigate the proteome of CD133-positive liver cancer stem cells for the purpose of identifying unique biomarkers that can be utilized for targeting liver cancer stem cells. Label-free quantitation was performed in combination with ID-based Elution time Alignment by Linear regression Quantitation (IDEAL-Q) and MaxQuant. Results Initially, IDEAL-Q analysis revealed that 151 proteins were differentially expressed in the CD133-positive hepatoma cells when compared with CD133-negative cells. We then analyzed these 151 differentially expressed proteins by MaxQuant software and identified 10 significantly up-regulated proteins. The results were further validated by RT-PCR, western blot, flow cytometry or immunofluorescent staining which revealed that prominin-1, annexin A1, annexin A3, transgelin, creatine kinase B, vimentin, and EpCAM were indeed highly expressed in the CD133-positive hepatoma cells. Conclusions These findings confirmed that mass spectrometry-based label-free quantitative proteomics can be used to gain insights into liver cancer stem cells.http://deepblue.lib.umich.edu/bitstream/2027.42/113089/1/12953_2012_Article_407.pd

Coenzyme Q0 from Antrodia cinnamomea

We investigated the anticancer effects of Antrodia cinnamomea, a medicinal mushroom from Taiwan, on A549 human lung cancer cells using the ethyl acetate extract from submerged culture filtrates. Our results showed that 2,3-dimethoxy-5-methyl-1,4-benzoquinone (coenzyme Q0; CoQ0) derived from A. cinnamomea submerged culture filtrates has anticancer activity. CoQ0 treatment reduced the viability of A549, HepG2, and SW480 cancer cell lines. Furthermore, CoQ0 induced reactive oxygen species (ROS) generation and apoptosis in A549 cells, which was inhibited by the antioxidant ascorbic acid. To our knowledge, these data demonstrate for the first time that CoQ0 derived from A. cinnamomea submerged culture filtrates exerts its anticancer effect through the induction of ROS-mediated apoptosis in A549 human lung cancer cells

Synthesis and characterization of visible-light-driven novel CuTa2O6 as a promising practical photocatalyst

In this work, the novel CuTa2O6 phase was successfully synthesized by the hydrothermal and followed by the calcination process. The X-ray diffraction pattern confirms the formation of different phases. At a low temperature, CuTa2O6 exhibits the orthorhombic phase, whereas, at a higher temperature, it underwent a phase transition to a cubic crystal structure. X-ray photoelectron spectroscopic results suggest the presence of all the elements (Cu, Ta, and O). The optical studies were carried out using a UV-Vis DRS spectrophotometer. FESEM images confirm the spherical-shaped particles for the sample annealed at a high temperature. The local atomic and electronic structures around Cu and the contribution of the Cu oxidation state in the CuTa2O6 system were determined by X-ray absorption spectroscopy. To investigate the effective usage of CuTa2O6 in treating wastewater, its photocatalytic activity was investigated by evaluating its use in the photodegradation of MO dye under visible light irradiation. Moreover, the prepared CuTa2O6 photocatalyst exhibits significant photocatalytic activity in the degradation of MO dye and shows excellent stability; it is therefore a promising material for potential use in a practical photocatalyst. The CuTa2O6 photocatalyst suggests an alternative avenue of research into effective photo-catalysts for solar hydrogen water splitting

Molecular Analysis of Precursor Lesions in Familial Pancreatic Cancer

PMCID: PMC3553106This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

Invasive fungal infection among hematopoietic stem cell transplantation patients with mechanical ventilation in the intensive care unit

<p>Abstract</p> <p>Background</p> <p>Invasive fungal infection (IFI) is associated with high morbidity and high mortality in hematopoietic stem cell transplantation (HSCT) patientsThe purpose of this study was to assess the characteristics and outcomes of HSCT patients with IFIs who are undergoing MV at a single institution in Taiwan.</p> <p>Methods</p> <p>We performed an observational retrospective analysis of IFIs in HSCT patients undergoing mechanical ventilation (MV) in an intensive care unit (ICU) from the year 2000 to 2009. The characteristics of these HSCT patients and risk factors related to IFIs were evaluated. The status of discharge, length of ICU stay, date of death and cause of death were also recorded.</p> <p>Results</p> <p>There were 326 HSCT patients at the Linkou Chang-Gung Memorial Hospital (Taipei, Taiwan) during the study period. Sixty of these patients (18%) were transferred to the ICU and placed on mechanical ventilators. A total of 20 of these 60 patients (33%) had IFIs. Multivariate analysis indicated that independent risk factors for IFI were admission to an ICU more than 40 days after HSCT, graft versus host disease (GVHD), and high dose corticosteroid (<it>p </it>< 0.01 for all). The overall ICU mortality rate was 88% (53 of 60 patients), and was not significantly different for patients with IFIs (85%) and those without IFIs (90%, <it>p </it>= 0.676).</p> <p>Conclusion</p> <p>There was a high incidence of IFIs in HSCT patients requiring MV in the ICU in our study cohort. The independent risk factors for IFI are ICU admission more than 40 days after HSCT, GVHD, and use of high-dose corticosteroid.</p