Vision Language Transformers: A Survey

Vision language tasks, such as answering questions about or generating captions that describe an image, are difficult tasks for computers to perform. A relatively recent body of research has adapted the pretrained transformer architecture introduced in \citet{vaswani2017attention} to vision language modeling. Transformer models have greatly improved performance and versatility over previous vision language models. They do so by pretraining models on a large generic datasets and transferring their learning to new tasks with minor changes in architecture and parameter values. This type of transfer learning has become the standard modeling practice in both natural language processing and computer vision. Vision language transformers offer the promise of producing similar advancements in tasks which require both vision and language. In this paper, we provide a broad synthesis of the currently available research on vision language transformer models and offer some analysis of their strengths, limitations and some open questions that remain

Exploring Transformers as Compact, Data-Efficient Language Models

Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch

Tiny Language Models Enriched with Multimodal Knowledge from Multiplex Networks

Large transformer language models trained exclusively on massive quantities of text are now the standard in NLP. In addition to the impractical amounts of data used to train them, they require enormous computational resources for training. Furthermore, they lack the rich array of sensory information available to humans, who can learn language with much less exposure to language. In this study, performed for submission in the BabyLM challenge, we show that we can improve a small transformer model’s data efficiency by enriching its embeddings by swapping the learned word embeddings from a tiny transformer model with vectors extracted from a custom multiplex network that encodes visual and sensorimotor information. Further, we use a custom variation of the ELECTRA model that contains less than 7 million parameters and can be trained end-to-end using a single GPU. Our experiments show that models using these embeddings outperform equivalent models when pretrained with only the small BabyLM dataset, containing only 10 million words of text, on a variety of natural language understanding tasks from the GLUE and SuperGLUE benchmarks and a variation of the BLiMP task

A Randomized Controlled Trial of a Faith-Placed, Lay Health Advisor Delivered Smoking Cessation Intervention for Rural Residents

Introduction. Rural US residents smoke at higher rates than urban or suburban residents. We report results from a community-based smoking cessation intervention in Appalachian Kentucky. Study design. Single-blind, group-randomized trial with outcome measurements at baseline, 17 weeks and 43 weeks. Setting/participants. This faith-placed CBPR project was located in six counties of rural Appalachian Kentucky. A total of 590 individual participants clustered in 28 churches were enrolled in the study. Intervention. Local lay health advisors delivered the 12-week Cooper/Clayton Method to Stop Smoking program, leveraging sociocultural factors to improve the cultural salience of the program for Appalachian smokers. Participants met with an interventionist for one 90 min group session once per week incorporating didactic information, group discussion, and nicotine replacement therapy. Main outcome measures. The primary outcome was self-reported smoking status. Secondary outcomes included Fagerström nicotine dependence, self-efficacy, and decisional balance. Results. With post-intervention data from 92% of participants, those in intervention group churches (N = 383) had 13.6 times higher odds of reporting quitting smoking one month post-intervention than participants in attention control group churches (N = 154, p \u3c 0.0001). In addition, although only 3.2% of attention control group participants reported quitting during the control period, 15.4% of attention control participants reported quitting smoking after receiving the intervention. A significant dose effect of the 12-session Cooper/Clayton Method was detected: for each additional session completed, the odds of quitting smoking increased by 26%. Conclusions. The Cooper/Clayton Method, delivered in rural Appalachian churches by lay health advisors, has strong potential to reduce smoking rates and improve individuals\u27 health

21-(4-Methyl­phenyl­sulfon­yl)-4,7,13,16-tetra­oxa-1,10,21-triaza­bicyclo­[8.8.5]tricosane-19,23-dione: an N-tosyl­ated macrobicyclic dilactam

The macrobicyclic title compound, C23H35N3O8S, contains two tertiary amide bridgehead N atoms and a toluene­sulfonamide N atom in the center of the five-atom bridging strand. The mol­ecule has a central cavity that is defined by the 18-membered ring identified by the N2O4 donor atom set and two 15-membered rings with N3O2 donor atom sets. The toluene­sulfonamide N atom adopts an exo orientation with respect to the central cavity, and the tosyl group is oriented on one side of the aza-bridging strand that connects the bridgehead N atoms

High Energy Gamma Ray Production from Proton Induced Reactions on D, C, Zn, Pb at Incident Energies of 104, 145, and 195 MeV

This research was sponsored by the National Science Foundation Grant NSF PHY-931478

Examining the safety of menstrual cups among rural primary school girls in western Kenya: observational studies nested in a randomised controlled feasibility study.

Examine the safety of menstrual cups against sanitary pads and usual practice in Kenyan schoolgirls. Observational studies nested in a cluster randomised controlled feasibility study. 30 primary schools in a health and demographic surveillance system in rural western Kenya. Menstruating primary schoolgirls aged 14-16 years participating in a menstrual feasibility study. Insertable menstrual cup, monthly sanitary pads or 'usual practice' (controls). Staphylococcus aureus vaginal colonization, Escherichia coli growth on sampled used cups, toxic shock syndrome or other adverse health outcomes. Among 604 eligible girls tested, no adverse event or TSS was detected over a median 10.9 months follow-up. S. aureusprevalence was 10.8%, with no significant difference over intervention time or between groups. Of 65 S.aureus positives at first test, 49 girls were retested and 10 (20.4%) remained positive. Of these, two (20%) sample isolates tested positive for toxic shock syndrome toxin-1; both girls were provided pads and were clinically healthy. Seven per cent of cups required replacements for loss, damage, dropping in a latrine or a poor fit. Of 30 used cups processed for E. coli growth, 13 (37.1%, 95% CI 21.1% to 53.1%) had growth. E. coli growth was greatest in newer compared with established users (53%vs22.2%, p=0.12). Among this feasibility sample, no evidence emerged to indicate menstrual cups are hazardous or cause health harms among rural Kenyan schoolgirls, but large-scale trials and post-marketing surveillance should continue to evaluate cup safety

Molecular Requirements for T Cell Recognition by a Major Histocompatibility Complex Class II–restricted T Cell Receptor: The Involvement of the Fourth Hypervariable Loop of the Vα Domain

The role of two central residues (K68, E69) of the fourth hypervariable loop of the Vα domain (HV4α) in antigen recognition by an MHC class II–restricted T cell receptor (TCR) has been analyzed. The TCR recognizes the NH2-terminal peptide of myelin basic protein (Ac1-11, acetylated at NH2 terminus) associated with the class II MHC molecule I-Au. Lysine 68 (K68) and glutamic acid 69 (E69) of HV4α have been mutated both individually and simultaneously to alanine (K68A, E69A). The responsiveness of transfectants bearing wild-type and mutated TCRs to Ac1-11–I-Au complexes has been analyzed in the presence and absence of expression of the coreceptor CD4. The data demonstrate that in the absence of CD4 expression, K68 plays a central role in antigen responsiveness. In contrast, the effect of mutating E69 to alanine is less marked. CD4 coexpression can partially compensate for the loss of activity of the K68A mutant transfectants, resulting in responses that, relative to those of the wild-type transfectants, are highly sensitive to anti-CD4 antibody blockade. The observations support models of T cell activation in which both the affinity of the TCR for cognate ligand and the involvement of coreceptors determine the outcome of the T cell–antigen-presenting cell interaction

Nuclear Reaction Network for Primordial Nucleosynthesis: a detailed analysis of rates, uncertainties and light nuclei yields

We analyze in details the standard Primordial Nucleosynthesis scenario. In particular we discuss the key theoretical issues which are involved in a detailed prediction of light nuclide abundances, as the weak reaction rates, neutrino decoupling and nuclear rate modeling. We also perform a new analysis of available data on the main nuclear processes entering the nucleosynthesis reaction network, with particular stress on their uncertainties as well as on their role in determining the corresponding uncertainties on light nuclide theoretical estimates. The current status of theoretical versus experimental results for 2H, 3He, 4He and 7Li is then discussed using the determination of the baryon density as obtained from Cosmic Microwave Background anisotropies.Comment: LaTeX, 83 pages, 30 .pdf figures. Some typos in the units of R-functions in appendix D and relative plots fixe

Catching Element Formation In The Act

Gamma-ray astronomy explores the most energetic photons in nature to address some of the most pressing puzzles in contemporary astrophysics. It encompasses a wide range of objects and phenomena: stars, supernovae, novae, neutron stars, stellar-mass black holes, nucleosynthesis, the interstellar medium, cosmic rays and relativistic-particle acceleration, and the evolution of galaxies. MeV gamma-rays provide a unique probe of nuclear processes in astronomy, directly measuring radioactive decay, nuclear de-excitation, and positron annihilation. The substantial information carried by gamma-ray photons allows us to see deeper into these objects, the bulk of the power is often emitted at gamma-ray energies, and radioactivity provides a natural physical clock that adds unique information. New science will be driven by time-domain population studies at gamma-ray energies. This science is enabled by next-generation gamma-ray instruments with one to two orders of magnitude better sensitivity, larger sky coverage, and faster cadence than all previous gamma-ray instruments. This transformative capability permits: (a) the accurate identification of the gamma-ray emitting objects and correlations with observations taken at other wavelengths and with other messengers; (b) construction of new gamma-ray maps of the Milky Way and other nearby galaxies where extended regions are distinguished from point sources; and (c) considerable serendipitous science of scarce events -- nearby neutron star mergers, for example. Advances in technology push the performance of new gamma-ray instruments to address a wide set of astrophysical questions.Comment: 14 pages including 3 figure