Impact of Tokenization on Language Models: An Analysis for Turkish

Tokenization is an important text preprocessing step to prepare input tokens for deep language models. WordPiece and BPE are de facto methods employed by important models, such as BERT and GPT. However, the impact of tokenization can be different for morphologically rich languages, such as Turkic languages, where many words can be generated by adding prefixes and suffixes. We compare five tokenizers at different granularity levels, i.e. their outputs vary from smallest pieces of characters to the surface form of words, including a Morphological-level tokenizer. We train these tokenizers and pretrain medium-sized language models using RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. We then fine-tune our models on six downstream tasks. Our experiments, supported by statistical tests, reveal that Morphological-level tokenizer has challenging performance with de facto tokenizers. Furthermore, we find that increasing the vocabulary size improves the performance of Morphological and Word-level tokenizers more than that of de facto tokenizers. The ratio of the number of vocabulary parameters to the total number of model parameters can be empirically chosen as 20% for de facto tokenizers and 40% for other tokenizers to obtain a reasonable trade-off between model size and performance.Comment: submitted to ACM TALLI

Anncolvar: Approximation of Complex Collective Variables by Artificial Neural Networks for Analysis and Biasing of Molecular Simulations

The state of a molecular system can be described in terms of collective variables. These low-dimensional descriptors of molecular structure can be used to monitor the state of the simulation, to calculate free energy profiles or to accelerate rare events by a bias potential or a bias force. Frequent calculation of some complex collective variables may slow down the simulation or analysis of trajectories. Moreover, many collective variables cannot be explicitly calculated for newly sampled structures. In order to address this problem, we developed a new package called anncolvar. This package makes it possible to build and train an artificial neural network model that approximates a collective variable. It can be used to generate an input for the open-source enhanced sampling simulation PLUMED package, so the collective variable can be monitored and biased by methods available in this program. The computational efficiency and the accuracy of anncolvar are demonstrated on selected molecular systems (cyclooctane derivative, Trp-cage miniprotein) and selected collective variables (Isomap, molecular surface area)

Higher COVID-19 pneumonia risk associated with anti-IFN-α than with anti-IFN-ω auto-Abs in children

We found that 19 (10.4%) of 183 unvaccinated children hospitalized for COVID-19 pneumonia had autoantibodies (auto-Abs) neutralizing type I IFNs (IFN-alpha 2 in 10 patients: IFN-alpha 2 only in three, IFN-alpha 2 plus IFN-omega in five, and IFN-alpha 2, IFN-omega plus IFN-beta in two; IFN-omega only in nine patients). Seven children (3.8%) had Abs neutralizing at least 10 ng/ml of one IFN, whereas the other 12 (6.6%) had Abs neutralizing only 100 pg/ml. The auto-Abs neutralized both unglycosylated and glycosylated IFNs. We also detected auto-Abs neutralizing 100 pg/ml IFN-alpha 2 in 4 of 2,267 uninfected children (0.2%) and auto-Abs neutralizing IFN-omega in 45 children (2%). The odds ratios (ORs) for life-threatening COVID-19 pneumonia were, therefore, higher for auto-Abs neutralizing IFN-alpha 2 only (OR [95% CI] = 67.6 [5.7-9,196.6]) than for auto-Abs neutralizing IFN-. only (OR [95% CI] = 2.6 [1.2-5.3]). ORs were also higher for auto-Abs neutralizing high concentrations (OR [95% CI] = 12.9 [4.6-35.9]) than for those neutralizing low concentrations (OR [95% CI] = 5.5 [3.1-9.6]) of IFN-omega and/or IFN-alpha 2

Vaccine breakthrough hypoxemic COVID-19 pneumonia in patients with auto-Abs neutralizing type I IFNs

Life-threatening `breakthrough' cases of critical COVID-19 are attributed to poor or waning antibody response to the SARS- CoV-2 vaccine in individuals already at risk. Pre-existing autoantibodies (auto-Abs) neutralizing type I IFNs underlie at least 15% of critical COVID-19 pneumonia cases in unvaccinated individuals; however, their contribution to hypoxemic breakthrough cases in vaccinated people remains unknown. Here, we studied a cohort of 48 individuals ( age 20-86 years) who received 2 doses of an mRNA vaccine and developed a breakthrough infection with hypoxemic COVID-19 pneumonia 2 weeks to 4 months later. Antibody levels to the vaccine, neutralization of the virus, and auto- Abs to type I IFNs were measured in the plasma. Forty-two individuals had no known deficiency of B cell immunity and a normal antibody response to the vaccine. Among them, ten (24%) had auto-Abs neutralizing type I IFNs (aged 43-86 years). Eight of these ten patients had auto-Abs neutralizing both IFN-a2 and IFN-., while two neutralized IFN-omega only. No patient neutralized IFN-ss. Seven neutralized 10 ng/mL of type I IFNs, and three 100 pg/mL only. Seven patients neutralized SARS-CoV-2 D614G and the Delta variant (B.1.617.2) efficiently, while one patient neutralized Delta slightly less efficiently. Two of the three patients neutralizing only 100 pg/mL of type I IFNs neutralized both D61G and Delta less efficiently. Despite two mRNA vaccine inoculations and the presence of circulating antibodies capable of neutralizing SARS-CoV-2, auto-Abs neutralizing type I IFNs may underlie a significant proportion of hypoxemic COVID-19 pneumonia cases, highlighting the importance of this particularly vulnerable population

Autoantibodies against type I IFNs in patients with critical influenza pneumonia

In an international cohort of 279 patients with hypoxemic influenza pneumonia, we identified 13 patients (4.6%) with autoantibodies neutralizing IFN-alpha and/or -omega, which were previously reported to underlie 15% cases of life-threatening COVID-19 pneumonia and one third of severe adverse reactions to live-attenuated yellow fever vaccine. Autoantibodies neutralizing type I interferons (IFNs) can underlie critical COVID-19 pneumonia and yellow fever vaccine disease. We report here on 13 patients harboring autoantibodies neutralizing IFN-alpha 2 alone (five patients) or with IFN-omega (eight patients) from a cohort of 279 patients (4.7%) aged 6-73 yr with critical influenza pneumonia. Nine and four patients had antibodies neutralizing high and low concentrations, respectively, of IFN-alpha 2, and six and two patients had antibodies neutralizing high and low concentrations, respectively, of IFN-omega. The patients' autoantibodies increased influenza A virus replication in both A549 cells and reconstituted human airway epithelia. The prevalence of these antibodies was significantly higher than that in the general population for patients 70 yr of age (3.1 vs. 4.4%, P = 0.68). The risk of critical influenza was highest in patients with antibodies neutralizing high concentrations of both IFN-alpha 2 and IFN-omega (OR = 11.7, P = 1.3 x 10(-5)), especially those <70 yr old (OR = 139.9, P = 3.1 x 10(-10)). We also identified 10 patients in additional influenza patient cohorts. Autoantibodies neutralizing type I IFNs account for similar to 5% of cases of life-threatening influenza pneumonia in patients <70 yr old

Impact of opioid-free analgesia on pain severity and patient satisfaction after discharge from surgery: multispecialty, prospective cohort study in 25 countries

Background: Balancing opioid stewardship and the need for adequate analgesia following discharge after surgery is challenging. This study aimed to compare the outcomes for patients discharged with opioid versus opioid-free analgesia after common surgical procedures.Methods: This international, multicentre, prospective cohort study collected data from patients undergoing common acute and elective general surgical, urological, gynaecological, and orthopaedic procedures. The primary outcomes were patient-reported time in severe pain measured on a numerical analogue scale from 0 to 100% and patient-reported satisfaction with pain relief during the first week following discharge. Data were collected by in-hospital chart review and patient telephone interview 1 week after discharge.Results: The study recruited 4273 patients from 144 centres in 25 countries; 1311 patients (30.7%) were prescribed opioid analgesia at discharge. Patients reported being in severe pain for 10 (i.q.r. 1-30)% of the first week after discharge and rated satisfaction with analgesia as 90 (i.q.r. 80-100) of 100. After adjustment for confounders, opioid analgesia on discharge was independently associated with increased pain severity (risk ratio 1.52, 95% c.i. 1.31 to 1.76; P &lt; 0.001) and re-presentation to healthcare providers owing to side-effects of medication (OR 2.38, 95% c.i. 1.36 to 4.17; P = 0.004), but not with satisfaction with analgesia (beta coefficient 0.92, 95% c.i. -1.52 to 3.36; P = 0.468) compared with opioid-free analgesia. Although opioid prescribing varied greatly between high-income and low- and middle-income countries, patient-reported outcomes did not.Conclusion: Opioid analgesia prescription on surgical discharge is associated with a higher risk of re-presentation owing to side-effects of medication and increased patient-reported pain, but not with changes in patient-reported satisfaction. Opioid-free discharge analgesia should be adopted routinely

Natural scene reconstruction from fMRI signals using generative latent diffusion

Abstract In neural decoding research, one of the most intriguing topics is the reconstruction of perceived natural images based on fMRI signals. Previous studies have succeeded in re-creating different aspects of the visuals, such as low-level properties (shape, texture, layout) or high-level features (category of objects, descriptive semantics of scenes) but have typically failed to reconstruct these properties together for complex scene images. Generative AI has recently made a leap forward with latent diffusion models capable of generating high-complexity images. Here, we investigate how to take advantage of this innovative technology for brain decoding. We present a two-stage scene reconstruction framework called “Brain-Diffuser”. In the first stage, starting from fMRI signals, we reconstruct images that capture low-level properties and overall layout using a VDVAE (Very Deep Variational Autoencoder) model. In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling “ROI-optimal” scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e.g. brain–computer interface) and fundamental neuroscience

Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs

International audienc