Building and Evaluating Open-Vocabulary Language Models

Language models have always been a fundamental NLP tool and application. This thesis focuses on open-vocabulary language models, i.e., models that can deal with novel and unknown words at runtime. We will propose both new ways to construct such models as well as use such models in cross-linguistic evaluations to answer questions of difficulty and language-specificity in modern NLP tools. We start by surveying linguistic background as well as past and present NLP approaches to tokenization and open-vocabulary language modeling (Mielke et al., 2021). Thus equipped, we establish desirable principles for such models, both from an engineering mindset as well as a linguistic one and hypothesize a model based on the marriage of neural language modeling and Bayesian nonparametrics to handle a truly infinite vocabulary, boasting attractive theoretical properties and mathematical soundness, but presenting practical implementation difficulties. As a compromise, we thus introduce a word-based two-level language model that still has many desirable characteristics while being highly feasible to run (Mielke and Eisner, 2019). Unlike the more dominant approaches of characters or subword units as one-layer tokenization it uses words; its key feature is the ability to generate novel words in context and in isolation. Moving on to evaluation, we ask: how do such models deal with the wide variety of languages of the world---are they struggling with some languages? Relating this question to a more linguistic one, are some languages inherently more difficult to deal with? Using simple methods, we show that indeed they are, starting with a small pilot study that suggests typological predictors of difficulty (Cotterell et al., 2018). Thus encouraged, we design a far bigger study with more powerful methodology, a principled and highly feasible evaluation and comparison scheme based again on multi-text likelihood (Mielke et al., 2019). This larger study shows that the earlier conclusion of typological predictors is difficult to substantiate, but also offers a new insight on the complexity of Translationese. Following that theme, we end by extending this scheme to machine translation models to answer questions traditional evaluation metrics like BLEU cannot (Bugliarello et al., 2020)

Are All Languages Equally Hard to Language-Model?

How cross-linguistically applicable are NLP models, specifically language models? A fair comparison between languages is tricky: not only do training corpora in different languages have different sizes and topics, some of which may be harder to predict than others, but standard metrics for language modeling depend on the orthography of a language. We argue for a fairer metric based on the bits per utterance using utterance-aligned multi-text. We conduct a study on 21 languages, training and testing both n-gram and LSTM language models on “the same” set of utterances in each language (modulo translation), demonstrating that in some languages, especially those with complex inflectional morphology, the textual expression of the information is harder to predict

Towards resolution of the Fermi surface in underdoped high-Tc superconductors

We survey recent experimental results including quantum oscillations and complementary measurements probing the electronic structure of underdoped cuprates, and theoretical proposals to explain them. We discuss quantum oscillations measured at high magnetic fields in the underdoped cuprates that reveal a small Fermi surface section comprising quasiparticles that obey Fermi-Dirac statistics, unaccompanied by other states of comparable thermodynamic mass at the Fermi level. The location of the observed Fermi surface section at the nodes is indicated by a body of evidence including the collapse in Fermi velocity measured by quantum oscillations, which is found to be associated with the nodal density of states observed in angular resolved photoemission, the persistence of quantum oscillations down to low fields in the vortex state, the small value of density of states from heat capacity and the multiple frequency quantum oscillation pattern consistent with nodal magnetic breakdown of bilayer-split pockets. A nodal Fermi surface pocket is further consistent with the observation of a density of states at the Fermi level concentrated at the nodes in photoemission experiments, and the antinodal pseudogap observed by photoemission, optical conductivity, nuclear magnetic resonance Knight shift, as well as other complementary diffraction, transport and thermodynamic measurements. One of the possibilities considered is that the small Fermi surface pockets observed at high magnetic fields can be understood in terms of Fermi surface reconstruction by a form of small wavevector charge order, observed over long lengthscales in experiments such as nuclear magnetic resonance and x-ray scattering, potentially accompanied by an additional mechanism to gap the antinodal density of states.Comment: 33 pages, 15 figures (this version updated with new figures and additional text, as published

Post-transplant cyclophosphamide versus antithymocyte globulin in patients with acute myeloid leukemia in first complete remission undergoing allogeneic stem cell transplantation from 10/10 HLA-matched unrelated donors

Background Graft-versus-host disease (GVHD) remains a major contributor to mortality and morbidity after allogeneic stem-cell transplantation (allo-HSCT). The updated recommendations suggest that rabbit antithymocyte globulin or anti-T-lymphocyte globulin (ATG) should be used for GVHD prophylaxis in patients undergoing matched-unrelated donor (MUD) allo-HSCT. More recently, using post-transplant cyclophosphamide (PTCY) in the haploidentical setting has resulted in low incidences of both acute (aGVHD) and chronic GVHD (cGVHD). Therefore, the aim of our study was to compare GVHD prophylaxis using either PTCY or ATG in patients with acute myeloid leukemia (AML) who underwent allo-HSCT in first remission (CR1) from a 10/10 HLA-MUD. Methods Overall, 174 and 1452 patients from the EBMT registry receiving PTCY and ATG were included. Cumulative incidence of aGVHD and cGVHD, leukemia-free survival, overall survival, non-relapse mortality, cumulative incidence of relapse, and refined GVHD-free, relapse-free survival were compared between the 2 groups. Propensity score matching was also performed in order to confirm the results of the main analysis Results No statistical difference between the PTCY and ATG groups was observed for the incidence of grade II-IV aGVHD. The same held true for the incidence of cGVHD and for extensive cGVHD. In univariate and multivariate analyses, no statistical differences were observed for all other transplant outcomes. These results were also confirmed using matched-pair analysis. Conclusion These results highlight that, in the10/10 HLA-MUD setting, the use of PTCY for GVHD prophylaxis may provide similar outcomes to those obtained with ATG in patients with AML in CR1.Peer reviewe

Innate immunodeficiency following genetic ablation of Mcl1 in natural killer cells

The cytokine IL-15 is required for natural killer (NK) cell homeostasis; however, the intrinsic mechanism governing this requirement remains unexplored. Here we identify the absolute requirement for myeloid cell leukaemia sequence-1 (Mcl1) in the sustained survival of NK cells in vivo. Mcl1 is highly expressed in NK cells and regulated by IL-15 in a dose-dependent manner via STAT5 phosphorylation and subsequent binding to the 3'-UTR of Mcl1. Specific deletion of Mcl1 in NK cells results in the absolute loss of NK cells from all tissues owing to a failure to antagonize pro-apoptotic proteins in the outer mitochondrial membrane. This NK lymphopenia results in mice succumbing to multiorgan melanoma metastases, being permissive to allogeneic transplantation and being resistant to toxic shock following polymicrobial sepsis challenge. These results clearly demonstrate a non-redundant pathway linking IL-15 to Mcl1 in the maintenance of NK cells and innate immune responses in vivo

Strong-coupling expansion and effective hamiltonians

When looking for analytical approaches to treat frustrated quantum magnets, it is often very useful to start from a limit where the ground state is highly degenerate. This chapter discusses several ways of deriving {effective Hamiltonians} around such limits, starting from standard {degenerate perturbation theory} and proceeding to modern approaches more appropriate for the derivation of high-order effective Hamiltonians, such as the perturbative continuous unitary transformations or contractor renormalization. In the course of this exposition, a number of examples taken from the recent literature are discussed, including frustrated ladders and other dimer-based Heisenberg models in a field, as well as the mapping between frustrated Ising models in a transverse field and quantum dimer models.Comment: To appear as a chapter in "Highly Frustrated Magnetism", Eds. C. Lacroix, P. Mendels, F. Mil

CNS Involvement at Initial Diagnosis and Risk of Relapse After Allogeneic HCT for Acute Lymphoblastic Leukemia in First Complete Remission

Outcomes of allogeneic hematopoietic cell transplantation (allo-HCT) for adult acute lymphoblastic leukemia (ALL) have improved over time. Studies have shown that total body irradiation (TBI) is the preferable type of myeloablative conditioning (MAC). However, outcomes based on central nervous system (CNS) involvement, namely CNS-positive versus CNS-negative, have not been compared. Here, we evaluated outcomes of 547 patients (CNS-positive = 96, CNS-negative = 451) who were allografted in the first complete remission (CR1) between 2009 and 2019. Primary endpoint was leukemia-free survival (LFS). Median follow-up was not different between the CNS-positive and CNS-negative groups (79 versus 67.2 months, P = 0.58). The CNS-positive group were younger (median age 31.3 versus 39.7 years, P = 0.004) and were allografted more recently (median year 2012 versus 2010, P = 0.003). In both groups, MAC was the preferred approach (82.3% versus 85.6%, P = 0.41). On multivariate analysis, the CNS-positive group had higher incidence of relapse (RI) (hazard ratio [HR] = 1.58 [95% confidence interval (CI) = 1.06-2.35], P = 0.025), but no adverse effect on LFS (HR = 1.38 [95% CI = 0.99-1.92], P = 0.057) or overall survival (OS) (HR = 1.28 [95% CI = 0.89-1.85], P = 0.18). A subgroup multivariate analysis limited to CNS-positive patients showed that a TBI-based MAC regimen resulted in better LFS (HR = 0.43 [95% CI = 0.22-0.83], P = 0.01) and OS (HR = 0.44 [95% CI = 0.21-0.92], P = 0.03) and lower RI (HR = 0.35 [95% CI = 0.15-0.79], P = 0.01). Another subgroup analysis in CNS-negative patients showed that MAC-TBI preparative regimens also showed a lower RI without a benefit in LFS or OS. While a MAC-TBI allo-HCT regimen may not be suitable to all, particularly for older patients with comorbidities, this approach should be considered for patients who are deemed fit and able to tolerate.Peer reviewe

Multi-dimensional modeling and simulation of semiconductor nanophotonic devices

Self-consistent modeling and multi-dimensional simulation of semiconductor nanophotonic devices is an important tool in the development of future integrated light sources and quantum devices. Simulations can guide important technological decisions by revealing performance bottlenecks in new device concepts, contribute to their understanding and help to theoretically explore their optimization potential. The efficient implementation of multi-dimensional numerical simulations for computer-aided design tasks requires sophisticated numerical methods and modeling techniques. We review recent advances in device-scale modeling of quantum dot based single-photon sources and laser diodes by self-consistently coupling the optical Maxwell equations with semiclassical carrier transport models using semi-classical and fully quantum mechanical descriptions of the optically active region, respectively. For the simulation of realistic devices with complex, multi-dimensional geometries, we have developed a novel hp-adaptive finite element approach for the optical Maxwell equations, using mixed meshes adapted to the multi-scale properties of the photonic structures. For electrically driven devices, we introduced novel discretization and parameter-embedding techniques to solve the drift-diffusion system for strongly degenerate semiconductors at cryogenic temperature. Our methodical advances are demonstrated on various applications, including vertical-cavity surface-emitting lasers, grating couplers and single-photon sources