Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.Comment: NAACL 201

How Optimal is Greedy Decoding for Extractive Question Answering?

Fine-tuned language models use greedy decoding to answer reading comprehension questions with relative success. However, this approach does not ensure that the answer is a span in the given passage, nor does it guarantee that it is the most probable one. Does greedy decoding actually perform worse than an algorithm that does adhere to these properties? To study the performance and optimality of greedy decoding, we present exact-extract, a decoding algorithm that efficiently finds the most probable answer span in the context. We compare the performance of T5 with both decoding algorithms on zero-shot and few-shot extractive question answering. When no training examples are available, exact-extract significantly outperforms greedy decoding. However, greedy decoding quickly converges towards the performance of exact-extract with the introduction of a few training examples, becoming more extractive and increasingly likelier to generate the most probable span as the training set grows. We also show that self-supervised training can bias the model towards extractive behavior, increasing performance in the zero-shot setting without resorting to annotated examples. Overall, our results suggest that pretrained language models are so good at adapting to extractive question answering, that it is often enough to fine-tune on a small training set for the greedy algorithm to emulate the optimal decoding strategy.Comment: AKBC 2022 12 pages, 3 figure

Generating Benchmarks for Factuality Evaluation of Language Models

Before deploying a language model (LM) within a given domain, it is important to measure its tendency to generate factually incorrect information in that domain. Existing factual generation evaluation methods focus on facts sampled from the LM itself, and thus do not control the set of evaluated facts and might under-represent rare and unlikely facts. We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We use our framework to create two benchmarks: Wiki-FACTOR and News-FACTOR. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score correlates with perplexity, but the two metrics do not always agree on model ranking; and (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation, as measured by human annotators. We make our data and code publicly available in https://github.com/AI21Labs/factor

Helicobacter pylori serology in autoimmune diseases - Fact or fiction?

Background: The pathogenesis of autoimmunity is presumed to be a complex process including genetic predisposition, hormonal balance and environmental factors such as infectious agents . Helicobacter pylori , a common bacterial infectious agent has been associated with a variety of autoimmune disorders. However, this bacteria is also thought to play a protective role in the development of multiple sclerosis (MS), systemic lupus erythematosus (SLE) and inflammatory bowel disease (IBD). We tested various links between anti- H. pylori (anti-HP) antibodies and a wide profile of autoimmune diseases and autoantibodies. Methods: A total of 1290 patients diagnosed with 14 different autoimmune diseases from two geographical areas (Europe and Latin America) and two groups of healthy matching controls (n = 385) were screened for the presence of H. pylori IgG antibodies by ' pylori detect ' kit. In parallel, a large profile belonging to three groups of autoantibodies was tested in all sera (anti-nuclear antibodies, autoantibodies associated with thrombophilia and gastrointestinal diseases). Results: Our data demonstrate associations between anti-HP antibodies and anti-phospholipid syndrome, giant cell arteritis, systemic sclerosis and primary biliary cirrhosis. Our data also support a previously known negative association between the prevalence of anti-HP antibodies and IBD. Additionally, links were made between seropositivity to H. pylori and the presence of anti-nuclear antibodies, dsDNA, anti-Ro and some thrombophiliaassociated antibodies, as well as negative associations with gastrointestinal-associated antibodies. Conclusions: Whether these links are epiphenomenal or H. pylori does play a causative role in the autoimmune diseases remains uncertain. The negative associations could possibly support the notion that in susceptible individuals infections may protect from the development of autoimmune diseases

Information advantage and dominant strategies in second-price auctions

We study a general model of common-value second-price auctions with differential information. We show that one of the bidders has an information advantage over the other bidders if and only if he possesses a dominant strategy. A dominant strategy is, in fact, unique, and is given by the conditional expectation of the common value with respect to his information field. Furthermore, when a bidder has information advantage, other bidders cannot make a profit

DOMINANCE SOLVABILITY OF SECOND-PRICE AUCTIONS WITH DIFFERENTIAL INFORMATION

We study a class of common-value second-price auctions with differential information. This class of common-value auctions is characterized by the property that each player’s information set is connected with respect to the common value. We show that the entire class is dominance solvable, and that there is a natural single-valued selection from the resulting set of sophisticated equilibria. Additionally, it is shown that bidder’s information advantage over others is rewarded in sophisticated equilibria