On the Inconsistencies of Conditionals Learned by Masked Language Models

Learning to predict masked tokens in a sequence has been shown to be a powerful pretraining objective for large language models. After training, such masked language models can provide distributions of tokens conditioned on bidirectional context. In this paper, we show that contrary to popular assumptions, such bidirectional conditionals often demonstrate considerable inconsistencies, i.e., they cannot be derived from a coherent joint distribution when considered together. We empirically quantify such inconsistencies in the simple scenario of bigram comparison for two common styles of masked language models: T5-style and BERT-style. For example, we show that T5 models often confuse their own preference regarding two similar bigrams. We show that inconsistencies exist ubiquitously in masked language models of diverse sizes and configurations, from RoBERTa-base to GLM-130B. As an initial attempt to address this issue during the inference phase, we propose Ensemble of Conditionals, a self-ensemble algorithm that jointly considers many inconsistent conditionals directly produced by the MLM to synthesize a distribution that is used as the model's final output. Such ensembling improves open-source SOTA results on LAMBADA.Comment: Updated version; Added more experiments on "Ensemble of Conditionals

Fruit and Vegetable Planting Restrictions: Do U.S. Farmers Even Notice?

Crop Production/Industries, Food Consumption/Nutrition/Food Safety,

Gaps in state funding mean that liberal cities in conservative states are more likely to lobby the federal government

When we hear the term 'lobbying' we tend to think mostly in terms of corporations and ideological groups working to influence policy outcomes at the state and federal level. But cities are often lobbyists too. In new research, Rebecca Goldstein and Hye Young You examine the causes and consequences of lobbying the federal government by American cities. They find that liberal cities in conservative states such as New Orleans, Tucson, and Houston were the most likely to lobby, a trend that they explain may be down to gaps between how much these cities and their states spend on public services and infrastructure

Estimates of Invariant Metrics on Pseudoconvex Domains of Finite Type in C

Let Ω be a smoothly bounded pseudoconvex domain in C3 and assume that z0∈bΩ is a point of finite 1-type in the sense of D’Angelo. Then, there are an admissible curve Γ⊂Ω∪{z0}, connecting points  q0∈Ω and z0∈bΩ, and a quantity M(z,X), along z∈Γ, which bounds from above and below the Bergman, Caratheodory, and Kobayashi metrics in a small constant and large constant sense