6,377 research outputs found
On the Inconsistencies of Conditionals Learned by Masked Language Models
Learning to predict masked tokens in a sequence has been shown to be a
powerful pretraining objective for large language models. After training, such
masked language models can provide distributions of tokens conditioned on
bidirectional context.
In this paper, we show that contrary to popular assumptions, such
bidirectional conditionals often demonstrate considerable inconsistencies,
i.e., they cannot be derived from a coherent joint distribution when considered
together. We empirically quantify such inconsistencies in the simple scenario
of bigram comparison for two common styles of masked language models: T5-style
and BERT-style. For example, we show that T5 models often confuse their own
preference regarding two similar bigrams. We show that inconsistencies exist
ubiquitously in masked language models of diverse sizes and configurations,
from RoBERTa-base to GLM-130B.
As an initial attempt to address this issue during the inference phase, we
propose Ensemble of Conditionals, a self-ensemble algorithm that jointly
considers many inconsistent conditionals directly produced by the MLM to
synthesize a distribution that is used as the model's final output. Such
ensembling improves open-source SOTA results on LAMBADA.Comment: Updated version; Added more experiments on "Ensemble of Conditionals
Fruit and Vegetable Planting Restrictions: Do U.S. Farmers Even Notice?
Crop Production/Industries, Food Consumption/Nutrition/Food Safety,
Gaps in state funding mean that liberal cities in conservative states are more likely to lobby the federal government
When we hear the term 'lobbying' we tend to think mostly in terms of corporations and ideological groups working to influence policy outcomes at the state and federal level. But cities are often lobbyists too. In new research, Rebecca Goldstein and Hye Young You examine the causes and consequences of lobbying the federal government by American cities. They find that liberal cities in conservative states such as New Orleans, Tucson, and Houston were the most likely to lobby, a trend that they explain may be down to gaps between how much these cities and their states spend on public services and infrastructure
Estimates of Invariant Metrics on Pseudoconvex Domains of Finite Type in C
Let Ω be a smoothly bounded pseudoconvex domain in C3 and assume that z0∈bΩ is a point of finite 1-type in the sense of D’Angelo. Then, there are an admissible curve Γ⊂Ω∪{z0}, connecting points  q0∈Ω and z0∈bΩ, and a quantity M(z,X), along z∈Γ, which bounds from above and below the Bergman, Caratheodory, and Kobayashi metrics in a small constant and large constant sense
- …