Masked Language Model Scoring

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end-to-end LibriSpeech model's WER by 30% relative and adds up to +1.7 BLEU on state-of-the-art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL's unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling computation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PPPLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore translations in multiple languages. We release our library for language model scoring at https://github.com/awslabs/mlm-scoring.Comment: ACL 2020 camera-ready (presented July 2020

Firelight Foundation: An interim evaluation report of the Early Learning Innovation Fund

The Hewlett Foundation in 2014 selected Management Systems International (MSI) to implement a midterm evaluation of the Early Learning Innovation Fund. This evaluation explores the concept and design of the Fund; progress in achieving the Hewlett Foundation's four intermediary outcomes; and Firelight's implementation of the innovation fund with a focus on its approach to capacity building and expanding innovative programs. This evaluation also reviews the quality of the sub-grantees' monitoring and evaluation (M&E) systems and explores the potential of conducting an impact evaluation of sub-grantee activities

International Investment Law

Since the middle of the twentieth century, the field of international investment protection has gone through a period of more or less continuous expansion. From a single bilateral investment treaty (‘BIT’) signed between Germany and Pakistan in November 1959, international investment law has seen the proliferation of some 3,200 investment treaties governing the treatment of foreign investors by the host States where they do business. As a historical matter, the substantive elements of modern investment law emerged from a loose network of customary international law protections that pre-existed the treaties now dominating the regime. Customary international law had long required host States to extend certain guarantees of decent treatment to foreign citizens within their jurisdiction. The systematic codification of these customary norms into a far-flung network of treaties began in earnest with the late nineteenth century emergence of so-called ‘friendship, commerce, and navigation’ treaties, which incorporated existing customary rules and adopted various new substantive requirements. The treaty network took its next step when BITs proper emerged in the mid-twentieth century, characterised principally by the extension of dispute resolution options to individual investors. As customary investment law was gradually codified at the retail level, the law of treaties began to loom much larger in meta-regulation of the regime. This chapter will explore some of the ways that the modern law of treaties interacts with the modern law of international investment protection. It will focus in particular on a handful of areas where the formal categories of treaty law map awkwardly onto the reality of modern investment law and adjudication

Article II Vests Executive Power, Not the Royal Prerogative

Article II of the United States Constitution vests “the executive power” in the President. For more than two hundred years, advocates of presidential power have claimed that this phrase was originally understood to include a bundle of national security and foreign affairs authorities. Their efforts have been highly successful. Among constitutional originalists, this so-called “Vesting Clause Thesis” is now conventional wisdom. But it is also demonstrably wrong. Based on an exhaustive review of the eighteenth-century bookshelf, this Article shows that the ordinary meaning of “executive power” referred unambiguously to a single, discrete, and potent authority: the power to execute law. This enforcement role was constitutionally crucial. Substantively, however, it extended only to the implementation of legal norms created by some other authority. It wasn’t just that the executive power was subject to legislative influence in a crude political sense; rather, the power was conceptually an empty vessel until there were laws or instructions that needed executing

Law Matters, Even to the Executive

In both constitutional and international law, many legal rules cannot be implemented without what most people would describe as the voluntary compliance of their target. Is that really “law”? Or is rule compliance in such circumstances just an expression of “interests”? Forget jurisprudence for the moment. As a practical matter, what does it mean to work as a lawyer in a field where the rules are not coercively enforced against private parties by an independent judiciary whose orders are implemented by a cooperative executive? This question has particularly high stakes for national security policy, where we find judicial deference at its highest, the centralization of modern government at its most pronounced, delegations of authority to the executive at their broadest, and contempt for idealism at its most self-satisfied. Two recent books on executive power prompt this return to such well-trodden ground. In The Executive Unbound: After the Madisonian Republic, Eric Posner and Adrian Vermeule claim that the constitutional rule-of-law apparatus is basically worthless. In Power and Constraint: The Accountable Presidency After 9/11, Jack Goldsmith says just about the opposite. This Review argues that Goldsmith is right and supplements his account by identifying a key mechanism in the political economy he describes. The Review begins by separating the various threads of argument advanced by Posner and Vermeule to expose how implausible their conceptual claims will seem to most lawyers. It then explores how their (largely unsupported) descriptive claims are contradicted by Goldsmith’s empirical account as well as by other evidence adduced here. The Review closes by suggesting that one of the most plausible causal mechanisms for the efficacy of law — the deep vein of respect for legality that characterizes our culture — is itself a primary target of Posner and Vermeule’s project

The Executive Power Clause

Article II of the Constitution vests “the executive power” in the President. Advocates of presidential power have long claimed that this phrase was originally understood as a term of art for the full suite of powers held by a typical eighteenth-century monarch. In its strongest form, this view yields a powerful presumption of indefeasible presidential authority in the arenas of foreign affairs and national security. This so-called Vesting Clause Thesis is conventional wisdom among constitutional originalists. But it is also demonstrably wrong. Based on a comprehensive review of Founding-era archives—including records of drafting, legislative, and ratification debates, committee files, private and official correspondence, diaries, newspapers, pamphlets, poetry, and other publications—this article not only refutes the Vesting Clause Thesis as a statement of the original understanding, but replaces it with a comprehensive affirmative account of the clause that is both historically and theoretically coherent. The Founding generation understood “executive power” to mean something both simple and specific: the power to execute law. This authority was constitutionally indispensable, but it extended only to the implementation of pre-existing legal norms and directives that had been created pursuant to some prior exercise of legislative authority. It wasn’t just that the use of executive power was subject to legislative influence in a crude political sense; rather, the power was conceptually an empty vessel until there were laws or instructions that needed executing

Sharing Experiences to Cultivate “a More Open Mind about Teaching”: A Co/Autoethnography of Pre-Collegiate Teaching Experiences

The purpose of this study was to create a third space between a teacher educator and an undergraduate student to explore teaching experiences while in high school within NFE programs in which we participated. We developed a series of five prompts that we responded to individually, meeting via Zoom to discuss our experiences and teacher learning. Our collaborative and iterative analysis revealed four categories related to our non-formal education teaching experiences: 1) how teacher learning was impacted by particular contexts in which it took place; 2) how our experiences revealed different and broader notions of teaching than we observed in our formal education experiences; 3) how certain contradictions shaped, and perhaps inhibited, our learning as teachers; and 4) what we learned from the teaching we did in NFE programs. Our findings depict how young people in recent years may accrue experiences teaching not accounted for in previous scholarship

Contemporary Practice of the United States Relating to International Law

• Progress Is Made Implementing U.S.-Russia Framework for Eliminating Syrian Chemical Weapons • United States Advocates for Syrian Peace Conference • United States Extends Deadline for Signing of Bilateral Security Agreement with Afghanistan • China Announces New Air Defense Identification Zone over East China Sea, Prompting U.S. Response • United States and Six Other States Reach Interim Agreement on Iranian Nuclear Progra