RNNs Implicitly Implement Tensor Product Representations

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated by TPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations.Comment: Accepted to ICLR 201

Touch down in Pittsburghese

In standard American English, down may take a DP object only if the DP indicates a path, as in I walked down the street. However, for some speakers of Pittsburgh English, it is also grammatical for down to take a DP object indicating a location or goal, as in She works down Baltimore (meaning ‘She works down in Baltimore’). In this work, I describe the distributional properties of this usage, which I name “touch down.” Based on these properties, I propose the syntactic analysis that touch down licenses a silent preposition where standard American English has an overt preposition, and that this silent preposition incorporates into down

Law Student Advocates and Conflicts of Interest

Law students who represent clients under attorney supervision are subject to no clear conflict of interest rules. Whether they are considered lawyers or nonlawyers for purposes of each state\u27s ethics rules is uncertain. Available rules governing lawyer and nonlawyer conflicts of interest ignore the competing interests of legal education, law student employment, clients, and public service. This Comment proposes a student conflict of interest rule that balances these interests by (1) holding student advocates to high ethical standards and (2) allowing screening to cure most conflicts that occur within student representation and that would otherwise handicap students in future employmen

A Coherent Methodology For First Amendment Speech and Religion Clause Cases

It seems clear that any deliberate effort by government to impose religious orthodoxy will be held unconstitutional per se. A religiously motivated restriction on disfavored religious practices will be held to violate the Free Exercise Clause. Similarly, a religiously motivated attempt to promote or subsidize favored religious practices will be held to violate the Establishment Clause. These complimentary restrictions are now so ingrained in our political culture that the legislatures rarely transgress them. The problem that has bedeviled the Supreme Court for many years is that government regulatory schemes and benefit programs designed to serve purely nonreligious objectives inevitably impact on religion inadvertently. In applying the Free Exercise Clause to cases of inadvertent interference, the Court adopted one fundamental doctrinal construct, promptly overruled that construct, adopted a nearly opposite principle, and then years later resurrected the original approach. In applying the Establishment Clause, the Court consistently articulated the same principle or test over many years, but produced a series of apparently inconsistent results. Most recently, the Court has consciously avoided articulating any standard or test in finding that a governmental action violates the Establishment Clause. The task confronting the Court is to develop a coherent jurisprudence to deal with the frequent inadvertent collisions between governmental actions and the absolute prohibitions of the two religion clauses. Unfortunately, the Supreme Court appears unaware that this is precisely the same systemic jurisprudential question that is presented when similar regulations inadvertently affect the interests protected by the Free Speech Clause. It is the central thesis of this Article that the conceptual methodology developed by the Court for dealing with inadvertence in the free speech context is the only sensible approach to the inadvertence problem in the context of any of the absolutely worded prohibitions of the First Amendment, including the two religion clauses. This Article concludes with an argument that those current threads in religion clause jurisprudence that appear to parallel the free speech methodology should be refined and reinforced while those that diverge from the free speech methodology should be abandoned